Projector Is All You Train

arXiv:2608.197262026-08-21

让AI学会理解3D这类新信息时,不需要重新训练整个大语言模型

能理解图像、声音或3D形状等多种信息的多模态AI,通常需要同时训练语言模型主体和一个连接新数据类型的小型转换模块,叫做投影器。这项研究发现,只训练投影器、完全冻结语言模型主体,也能达到和联合训练相当甚至更好的3D理解效果。而且只训练投影器速度快约两倍,还能完全避免联合训练带来的语言模型原有能力受损的问题。

他们做了什么

研究以处理3D点云(用大量点的坐标和颜色表示物体表面的一种3D数据形式)的模型PointLLM系列为基础,比较了两种训练方式:只训练投影器(语言模型主体完全冻结)与联合训练投影器加LoRA适配器(一种给大模型附加的小型可训练参数)
针对Qwen3.5-4B、Qwen3.5-9B、Llama-3.1-8B-Instruct三种不同的语言模型主体,都在单张A100 GPU上训练16小时,以保证比较公平
在ModelNet40、Objaverse、OmniObject3D三个数据集的3D物体分类和描述生成任务上,只训练投影器的模型整体上得分高于现有的PointLLM基线模型,在相同算力预算下也能与联合训练的模型保持相当的竞争力
只训练投影器时,单位时间内处理的训练样本数量大约是联合训练的两倍
联合训练会导致语言模型在语言、视觉、空间推理等已有能力上的基准测试成绩下降,即出现能力漂移;而只训练投影器由于完全不改动语言模型主体,从根本上不会出现这种问题

Figure 1: Architecture of 3D MLLMs in this paper. This figure depicts projector-only training, where all parameters are frozen except those of the projector. An alternative training regime is joint training, where all parameters are frozen except those of the projector and LoRA adapters of the LLM backbone.

Table 1: Configurations of MLLM variants. All variants are trained for the full 16 GPU hours. We save checkpoints at 2, 4, 8, 12, and 16 hours.
ID	LM Backbone	Projector Trained	LoRA Fine-tuned
P-Llama8B	Llama-3.1-8B-Instruct	Yes	No
J-Llama8B	Llama-3.1-8B-Instruct	Yes	Yes
P-Qwen4B	Qwen3.5-4B	Yes	No
J-Qwen4B	Qwen3.5-4B	Yes	Yes
P-Qwen9B	Qwen3.5-9B	Yes	No
J-Qwen9B	Qwen3.5-9B	Yes	Yes

Figure 2: Evaluation results of projector-only-trained MLLMs against baseline PointLLM models. Chart (a) shows generative 3D object classification results on ModelNet40 (M40.), Objaverse (Obj.), and OmniObject3D (Omni.) objects under a zero-shot setting. There are two prompt types per benchmark: an instruction-style prompt (I, “What is this?”) and a completion-style prompt (C, “This is an object of”). Each entry reports accuracy judged by GPT-5.6 Luna [28]. Chart (b) shows the aggregate precision score on 3D object captioning tasks under the three judge LLMs. More details are found in Appendix B.

Table 2: Generative 3D object classification results on ModelNet40 (M40.), Objaverse (Obj.), and OmniObject3D (Omni.) under a zero-shot setting as reported in Figure 2 (a).
Model	M40. (I)	M40. (C)	Obj. (I)	Obj. (C)	Omni. (I)	Omni. (C)
PointLLM-7B [43]	54.94	55.55	59.83	60.43	34.78	35.70
PointLLM-13B [43]	57.66	56.81	61.73	60.90	34.63	34.01
P-Llama8B	61.95	63.57	65.60	62.33	49.19	40.69
P-Qwen4B	59.56	59.56	60.90	60.83	41.07	40.25
P-Qwen9B	59.28	62.80	74.07	67.77	48.33	38.76
Mean	58.68	59.66	64.43	62.45	41.60	37.88

Figure 3: M40. (top) and Obj. (bottom) mean accuracy against GPU hours. Projector-only training generally scores higher than joint training across all GPU hours.

Table 3: 3D object captioning results on Objaverse as reported in Figure 2 (b). C refers to correctness, H to hallucination, and P to precision (aggregate of C and H).
	GPT-5.6 Luna	Claude Haiku 4.5	Gemini 3.5 Flash-Lite
Model	C	H ↓	P	C	H ↓	P	C	H ↓	P
PointLLM-7B [43]	4.26	2.12	66.75	3.37	1.95	63.30	2.75	1.99	58.08
PointLLM-13B [43]	4.35	2.10	67.43	3.47	1.91	64.46	2.82	1.92	59.49
P-Llama8B	5.61	2.17	72.12	3.76	1.75	68.19	3.29	1.87	63.75
P-Qwen4B	6.41	1.83	77.81	4.28	1.57	73.11	3.72	1.65	69.22
P-Qwen9B	6.58	1.95	77.19	4.30	1.68	71.90	3.76	1.71	68.72
Mean	–	–	72.26	–	–	68.19	–	–	63.85

Figure 4: M40. (top) and Obj. (bottom) mean accuracy against training samples. Plotted points in these charts correspond to the same GPU-hour thresholds that in turn correspond to the plotted points in Figure 3. Projector-only training sees more samples than joint training at each GPU-hour threshold as it is approximately twice as fast.

Table 4: Results for language, vision, and spatial reasoning benchmarks. The LoRA fine-tuned backbones are compared against the corresponding base backbone.
	Llama-3.1-8B-Instruct	Qwen3.5-4B	Qwen3.5-9B
Benchmark	P	J	P	J	P	J
Language
MMLU-Pro [39]	37.32	11.21↓	45.04	45.48	51.38	51.30
MMLU-Redux [15]	60.37	22.83↓	69.13	70.53	74.57	74.27
GPQA Diamond [35]	33.33	24.24↓	33.33	41.41↑	45.96	49.49↑
IFEval [51]	72.46	10.35↓	80.41	38.82↓	83.55	67.84↓
IFBench [29]	26.33	16.33↓	29.33	17.67↓	32.67	28.33↓
GSM8K [8]	86.96	0.61↓	90.90	81.96↓	92.34	92.34
WinoGrande [36]	61.40	49.57↓	65.59	68.35↑	74.19	73.24
OpenBookQA [26]	81.60	27.60↓	86.20	86.90	90.30	92.20↑
HumanEval [7]	64.02	0.00↓	82.93	71.95↓	84.76	80.49↓
Vision
MMMU [47]	–	–	49.44	54.45↑	54.23	59.02↑
MMMU-Pro [48]	–	–	32.60	37.23↑	42.72	43.41
MMMU-Pro Vision [48]	–	–	31.45	34.16↑	40.46	40.64
MMStar [6]	–	–	53.27	61.67↑	65.87	65.67
BabyVision [5]	–	–	19.07	14.43↓	16.49	15.98
RealWorldQA [42]	–	–	74.12	69.54↓	75.42	76.08
Spatial
ERQA [16]	–	–	45.25	42.25↓	45.75	44.00↓
EmbSpatialBench [14]	–	–	75.14	75.16	76.59	76.95
RefSpatialBench [50]	–	–	20.94	1.81↓	38.27	27.80↓
LingoQA [25]	–	–	70.40	58.80↓	75.00	68.40↓

Table 5: Projector and LoRA-adapter parameter counts, and projector input/output shapes, for each backbone.
Backbone	Encoder dim c	Tokens m	Backbone dim c′	Proj. params	LoRA params
Llama-3.1-8B-Instruct	384	513	4096	10.89M	41.94M
Qwen3.5-4B	384	513	2560	7.74M	32.46M
Qwen3.5-9B	384	513	4096	10.89M	43.28M

Table 6: The set of system message paraphrases, drawn uniformly at random during training.
System message
Answer the question or follow the instruction regarding the given 3D object.
Answer the following question or carry out the instruction about the provided 3D object.
Given a 3D object, respond to the question or instruction about it.
Consider the 3D object and answer the question or follow the instruction that follows.
Respond to the question or instruction concerning the presented 3D object.
Using the given 3D object, answer the question or complete the instruction.
You are given a 3D object; answer the question or follow the instruction about it.
Examine the 3D object and answer the accompanying question or instruction.
Provide an answer to the question or complete the instruction about the given 3D object.
Based on the 3D object shown, answer the question or follow the instruction.
Address the question or instruction about the provided 3D object.

Table 7: Training and evaluation hyperparameters, shared across every configuration except the learning rates.
Hyperparameter	Value
Optimizer	AdamW, β=(0.9,0.999), ϵ=10−8
Learning-rate schedule	constant, no warmup
Projector learning rate	2×10−3 (1×10−3 for Llama8B)
Weight decay	0.0
Gradient clipping	global norm 1.0
Micro-batch / accumulation / effective	12 / 2 / 24
Max response length	512 tokens
Mixed precision	bfloat16 autocast
Random seed	0
Hardware	1× NVIDIA A100-80GB
Gradient checkpointing	enabled
LoRA rank / α / dropout	16 / 32 / 0.05
LoRA target modules	all-linear
LoRA learning rate	2×10−5 (2×10−4 for Llama8B)

Table 8: Total number of training samples seen by each MLLM variant, average throughput, and average step rate after 16 GPU hours.
ID	Total samples	Avg. throughput (h-1)	Avg. step rate (h-1)
P-Llama8B	432,936	27,056	1,127
J-Llama8B	200,640	12,537	522
P-Qwen4B	480,768	30,045	1,252
J-Qwen4B	245,592	15,346	639
P-Qwen9B	325,176	20,321	847
J-Qwen9B	174,720	10,918	455

Table 9: Generative 3D object classification results on ModelNet40 (M40.), Objaverse (Obj.), and OmniObject3D (Omni.) under a zero-shot setting. Extension of Table 2.
Model	M40. (I)	M40. (C)	Obj. (I)	Obj. (C)	Omni. (I)	Omni. (C)
ShapeLLM-7B [32]	18.76	17.95	30.23	31.37	15.28	18.75
ShapeLLM-13B [32]	22.45	21.60	40.67	38.90	28.66	33.33
PointLLM-7B [43]	54.94	55.55	59.83	60.43	34.78	35.70
PointLLM-13B [43]	57.66	56.81	61.73	60.90	34.63	34.01
PointLLM-R [3]	62.28	62.64	65.23	65.40	38.81	38.69
MiniGPT-3D [37]	63.41	62.84	63.97	63.00	41.10	39.27
P-Llama8B	61.95	63.57	65.60	62.33	49.19	40.69
J-Llama8B	49.72	57.37	66.80	59.53	42.34	37.17
J-NoLoRA-Llama8B	36.02	40.24	58.90	51.07	39.62	40.69
P-Qwen4B	59.56	59.56	60.90	60.83	41.07	40.25
J-Qwen4B	58.27	59.60	71.00	68.80	48.17	47.32
J-NoLoRA-Qwen4B	54.94	53.24	58.73	57.97	27.08	27.49
P-Qwen9B	59.28	62.80	74.07	67.77	48.33	38.76
J-Qwen9B	52.59	58.75	75.23	66.23	51.32	44.86
J-NoLoRA-Qwen9B	51.62	52.63	49.13	56.93	24.51	28.74
Curriculum-Qwen4B	53.48	54.34	74.80	75.63	46.08	46.73

Table 10: 3D object captioning results on Objaverse. Extension of Table 3.
	GPT-5.6 Luna	Claude Haiku 4.5	Gemini 3.5 Flash-Lite
Model	C	H ↓	P	C	H ↓	P	C	H ↓	P
ShapeLLM-7B [32]	2.23	1.52	59.47	1.65	1.46	53.15	1.03	1.69	37.85
ShapeLLM-13B [32]	2.68	1.84	59.32	2.04	1.68	54.93	1.39	2.02	40.81
PointLLM-7B [43]	4.26	2.12	66.75	3.37	1.95	63.30	2.75	1.99	58.08
PointLLM-13B [43]	4.35	2.10	67.43	3.47	1.91	64.46	2.82	1.92	59.49
PointLLM-R [3]	3.38	1.18	74.10	2.50	1.10	69.42	2.48	1.17	67.94
MiniGPT-3D [37]	4.51	2.01	69.16	3.19	1.93	62.32	2.70	1.95	58.07
P-Llama8B	5.61	2.17	72.12	3.76	1.75	68.19	3.29	1.87	63.75
J-Llama8B	6.09	2.35	72.15	4.01	1.95	67.28	3.41	2.11	61.79
J-NoLoRA-Llama8B	3.14	2.96	51.48	2.35	2.08	52.96	1.69	2.31	42.15
P-Qwen4B	6.41	1.83	77.81	4.28	1.57	73.11	3.72	1.65	69.22
J-Qwen4B	6.73	1.97	77.33	4.43	1.67	72.67	3.91	1.71	69.52
J-NoLoRA-Qwen4B	5.44	4.23	56.26	4.30	2.53	62.97	2.92	3.19	47.80
P-Qwen9B	6.58	1.95	77.19	4.30	1.68	71.90	3.76	1.71	68.72
J-Qwen9B	6.83	2.02	77.19	4.46	1.73	72.00	3.92	1.80	68.52
J-NoLoRA-Qwen9B	5.14	4.74	52.02	3.84	3.02	56.00	2.58	3.65	41.46
Curriculum-Qwen4B	6.52	1.95	76.96	4.36	1.67	72.33	3.76	1.74	68.38

Table 11: Generative zero-shot classification accuracy against wall-clock hours. This is calculated as the average of the I and C variants of the ModelNet40 and Objaverse benchmarks, directly comparable to Figure 3.
	M40. (Avg.)	Obj. (Avg.)
Model	2 h	4 h	8 h	12 h	16 h	2 h	4 h	8 h	12 h	16 h
P-Qwen4B	54.23	54.76	58.33	63.53	59.56	58.10	62.52	67.25	64.90	60.87
J-Qwen4B	46.84	51.64	51.03	52.09	58.93	58.20	66.50	73.28	75.05	69.90
P-Qwen9B	51.38	53.99	57.03	61.06	61.04	57.47	63.07	67.80	67.35	70.92
J-Qwen9B	39.49	46.84	54.52	58.53	55.67	51.20	60.48	71.08	67.90	70.73
P-Llama8B	53.85	57.22	57.80	56.14	62.76	51.48	57.58	66.10	70.07	63.97
J-Llama8B	45.50	48.28	47.49	56.62	53.55	47.80	54.02	62.28	65.23	63.17

Table 12: Example generations on long captioning tasks. The MLLMs are given the prompt “Caption this 3D model in detail.” All prompts are independently provided to the MLLMs. The answers shown in this table were generated by P-Llama8B and J-Llama8B at 16 hours of training, as well as the two PointLLM models.
Object
PointLLM-7B	The 3D model represents a distinctive, green toy car that stands out due to its unconventional feature - a cornered driving unit. Designed for play, this model is aesthetically appealing with an eye-catching bright green color that would attract young children. The unique driving unit located at the corner provides an interesting twist to the traditional car design, promoting creative play and imaginative scenarios. However, despite its playful design, it is still a toy and should be handled with care.	The 3D object is a lively representation of a blue whale, which is the largest mammal on Earth. It is depicted in a playful cartoon style with an eye-catching cobalt-blue hue. The whale features proportional body parts, including a small dorsal fin, enormous mouth, and flat, wide head. It seems to be designed for entertainment or educational purposes, possibly for children, given its cartoon-like attributes and the significance of the whale in marine life.
PointLLM-13B	The 3D object model represents a small toy car, with the primary colour being a vibrant green. It features four black wheels that contrast well with the green body. Two red lights are situated at the front, possibly to mimic the headlights of an actual car. This model, being a toy, is typically used for children’s play, aiding in their motor skill development and imagination. The car’s simplistic design might be suitable for younger age groups.	This is a 3D model of a blue whale, the largest mammal on Earth, depicted in a cartoon style. The whale is distinguished by its small dorsal fin, enormous mouth, and flat, wide head. Despite the cartoon aesthetic, the model accurately represents the unique features of a blue whale. It could be used for educational purposes or in animated movies or documentaries.
P-Llama8B	The object is a green and white van with a boxy shape, featuring a prominent front grille, round headlights, and a rear window. It has a flat roof and a side door with a handle. The van has a distinctive green and white color scheme with black accents. The wheels are circular and black, and the vehicle has a distinctive front bumper.	The object is a sleek, streamlined shark with a pointed snout and a series of sharp, triangular fins along its back. The body is predominantly blue, with a lighter underside and a distinctive pattern of darker spots. The fins are a lighter shade, and the tail is long and narrow, tapering to a sharp point. The shark has a prominent dorsal fin and a smaller pectoral fin, with a series of gill slits visible along the sides.
J-Llama8B	The object is a green vehicle with a boxy design, featuring a large windshield, a rear window, and a front grille. It has a roof rack and a spare tire mounted on the rear. The vehicle appears to be a van, with a prominent front bumper and a rear section that includes a license plate. The wheels are circular, and the vehicle has a simple, utilitarian appearance.	The object is a stylized representation of a fish, characterized by a streamlined body with a pointed snout and a forked tail. The body is primarily blue with a white underbelly, and the fins are white. The fish has a smooth, abstract appearance with a minimalistic design.

为什么重要

对于开发AI系统的人来说,这意味着为模型增加新的理解能力(比如3D)时,不必重新训练整个庞大的语言模型,只需训练一个很小的连接模块,就能节省大量计算资源和时间,同时保留模型原有的能力。这也为一种模块化思路打开了可能:同一个语言模型可以搭配多个针对不同模态、各自独立训练的投影器。

本文术语

多模态大语言模型(MLLM) · 在文本之外,还能理解图像、3D形状、声音等其他类型信息的AI模型
投影器(projector) · 把3D点云等其他类型数据转换成语言模型能够处理的形式的小型神经网络模块
LoRA适配器 · 一种不用重新训练整个大模型、只训练少量附加参数就能高效微调模型的技术
点云(point cloud) · 用大量带位置和颜色信息的点来表示物体表面形状的一种3D数据格式
LLM充当评委(LLM-as-a-Judge) · 用另一个大语言模型代替人工,来判断AI答案是否正确的评测方法

论文原文摘要（英文）

The typical training process of a multimodal large language model (MLLM) involves adapting both the language model backbone and the projector between the backbone and a modality-specific encoder. We ask whether fine-tuning the backbone of an MLLM is necessary to adapt it to a new modality. Through experiments on 3D MLLMs, we find that training only the projector is sufficient to achieve strong multimodal performance relative to existing baseline models and our jointly trained MLLMs with the same encoder and backbone. We also show that joint training leads to undesirable drift in existing capabilities of the language model, which projector-only training avoids by definition. Furthermore, projector-only training has approximately twice the training sample throughput of joint training. We validate our findings across different language model backbones via 3D classification and captioning benchmarks as well as standard benchmarks evaluating language, vision, and spatial reasoning capabilities.

作者 · Nyx Iskandar, Saathvik Selvan, Slater Victoroff

在 arXiv 阅读

METAL LAB 最新报道

图片来源： Nyx Iskandar et al., arXiv:2608.19726, CC BY 4.0

Projector Is All You Train

他们做了什么

为什么重要

本文术语

最新论文

METAL LAB 最新报道