You don't need to retrain the whole language model to teach it a new sense like 3D
Multimodal AI models that understand things like images, sound, or 3D shapes usually get trained by updating both the core language model and a small connector piece, called a projector, that links it to the new data type. This paper shows that training only the projector, while keeping the language model completely frozen, achieves 3D understanding performance comparable to or better than models where both parts are trained together. On top of that, projector-only training runs about twice as fast and never damages the language model's pre-existing abilities, unlike joint training which does.
What they did
Using 3D point cloud models (built on the PointLLM setup) as a testbed, the researchers compared training only the projector (with the language model backbone completely frozen) against jointly training the projector plus LoRA adapters (small add-on parameters) inside the backbone.
They tested three different language model backbones, Qwen3.5-4B, Qwen3.5-9B, and Llama-3.1-8B-Instruct, training each for a fixed 16 hours on a single A100 GPU for a fair comparison.
On 3D object classification and captioning benchmarks using ModelNet40, Objaverse, and OmniObject3D datasets, projector-only models generally scored higher than the existing PointLLM baseline models and stayed competitive with the jointly trained models under the same compute budget.
Projector-only training processed roughly twice as many training samples per hour as joint training.
Joint training caused measurable drift, meaning the language model's original language, vision, and spatial reasoning abilities got worse, while projector-only training avoided this entirely simply because the backbone was never touched.
Figure 1: Architecture of 3D MLLMs in this paper. This figure depicts projector-only training, where all parameters are frozen except those of the projector. An alternative training regime is joint training, where all parameters are frozen except those of the projector and LoRA adapters of the LLM backbone.
Table 1: Configurations of MLLM variants. All variants are trained for the full 16 GPU hours. We save checkpoints at 2, 4, 8, 12, and 16 hours.
ID
LM Backbone
Projector Trained
LoRA Fine-tuned
P-Llama8B
Llama-3.1-8B-Instruct
Yes
No
J-Llama8B
Llama-3.1-8B-Instruct
Yes
Yes
P-Qwen4B
Qwen3.5-4B
Yes
No
J-Qwen4B
Qwen3.5-4B
Yes
Yes
P-Qwen9B
Qwen3.5-9B
Yes
No
J-Qwen9B
Qwen3.5-9B
Yes
Yes
Figure 2: Evaluation results of projector-only-trained MLLMs against baseline PointLLM models. Chart (a) shows generative 3D object classification results on ModelNet40 (M40.), Objaverse (Obj.), and OmniObject3D (Omni.) objects under a zero-shot setting. There are two prompt types per benchmark: an instruction-style prompt (I, “What is this?”) and a completion-style prompt (C, “This is an object of”). Each entry reports accuracy judged by GPT-5.6 Luna [28]. Chart (b) shows the aggregate precision score on 3D object captioning tasks under the three judge LLMs. More details are found in Appendix B.
Table 2: Generative 3D object classification results on ModelNet40 (M40.), Objaverse (Obj.), and OmniObject3D (Omni.) under a zero-shot setting as reported in Figure 2 (a).
Model
M40. (I)
M40. (C)
Obj. (I)
Obj. (C)
Omni. (I)
Omni. (C)
PointLLM-7B [43]
54.94
55.55
59.83
60.43
34.78
35.70
PointLLM-13B [43]
57.66
56.81
61.73
60.90
34.63
34.01
P-Llama8B
61.95
63.57
65.60
62.33
49.19
40.69
P-Qwen4B
59.56
59.56
60.90
60.83
41.07
40.25
P-Qwen9B
59.28
62.80
74.07
67.77
48.33
38.76
Mean
58.68
59.66
64.43
62.45
41.60
37.88
Figure 3: M40. (top) and Obj. (bottom) mean accuracy against GPU hours. Projector-only training generally scores higher than joint training across all GPU hours.
Table 3: 3D object captioning results on Objaverse as reported in Figure 2 (b). C refers to correctness, H to hallucination, and P to precision (aggregate of C and H).
GPT-5.6 Luna
Claude Haiku 4.5
Gemini 3.5 Flash-Lite
Model
C
H ↓
P
C
H ↓
P
C
H ↓
P
PointLLM-7B [43]
4.26
2.12
66.75
3.37
1.95
63.30
2.75
1.99
58.08
PointLLM-13B [43]
4.35
2.10
67.43
3.47
1.91
64.46
2.82
1.92
59.49
P-Llama8B
5.61
2.17
72.12
3.76
1.75
68.19
3.29
1.87
63.75
P-Qwen4B
6.41
1.83
77.81
4.28
1.57
73.11
3.72
1.65
69.22
P-Qwen9B
6.58
1.95
77.19
4.30
1.68
71.90
3.76
1.71
68.72
Mean
–
–
72.26
–
–
68.19
–
–
63.85
Figure 4: M40. (top) and Obj. (bottom) mean accuracy against training samples. Plotted points in these charts correspond to the same GPU-hour thresholds that in turn correspond to the plotted points in Figure 3. Projector-only training sees more samples than joint training at each GPU-hour threshold as it is approximately twice as fast.
Table 4: Results for language, vision, and spatial reasoning benchmarks. The LoRA fine-tuned backbones are compared against the corresponding base backbone.
Llama-3.1-8B-Instruct
Qwen3.5-4B
Qwen3.5-9B
Benchmark
P
J
P
J
P
J
Language
MMLU-Pro [39]
37.32
11.21↓
45.04
45.48
51.38
51.30
MMLU-Redux [15]
60.37
22.83↓
69.13
70.53
74.57
74.27
GPQA Diamond [35]
33.33
24.24↓
33.33
41.41↑
45.96
49.49↑
IFEval [51]
72.46
10.35↓
80.41
38.82↓
83.55
67.84↓
IFBench [29]
26.33
16.33↓
29.33
17.67↓
32.67
28.33↓
GSM8K [8]
86.96
0.61↓
90.90
81.96↓
92.34
92.34
WinoGrande [36]
61.40
49.57↓
65.59
68.35↑
74.19
73.24
OpenBookQA [26]
81.60
27.60↓
86.20
86.90
90.30
92.20↑
HumanEval [7]
64.02
0.00↓
82.93
71.95↓
84.76
80.49↓
Vision
MMMU [47]
–
–
49.44
54.45↑
54.23
59.02↑
MMMU-Pro [48]
–
–
32.60
37.23↑
42.72
43.41
MMMU-Pro Vision [48]
–
–
31.45
34.16↑
40.46
40.64
MMStar [6]
–
–
53.27
61.67↑
65.87
65.67
BabyVision [5]
–
–
19.07
14.43↓
16.49
15.98
RealWorldQA [42]
–
–
74.12
69.54↓
75.42
76.08
Spatial
ERQA [16]
–
–
45.25
42.25↓
45.75
44.00↓
EmbSpatialBench [14]
–
–
75.14
75.16
76.59
76.95
RefSpatialBench [50]
–
–
20.94
1.81↓
38.27
27.80↓
LingoQA [25]
–
–
70.40
58.80↓
75.00
68.40↓
Table 5: Projector and LoRA-adapter parameter counts, and projector input/output shapes, for each backbone.
Backbone
Encoder dim c
Tokens m
Backbone dim c′
Proj. params
LoRA params
Llama-3.1-8B-Instruct
384
513
4096
10.89M
41.94M
Qwen3.5-4B
384
513
2560
7.74M
32.46M
Qwen3.5-9B
384
513
4096
10.89M
43.28M
Table 6: The set of system message paraphrases, drawn uniformly at random during training.
System message
Answer the question or follow the instruction regarding the given 3D object.
Answer the following question or carry out the instruction about the provided 3D object.
Given a 3D object, respond to the question or instruction about it.
Consider the 3D object and answer the question or follow the instruction that follows.
Respond to the question or instruction concerning the presented 3D object.
Using the given 3D object, answer the question or complete the instruction.
You are given a 3D object; answer the question or follow the instruction about it.
Examine the 3D object and answer the accompanying question or instruction.
Provide an answer to the question or complete the instruction about the given 3D object.
Based on the 3D object shown, answer the question or follow the instruction.
Address the question or instruction about the provided 3D object.
Table 7: Training and evaluation hyperparameters, shared across every configuration except the learning rates.
Hyperparameter
Value
Optimizer
AdamW, β=(0.9,0.999), ϵ=10−8
Learning-rate schedule
constant, no warmup
Projector learning rate
2×10−3 (1×10−3 for Llama8B)
Weight decay
0.0
Gradient clipping
global norm 1.0
Micro-batch / accumulation / effective
12 / 2 / 24
Max response length
512 tokens
Mixed precision
bfloat16 autocast
Random seed
0
Hardware
1× NVIDIA A100-80GB
Gradient checkpointing
enabled
LoRA rank / α / dropout
16 / 32 / 0.05
LoRA target modules
all-linear
LoRA learning rate
2×10−5 (2×10−4 for Llama8B)
Table 8: Total number of training samples seen by each MLLM variant, average throughput, and average step rate after 16 GPU hours.
ID
Total samples
Avg. throughput (h-1)
Avg. step rate (h-1)
P-Llama8B
432,936
27,056
1,127
J-Llama8B
200,640
12,537
522
P-Qwen4B
480,768
30,045
1,252
J-Qwen4B
245,592
15,346
639
P-Qwen9B
325,176
20,321
847
J-Qwen9B
174,720
10,918
455
Table 9: Generative 3D object classification results on ModelNet40 (M40.), Objaverse (Obj.), and OmniObject3D (Omni.) under a zero-shot setting. Extension of Table 2.
Model
M40. (I)
M40. (C)
Obj. (I)
Obj. (C)
Omni. (I)
Omni. (C)
ShapeLLM-7B [32]
18.76
17.95
30.23
31.37
15.28
18.75
ShapeLLM-13B [32]
22.45
21.60
40.67
38.90
28.66
33.33
PointLLM-7B [43]
54.94
55.55
59.83
60.43
34.78
35.70
PointLLM-13B [43]
57.66
56.81
61.73
60.90
34.63
34.01
PointLLM-R [3]
62.28
62.64
65.23
65.40
38.81
38.69
MiniGPT-3D [37]
63.41
62.84
63.97
63.00
41.10
39.27
P-Llama8B
61.95
63.57
65.60
62.33
49.19
40.69
J-Llama8B
49.72
57.37
66.80
59.53
42.34
37.17
J-NoLoRA-Llama8B
36.02
40.24
58.90
51.07
39.62
40.69
P-Qwen4B
59.56
59.56
60.90
60.83
41.07
40.25
J-Qwen4B
58.27
59.60
71.00
68.80
48.17
47.32
J-NoLoRA-Qwen4B
54.94
53.24
58.73
57.97
27.08
27.49
P-Qwen9B
59.28
62.80
74.07
67.77
48.33
38.76
J-Qwen9B
52.59
58.75
75.23
66.23
51.32
44.86
J-NoLoRA-Qwen9B
51.62
52.63
49.13
56.93
24.51
28.74
Curriculum-Qwen4B
53.48
54.34
74.80
75.63
46.08
46.73
Table 10: 3D object captioning results on Objaverse. Extension of Table 3.
GPT-5.6 Luna
Claude Haiku 4.5
Gemini 3.5 Flash-Lite
Model
C
H ↓
P
C
H ↓
P
C
H ↓
P
ShapeLLM-7B [32]
2.23
1.52
59.47
1.65
1.46
53.15
1.03
1.69
37.85
ShapeLLM-13B [32]
2.68
1.84
59.32
2.04
1.68
54.93
1.39
2.02
40.81
PointLLM-7B [43]
4.26
2.12
66.75
3.37
1.95
63.30
2.75
1.99
58.08
PointLLM-13B [43]
4.35
2.10
67.43
3.47
1.91
64.46
2.82
1.92
59.49
PointLLM-R [3]
3.38
1.18
74.10
2.50
1.10
69.42
2.48
1.17
67.94
MiniGPT-3D [37]
4.51
2.01
69.16
3.19
1.93
62.32
2.70
1.95
58.07
P-Llama8B
5.61
2.17
72.12
3.76
1.75
68.19
3.29
1.87
63.75
J-Llama8B
6.09
2.35
72.15
4.01
1.95
67.28
3.41
2.11
61.79
J-NoLoRA-Llama8B
3.14
2.96
51.48
2.35
2.08
52.96
1.69
2.31
42.15
P-Qwen4B
6.41
1.83
77.81
4.28
1.57
73.11
3.72
1.65
69.22
J-Qwen4B
6.73
1.97
77.33
4.43
1.67
72.67
3.91
1.71
69.52
J-NoLoRA-Qwen4B
5.44
4.23
56.26
4.30
2.53
62.97
2.92
3.19
47.80
P-Qwen9B
6.58
1.95
77.19
4.30
1.68
71.90
3.76
1.71
68.72
J-Qwen9B
6.83
2.02
77.19
4.46
1.73
72.00
3.92
1.80
68.52
J-NoLoRA-Qwen9B
5.14
4.74
52.02
3.84
3.02
56.00
2.58
3.65
41.46
Curriculum-Qwen4B
6.52
1.95
76.96
4.36
1.67
72.33
3.76
1.74
68.38
Table 11: Generative zero-shot classification accuracy against wall-clock hours. This is calculated as the average of the I and C variants of the ModelNet40 and Objaverse benchmarks, directly comparable to Figure 3.
M40. (Avg.)
Obj. (Avg.)
Model
2 h
4 h
8 h
12 h
16 h
2 h
4 h
8 h
12 h
16 h
P-Qwen4B
54.23
54.76
58.33
63.53
59.56
58.10
62.52
67.25
64.90
60.87
J-Qwen4B
46.84
51.64
51.03
52.09
58.93
58.20
66.50
73.28
75.05
69.90
P-Qwen9B
51.38
53.99
57.03
61.06
61.04
57.47
63.07
67.80
67.35
70.92
J-Qwen9B
39.49
46.84
54.52
58.53
55.67
51.20
60.48
71.08
67.90
70.73
P-Llama8B
53.85
57.22
57.80
56.14
62.76
51.48
57.58
66.10
70.07
63.97
J-Llama8B
45.50
48.28
47.49
56.62
53.55
47.80
54.02
62.28
65.23
63.17
Table 12: Example generations on long captioning tasks. The MLLMs are given the prompt “Caption this 3D model in detail.” All prompts are independently provided to the MLLMs. The answers shown in this table were generated by P-Llama8B and J-Llama8B at 16 hours of training, as well as the two PointLLM models.
Object
PointLLM-7B
The 3D model represents a distinctive, green toy car that stands out due to its unconventional feature - a cornered driving unit. Designed for play, this model is aesthetically appealing with an eye-catching bright green color that would attract young children. The unique driving unit located at the corner provides an interesting twist to the traditional car design, promoting creative play and imaginative scenarios. However, despite its playful design, it is still a toy and should be handled with care.
The 3D object is a lively representation of a blue whale, which is the largest mammal on Earth. It is depicted in a playful cartoon style with an eye-catching cobalt-blue hue. The whale features proportional body parts, including a small dorsal fin, enormous mouth, and flat, wide head. It seems to be designed for entertainment or educational purposes, possibly for children, given its cartoon-like attributes and the significance of the whale in marine life.
PointLLM-13B
The 3D object model represents a small toy car, with the primary colour being a vibrant green. It features four black wheels that contrast well with the green body. Two red lights are situated at the front, possibly to mimic the headlights of an actual car. This model, being a toy, is typically used for children’s play, aiding in their motor skill development and imagination. The car’s simplistic design might be suitable for younger age groups.
This is a 3D model of a blue whale, the largest mammal on Earth, depicted in a cartoon style. The whale is distinguished by its small dorsal fin, enormous mouth, and flat, wide head. Despite the cartoon aesthetic, the model accurately represents the unique features of a blue whale. It could be used for educational purposes or in animated movies or documentaries.
P-Llama8B
The object is a green and white van with a boxy shape, featuring a prominent front grille, round headlights, and a rear window. It has a flat roof and a side door with a handle. The van has a distinctive green and white color scheme with black accents. The wheels are circular and black, and the vehicle has a distinctive front bumper.
The object is a sleek, streamlined shark with a pointed snout and a series of sharp, triangular fins along its back. The body is predominantly blue, with a lighter underside and a distinctive pattern of darker spots. The fins are a lighter shade, and the tail is long and narrow, tapering to a sharp point. The shark has a prominent dorsal fin and a smaller pectoral fin, with a series of gill slits visible along the sides.
J-Llama8B
The object is a green vehicle with a boxy design, featuring a large windshield, a rear window, and a front grille. It has a roof rack and a spare tire mounted on the rear. The vehicle appears to be a van, with a prominent front bumper and a rear section that includes a license plate. The wheels are circular, and the vehicle has a simple, utilitarian appearance.
The object is a stylized representation of a fish, characterized by a streamlined body with a pointed snout and a forked tail. The body is primarily blue with a white underbelly, and the fins are white. The fish has a smooth, abstract appearance with a minimalistic design.
Why it matters
For anyone building AI systems that need to understand new data types, this means you can add a new capability by training a small, cheap component instead of retraining the whole giant model, saving compute and time while keeping the original model's skills intact. It also points toward a modular approach where one frozen language model could be paired with many independently trained projectors for different modalities.
Terms in this paper
MLLM (multimodal large language model) · an AI model extended to understand not just text but other data types like images, 3D shapes, or audio
Projector · a small neural network component that translates data from another modality, like 3D points, into a form the language model can process
LoRA adapter · a lightweight add-on set of parameters used to fine-tune a large model efficiently without retraining all of it
Point cloud · a way of representing a 3D object's surface as a set of individual points, each with a position and sometimes color
LLM-as-a-Judge · an evaluation method where another large language model, instead of a human, scores whether an AI's answer is correct
Original abstract (English)
The typical training process of a multimodal large language model (MLLM) involves adapting both the language model backbone and the projector between the backbone and a modality-specific encoder. We ask whether fine-tuning the backbone of an MLLM is necessary to adapt it to a new modality. Through experiments on 3D MLLMs, we find that training only the projector is sufficient to achieve strong multimodal performance relative to existing baseline models and our jointly trained MLLMs with the same encoder and backbone. We also show that joint training leads to undesirable drift in existing capabilities of the language model, which projector-only training avoids by definition. Furthermore, projector-only training has approximately twice the training sample throughput of joint training. We validate our findings across different language model backbones via 3D classification and captioning benchmarks as well as standard benchmarks evaluating language, vision, and spatial reasoning capabilities.
![Figure 2: Evaluation results of projector-only-trained MLLMs against baseline PointLLM models. Chart (a) shows generative 3D object classification results on ModelNet40 (M40.), Objaverse (Obj.), and OmniObject3D (Omni.) objects under a zero-shot setting. There are two prompt types per benchmark: an instruction-style prompt (I, “What is this?”) and a completion-style prompt (C, “This is an object of”). Each entry reports accuracy judged by GPT-5.6 Luna [28]. Chart (b) shows the aggregate precision score on 3D object captioning tasks under the three judge LLMs. More details are found in Appendix B.](https://media.metallab.ai/papers/2608.19726/f1.png)
