工作日早上 7 点读 AI，周日早上 8 点读周报订阅邮件

METAL LAB

Hunyuan3D-Buffalo 1.0: A Unified Multimodal Model for Scalable 3D Generation, Understanding, and Editing

arXiv:2608.027112026-08-02

腾讯混元团队推出Buffalo 1.0,把3D理解、生成和编辑统一到一个模型里

Hunyuan3D-Buffalo 1.0用一个模型同时完成理解3D物体、根据文本生成3D物体、按指令编辑3D物体、以及按文本要求生成局部部件这四件事。为此团队自建了一套自动化数据流程,构造出总量8700万条的3D多模态训练语料,并用名为Nano3D-v2的自动化流程专门解决3D编辑数据稀缺的问题。实验显示该模型在文本生成3D和3D编辑基准上超越了此前方法,并发现生成能力变强会连带提升编辑能力。

METAL LAB 解读图

Hunyuan3D-Buffalo 1.0 的结构流程

证据状态已报告实测结果

  1. 数据引擎全自动流程构建2500万理解样本、5000万文本到3D配对、1200万编辑配对,总计8700万条
  2. Nano3D-v2五阶段:视角选择、编辑区域预测、体素编辑、几何纹理精修、VLM校验,自动生成编辑训练配对
  3. Hunyuan3D-VLM读取3D点云,理解物体的语义、结构与部件位置,为生成提供条件信息
  4. Hunyuan3D DiT接收VLM提供的条件与源物体表征,以扩散方式生成或编辑3D形状
  5. 基准验证在Edit3D-Bench上相比Steer3D,CD降低86.7%,F1提升2.39倍
这是 METAL LAB 制作的解读图，并非论文作者提供的原图。

他们做了什么

  1. 团队用全自动流程构建了总计8700万条的3D多模态训练语料,包含2500万条理解样本、5000万条文本到3D配对、1200万条编辑配对。
  2. 3D视觉语言模型Hunyuan3D-VLM负责理解物体的语义、结构和空间位置信息,并把这些信息转成条件提供给扩散模型Hunyuan3D DiT,由后者真正生成3D形状。
  3. 为解决编辑数据稀缺问题,团队设计了基于智能体的Nano3D-v2流程,分五个阶段:选取最佳编辑视角、用学习到的模型在体素级别定位编辑区域、进行体素编辑、精细几何与纹理修补、以及基于视觉语言模型的质量校验。
  4. 在Edit3D-Bench基准上,相比此前最强基线Steer3D,该模型把形状误差指标Chamfer Distance从0.0684降到0.0091(相对降低86.7%),平均F1指标从0.2729提升到0.6515(提升2.39倍)。
  5. 仅给文本到3D训练数据额外加入1000条鸡头相关样本、完全不添加任何编辑数据,模型就自动获得了替换该部位的编辑能力,说明扩大生成数据也能带动编辑能力提升。
Hunyuan3D-Buffalo 1.0: A Unified Multimodal Model for Scalable 3D Generation, Understanding, and Editing figure 0
Table 1: Overview of the full training corpus across 3D understanding, text-to-3D generation, and instruction-guided 3D editing.
CapabilitySubset#Samples
3D understandingText / image / 3D instruction data∼25M
Text-to-3DText–asset pairs∼50M
3D editingHuman edits∼7M
Object edits∼3M
Part generation∼2M
Subtotal∼12M
Figure 1: Hunyuan3D-Buffalo 1.0 is an unified 3D multimodal framework that combines autoregressive modeling with diffusion-based generation, enabling 3D understanding, text-to-3D generation, 3D editing, and text-grounded part generation within a single architecture.
Figure 1: Hunyuan3D-Buffalo 1.0 is an unified 3D multimodal framework that combines autoregressive modeling with diffusion-based generation, enabling 3D understanding, text-to-3D generation, 3D editing, and text-grounded part generation within a single architecture.
Table 2: Six caption tiers produced per asset.
TierLengthEmphasis
Detailed4–6 sentences (≤120 w)subject, parts, pose, features
Main24–30 tokensstructure + key parts
Simplified15–20 tokensmain parts and fit
Paraphrase15–20 tokensreworded Simplified
Short6–10 tokenssubject + ≤1 attribute
Tags≤8 keywordsdisentangled keywords
Figure 2: Pipeline of constructing text-to-3D training corpus.
Figure 2: Pipeline of constructing text-to-3D training corpus.
Table 3: Comparison of part-level question answering and object-level captioning on UniPart-Bench [103]. The two tasks assess complementary aspects of 3D understanding, including localized part-aware reasoning and holistic object-level semantic description.
ModelPart Understanding Q&AOverall 3D Object Captioning
SBERTSimCSEBLEU-1ROUGE-LMETEORSBERTSimCSEBLEU-1ROUGE-LMETEOR
GPT4Point [63]48.3245.1715.1622.5516.1925.6027.0011.5012.0012.70
PointLLM-7B [93]61.3058.4821.7829.2622.4542.7942.4411.5814.3916.90
PointLLM-13B [93]56.3651.4721.4029.1621.8043.5143.1213.5415.7417.45
ShapeLLM-13B [62]61.1957.2623.3232.5624.4525.1527.1411.7712.1412.84
ShapeLLM-Omni-7B [103]57.3551.1622.7729.5723.2431.1831.9317.7919.0414.30
Part-X-MLLM [78]78.9884.2540.5442.2634.2453.8251.9736.0438.1130.71
UniVerse3D [101]83.1187.1646.7943.9442.0565.1866.2542.7544.1741.11
Hunyuan3D-VLM (Ours)85.4789.0649.9545.0145.7972.9473.6050.9352.8450.47
Figure 3: Pipeline of constructing 3D editing training corpus (Nano3D-v2).
Figure 3: Pipeline of constructing 3D editing training corpus (Nano3D-v2).
Table 4: Detailed all-task evaluation of Hunyuan3D-VLM on UniPart-Bench [103]. The benchmark covers pure box listing, multi-part grounding, single-part grounding, box-to-text generation, and part-level question answering.
TaskNameIoUSBERTSimCSEBLEU-1ROUGE-LMETEOR
0Pure box listing0.864-----
1Multi-Part Grounding (Q1)0.88068.0068.5552.0652.0926.35
2Multi-Part Grounding (Q2)0.84470.9269.4740.0441.8638.19
3Single-Part Grounding (Q1)0.62678.9577.9245.7447.4744.07
4Single-Part Grounding (Q2)0.525-----
5Box-to-Text (Q1)-67.6468.2749.8950.0025.45
6Box-to-Text (Q2)-74.1372.9942.0144.2840.96
7Part QA0.63385.4789.0649.9545.0145.79
Figure 4: Examples of editing pairs in the training corpus created by Nano3D-v2.
Figure 4: Examples of editing pairs in the training corpus created by Nano3D-v2.
Table 5: Results of the user study for text-to-3D generation. We report the preference rate (%), i.e., the percentage of comparisons in which each model is selected as the best among the four candidates. In each comparison, the four results are shown side by side in randomized order, and participants pick the single best result for each criterion (a “tie” option is also allowed, so columns may sum to slightly below 100%). Higher is better, with a random-choice baseline of 25%. Best results are shown in bold.
ModelText alignmentGeometry qualityOverall preference
Universe3D [101]8.27.48.3
TRELLIS [92]14.912.414.4
Omni123 [100]17.521.018.4
Hunyuan3D-Buffalo 1.0 (Ours)55.257.156.6
Figure 5: Examples of multi-round editing by Nano3D-v2.
Figure 5: Examples of multi-round editing by Nano3D-v2.
Table 6: Ablation on the quantity of training data. We report the preference rate (%), i.e., the percentage of comparisons in which a model is chosen as the best among the three variants. In each comparison, the three results are shown side by side in randomized order, and participants select the best result for each criterion (a “tie” option is also allowed, so columns may sum to slightly below 100%). Higher is better, and the random-choice baseline is 33.3%. Best results are shown in bold.
Num. of samplesText alignmentGeometry qualityOverall preference
300w9.98.88.4
1500w28.829.028.6
5000w54.557.457.5
Figure 6: Hunyuan3D-Buffalo 1.0 pipeline. The framework unifies 3D QA and grounding, text-to-3D generation, and 3D editing through a shared Hunyuan3D-VLM backbone, which connects language, 3D representations, and generative Hunyuan3D DiT modules for multimodal understanding, generation, and editing.
Figure 6: Hunyuan3D-Buffalo 1.0 pipeline. The framework unifies 3D QA and grounding, text-to-3D generation, and 3D editing through a shared Hunyuan3D-VLM backbone, which connects language, 3D representations, and generative Hunyuan3D DiT modules for multimodal understanding, generation, and editing.
Table 7: Quantitative comparison of language-guided 3D shape editing on Edit3D-Bench [85]. Both CLIP-conditioned and 3D-VLM-conditioned variants of Hunyuan3D-Buffalo 1.0 are included to analyze the effect of stronger 3D instruction understanding.
MethodAddRemoveAvg
CD ↓F1 ↑CD ↓F1 ↑CD ↓F1 ↑
ShapeLLM-Omni [103]0.25460.08770.22370.11660.23920.1022
3DEditFormer [90]0.16760.19550.13420.18360.15090.1896
Tailor3D [64]0.16610.12170.17550.13520.17080.1285
Steer3D [85]0.14040.24140.09760.30440.11900.2729
Omni123 [100]0.07360.17430.06320.22590.06840.2001
Hunyuan3D-Buffalo 1.0 w/ CLIP (Ours)0.01540.56570.01620.70150.01580.6336
Hunyuan3D-Buffalo 1.0 w/ 3D-VLM (Ours)0.01270.56100.00540.74200.00910.6515
Figure 8: Qualitative text to 3D results.
Figure 8: Qualitative text to 3D results.

研究结果

  • 在Edit3D-Bench上,采用3D-VLM条件的Hunyuan3D-Buffalo 1.0相比最强基线Steer3D,把平均Chamfer Distance从0.0684降至0.0091(相对降低86.7%),平均F1从0.2729提升至0.6515(提升2.39倍)。
  • 在添加与删除两类编辑任务中,该模型分别取得CD/F1为0.0127/0.5610和0.0054/0.7420,说明它既能精准局部编辑,又能较好保留其余几何结构。
  • 把条件输入从CLIP嵌入换成3D-VLM后,平均CD从0.0158降到0.0091,平均F1从0.6336提升到0.6515,证明更强的3D理解能力能提升编辑效果。
  • 仅给文本到3D数据额外加入1000条鸡头相关样本,完全不加任何编辑数据,模型就获得了编辑该部位的新能力。
  • 在定性比较中,Hunyuan3D-Buffalo 1.0在遵循编辑指令与保留原始形状之间的平衡表现优于对比模型Omni123和Steer3D。
Figure 9: Qualitative shape editing results. Our method significantly outperforms all baselines in both geometric consistency before and after editing, and responsiveness to editing instructions.
Figure 9: Qualitative shape editing results. Our method significantly outperforms all baselines in both geometric consistency before and after editing, and responsiveness to editing instructions.

可应用场景

  • 游戏与动画资产制作中,通过文本指令对3D角色或道具进行局部修改(如添加眼镜、去除翅膀)的工具
  • 将3D物体按部件拆解再重新组合、构建可复用3D资产库的流程
  • 针对3D资产提供问答或部件定位(grounding)功能的3D检索与标注工具
Figure 10: Scaling up text-to-3d data facilitates 3D editing. Model A is our base model; when instructed to edit an object by replacing its head with a chicken head, it fails to produce a satisfactory result. Model B is built upon Model A by adding only 1,000 additional chicken samples for the text-to-3D task during the Omni pre-training stage—crucially, without introducing any new editing data. After incorporating this text-to-3D data, the model can successfully replace the head with a chicken head. This suggests a clear direction: to improve 3D editing, the text-to-3D generation capability should be maximized as much as possible. Since constructing text-to-3D data is far less costly than constructing 3D editing data, scaling up text-to-3D data is a relatively more feasible path toward stronger 3D editing.
Figure 10: Scaling up text-to-3d data facilitates 3D editing. Model A is our base model; when instructed to edit an object by replacing its head with a chicken head, it fails to produce a satisfactory result. Model B is built upon Model A by adding only 1,000 additional chicken samples for the text-to-3D task during the Omni pre-training stage—crucially, without introducing any new editing data. After incorporating this text-to-3D data, the model can successfully replace the head with a chicken head. This suggests a clear direction: to improve 3D editing, the text-to-3D generation capability should be maximized as much as possible. Since constructing text-to-3D data is far less costly than constructing 3D editing data, scaling up text-to-3D data is a relatively more feasible path toward stronger 3D editing.

局限与待验证事项

  • 目前模型主要聚焦几何编辑,尚未涉及纹理编辑,包含颜色、材质变化的完整编辑仍是未来工作。
  • 编辑数据构建流程难以保证编辑框内部非编辑区域的一致性,这种不一致会作为噪声混入训练过程。
  • 文本到3D数据的描述仍依赖Gemini等多模态语言模型生成,这些描述存在一定模糊性,给训练数据带来噪声。
  • 基于TRELLIS式的多阶段扩散架构在追求高质量几何时需要多阶段编辑,这从根本上限制了可扩展性。
  • 作者自己指出目前数据的规模和质量都还未达到理想水平,后续仍需要进一步扩大数据规模。
Figure 11: Qualitative part generation results.
Figure 11: Qualitative part generation results.

为什么重要

过去3D的理解、生成、编辑模型大多各自独立开发,这项工作用实测结果证明把三者放进同一个模型能让能力互相促进。对于做游戏资产、动画或产品设计工具的人来说,这提供了一条不必分别维护多套流程、就能同时具备理解、生成与编辑能力的路径。

Figure 12: Qualitative shape editing results.
Figure 12: Qualitative shape editing results.

本文术语

  • Nano3D-v2 · 一套基于智能体的自动化流程,输入源3D物体和编辑指令后自动生成编辑后的3D物体,用于构建训练数据
  • Hunyuan3D-VLM · 一个专门处理3D点云数据的视觉语言模型,能理解物体的含义、结构和各部件所在位置
  • DiT(扩散Transformer) · 一种从随机噪声出发、逐步生成目标结果(这里是3D形状)的扩散式生成模型
  • Chamfer Distance · 衡量两个3D形状表面平均距离的指标,数值越小说明编辑结果在几何上越接近原本目标
  • 体素(voxel) · 把3D空间划分成的网格最小单元,相当于3D版本的像素

论文原文摘要（英文）

Recent advances in image generation have demonstrated the potential of unified multimodal models that integrate understanding, generation, and editing. However, unified 3D modeling remains constrained by scarce multimodal data, particularly the lack of large-scale and geometrically consistent editing data. To address this limitation, we propose Hunyuan3D-Buffalo 1.0, a unified framework supporting 3D understanding, text-to-3D generation, instruction-guided 3D editing, and text-grounded part gene

作者 · Junliang Ye

在 arXiv 阅读

最新论文

全部论文 →

METAL LAB 最新报道

图片来源： Junliang Ye et al., arXiv:2608.02711, arxiv-nonexclusive