每天早上一封邮件，把昨天的 AI 梳理好订阅邮件

METAL LAB

When Machines Speak: A Unified Generative Framework for Integrating Machine-Native Symbols into Pretrained Large Language Models

arXiv:2608.195292026-08-21

让预训练大语言模型把电影推荐编码、法律判例引用这类机器专用符号，和普通文字一起生成的统一框架

预训练大语言模型天生只认识自然语言词元，遇到推荐系统商品编码、图结构关系这类用离散机器符号表示的信息，以往只能硬翻译成文字，或者另建专用模型。本文提出UniLang，把这些机器符号当作新词元加入大语言模型的词表，用对比学习把它们的含义对齐到模型已有的语义空间，再让文字和机器符号在同一个自回归生成过程里一起被预测出来。在预测用户下一个想看/想买的物品，以及预测法律判例中会被引用的具体句子这两个结构完全不同的任务上，UniLang都超过了各自领域的强基线方法。

他们做了什么

  1. 问题所在：推荐系统的商品编码、图数据中的关系结构等信息用离散的机器专用符号表示，不在预训练大语言模型原有的词表范围内，模型难以直接生成这类符号。
  2. 构造编码：先用文本编码器把物品描述（如电影标题和类型）转成向量，再用RQ-VAE（残差量化变分自编码器）把向量压缩成一串离散数字码，构成该物品的Semantic ID。
  3. 融入词表：把1024个代表这些数字码的新词元加入大语言模型词表，并用InfoNCE对比学习让每个符号的向量提前对齐到该物品对应文字描述的向量，使符号获得模型能理解的语义。
  4. 联合微调：将对齐好的符号词元与原有文字词元合并成一张词表，在Llama-3.2-1B-Instruct模型上用LoRA微调，让模型按统一的自回归方式生成文字与符号交织的序列。
  5. 实验结果：在电影/商品下一项推荐任务上，相比最强基线NDCG@5最高提升151.7%（MovieLens-20M数据集）；在法律判例引用预测任务上，相比最强基线Recall@1最高提升49.4%（10k数据集）；消融实验还表明，如果不做对比学习预对齐，模型训练会直接失败。
When Machines Speak: A Unified Generative Framework for Integrating Machine-Native Symbols into Pretrained Large Language Models figure 0
Figure 1: Example of the sequential prediction task.
Figure 1: Example of the sequential prediction task.
Table 2: Run-to-run variability on MovieLens-20M (mean ± SE).
metricmean ± standard error
Recall@50.1908 ± 0.00016
NDCG@50.1378 ± 0.00017
Recall@100.2596 ± 0.00014
NDCG@100.1600 ± 0.00014
Figure 2: Example of the legal precedent prediction task. Given context from the citing opinion (Garcia-Giraldo v. United States), predict the quotation sentence(s) or paragraph(s) from the cited opinion (United States v. Broce), which is unknown at inference time.
Figure 2: Example of the legal precedent prediction task. Given context from the citing opinion (Garcia-Giraldo v. United States), predict the quotation sentence(s) or paragraph(s) from the cited opinion (United States v. Broce), which is unknown at inference time.
When Machines Speak: A Unified Generative Framework for Integrating Machine-Native Symbols into Pretrained Large Language Models figure 3
Table 4: Statistics of the sequential recommendation datasets.
Dataset#users#items#actionsAvg. lengthDensity
Beauty40,22654,5420.35m8.80.02%
ML-1m6,0403,4161m163.54.79%
ML-20m138,49326,74420m144.40.54%
When Machines Speak: A Unified Generative Framework for Integrating Machine-Native Symbols into Pretrained Large Language Models figure 4
When Machines Speak: A Unified Generative Framework for Integrating Machine-Native Symbols into Pretrained Large Language Models figure 5
Table 6: Summary statistics of legal precedent dataset text features
FeatureMeanStdMinMax
Length of cited text (chars)3062252418,342
Length of citing context (chars)562216514,062
Figure 4: Ablation test on MovieLens-20m.
Figure 4: Ablation test on MovieLens-20m.
When Machines Speak: A Unified Generative Framework for Integrating Machine-Native Symbols into Pretrained Large Language Models figure 7
Table 7: Validation subset sizes for model selection
DatasetValidation sizeSample size
Beauty40,2265,000
MovieLens-1m6,0401,000
MovieLens-20m138,4931,000
10k103,8121,000
20k134,7371,000
50k190,0511,000
When Machines Speak: A Unified Generative Framework for Integrating Machine-Native Symbols into Pretrained Large Language Models figure 8
When Machines Speak: A Unified Generative Framework for Integrating Machine-Native Symbols into Pretrained Large Language Models figure 9
Table 8: RQ-VAE hyperparameters.
ML-1MLePaRD
HyperparameterBeauty
ML-20M10k / 20k / 50k
Figure 6: RQ-VAE training progress on different datasets.
Figure 6: RQ-VAE training progress on different datasets.
When Machines Speak: A Unified Generative Framework for Integrating Machine-Native Symbols into Pretrained Large Language Models figure 11
Table 9: Machine token alignment hyperparameters.
ML-1MLePaRD
DatasetBeauty
ML-20M10k / 20k / 50k
Table 10: SFT hyperparameters.
ML-1MLePaRD
DatasetBeauty
ML-20M10k / 20k / 50k

为什么重要

这为推荐系统、医疗编码、图结构数据等大量使用机器符号而非自然语言表示信息的实际系统，提供了一条直接借用预训练大语言模型语言知识和世界知识的路径，不必再硬翻译成文字或为每个任务单独建模型。同一套框架能处理两种结构完全不同的预测问题，说明这一思路有望推广到更多符号化领域。

本文术语

  • RQ-VAE（残差量化变分自编码器） · 通过多级量化把连续向量压缩成一串离散数字码的自编码器方法
  • Semantic ID · 用一串离散数字码表示单个物品的机器专用标识符
  • InfoNCE对比学习 · 让匹配的表示彼此靠近、不匹配的表示彼此远离，从而对齐两个向量空间的训练方法
  • LoRA（低秩适配） · 不重新训练整个模型，只训练少量新增参数即可完成微调的轻量方法
  • 自回归生成 · 根据已经生成的词元依次预测下一个词元，逐步生成整段序列的方式

论文原文摘要（英文）

Many real-world AI systems represent entities, behaviors, and structured information using discrete machine-native symbols rather than natural language. While these representations are compact and preserve task-relevant structure, they lie outside the linguistic token space of pretrained large language models (LLMs), creating a fundamental divide between language modeling and structured prediction. We introduce UniLang, a unified generative framework that bridges this divide by extending pretrained LLMs to treat machine-native symbols as first-class generative units alongside natural-language tokens. UniLang expands the LLM's vocabulary and embedding space with grounded machine-native representations, enabling textual and symbolic tokens to be jointly modeled and generated under a single autoregressive objective. This unified interface allows pretrained LLMs to directly operate on machine-native representations without requiring them to be verbalized as natural language or relying on task-specific architectures. We evaluate UniLang on two structurally distinct tasks, sequential recommendation and legal precedent prediction, spanning different domains and types of structured prediction. Across both tasks, UniLang consistently outperforms strong baselines, demonstrating a path toward extending pretrained LLMs beyond language and using them as a common generative modeling backbone for heterogeneous machine-native representations.

作者 · Su Yan, Rakesh Iyer

在 arXiv 阅读

最新论文

全部论文 →

METAL LAB 最新报道

图片来源： Su Yan et al., arXiv:2608.19529, CC BY 4.0