When Machines Speak: A Unified Generative Framework for Integrating Machine-Native Symbols into Pretrained Large Language Models
arXiv:2608.195292026-08-21
A framework that lets pretrained LLMs generate machine-only codes -- like recommendation item IDs or legal citation markers -- right alongside plain text
Pretrained LLMs can only natively handle natural-language tokens, so systems that represent information as discrete machine codes (recommendation item IDs, graph structures, etc.) have had to either translate everything into text or rely on separate task-specific models. This paper introduces UniLang, which adds these machine codes as new vocabulary tokens to a pretrained LLM, aligns their meaning to the LLM's embedding space via contrastive learning, and then generates text and machine codes together under one autoregressive objective. Tested on two very different problems -- predicting a user's next movie/product and predicting which legal precedent sentence gets quoted -- UniLang beat strong specialized baselines on both.
What they did
The problem: information like recommendation item codes or graph relations is represented as discrete machine-native symbols rather than natural language, so it sits outside a pretrained LLM's normal token vocabulary and is hard for the LLM to generate directly.
Building the codes: item descriptions (e.g., a movie's title and genre) are embedded with a text encoder and then compressed by an RQ-VAE (Residual Quantized VAE) into a short sequence of discrete codes, called a Semantic ID (SID), representing each item.
Grounding into the LLM: 1,024 new tokens for these SIDs are added to the LLM's vocabulary, and their embeddings are pre-trained with an InfoNCE contrastive loss so a code's embedding lines up with the embedding of its matching text description -- giving the raw codes real meaning inside the LLM's space.
Joint fine-tuning: the aligned code tokens and the LLM's original text tokens are merged into one vocabulary, and a Llama-3.2-1B-Instruct model is fine-tuned with LoRA to autoregressively generate mixed text-and-code sequences for each task.
Results: on movie/product next-item recommendation, UniLang improved NDCG@5 by up to 151.7% over the strongest baseline (MovieLens-20M); on legal precedent prediction it improved Recall@1 by up to 49.4% over the strongest baseline (10k dataset); ablations showed training fails entirely without the contrastive pre-alignment step.
Figure 1: Example of the sequential prediction task.
Table 2: Run-to-run variability on MovieLens-20M (mean ± SE).
metric
mean ± standard error
Recall@5
0.1908 ± 0.00016
NDCG@5
0.1378 ± 0.00017
Recall@10
0.2596 ± 0.00014
NDCG@10
0.1600 ± 0.00014
Figure 2: Example of the legal precedent prediction task. Given context from the citing opinion (Garcia-Giraldo v. United States), predict the quotation sentence(s) or paragraph(s) from the cited opinion (United States v. Broce), which is unknown at inference time.
Table 4: Statistics of the sequential recommendation datasets.
Dataset
#users
#items
#actions
Avg. length
Density
Beauty
40,226
54,542
0.35m
8.8
0.02%
ML-1m
6,040
3,416
1m
163.5
4.79%
ML-20m
138,493
26,744
20m
144.4
0.54%
Table 6: Summary statistics of legal precedent dataset text features
Feature
Mean
Std
Min
Max
Length of cited text (chars)
306
225
24
18,342
Length of citing context (chars)
562
216
5
14,062
Figure 4: Ablation test on MovieLens-20m.
Table 7: Validation subset sizes for model selection
Dataset
Validation size
Sample size
Beauty
40,226
5,000
MovieLens-1m
6,040
1,000
MovieLens-20m
138,493
1,000
10k
103,812
1,000
20k
134,737
1,000
50k
190,051
1,000
Table 8: RQ-VAE hyperparameters.
ML-1M
LePaRD
Hyperparameter
Beauty
ML-20M
10k / 20k / 50k
Figure 6: RQ-VAE training progress on different datasets.
Table 9: Machine token alignment hyperparameters.
ML-1M
LePaRD
Dataset
Beauty
ML-20M
10k / 20k / 50k
Table 10: SFT hyperparameters.
ML-1M
LePaRD
Dataset
Beauty
ML-20M
10k / 20k / 50k
Why it matters
It offers a practical path for real systems -- recommenders, medical coding, graph-structured data -- that store information as machine codes to directly tap into a pretrained LLM's language and world knowledge, instead of clumsily converting everything to text or building a separate model per task. One framework handling two structurally unrelated prediction problems suggests broader reuse potential across other symbolic domains.
Terms in this paper
RQ-VAE (Residual Quantized VAE) · An autoencoder that compresses continuous vectors into a short sequence of discrete codes through multiple quantization stages
Semantic ID (SID) · A machine-native identifier representing one item as a short sequence of discrete codes
InfoNCE contrastive loss · A training objective that pulls matching representations together and pushes non-matching ones apart to align two embedding spaces
LoRA (Low-Rank Adaptation) · A lightweight fine-tuning method that trains a small set of added parameters instead of the whole model
autoregressive generation · Generating a sequence one token at a time, each based on the tokens produced so far
Original abstract (English)
Many real-world AI systems represent entities, behaviors, and structured information using discrete machine-native symbols rather than natural language. While these representations are compact and preserve task-relevant structure, they lie outside the linguistic token space of pretrained large language models (LLMs), creating a fundamental divide between language modeling and structured prediction. We introduce UniLang, a unified generative framework that bridges this divide by extending pretrained LLMs to treat machine-native symbols as first-class generative units alongside natural-language tokens. UniLang expands the LLM's vocabulary and embedding space with grounded machine-native representations, enabling textual and symbolic tokens to be jointly modeled and generated under a single autoregressive objective. This unified interface allows pretrained LLMs to directly operate on machine-native representations without requiring them to be verbalized as natural language or relying on task-specific architectures. We evaluate UniLang on two structurally distinct tasks, sequential recommendation and legal precedent prediction, spanning different domains and types of structured prediction. Across both tasks, UniLang consistently outperforms strong baselines, demonstrating a path toward extending pretrained LLMs beyond language and using them as a common generative modeling backbone for heterogeneous machine-native representations.
