Image: METAL
Summary
- Researchers from the Tencent Hunyuan team, Renmin University of China, and The Chinese University of Hong Kong released EvolveScaler, which builds data for information-evolution contexts, where later records revise or revoke earlier ones, by defining them in code first.
- With 117 task prototypes, 159 question operators, and five difficulty tiers, they produced about 35,100 training examples and 585 evaluation instances, and the median avg@5 of 14 models fell from 71.2 to 11.3.
- Continued training of an internal A3B model on 6,000 examples raised the average across eight external benchmarks by 5.25 points, and about half of the failures came from summing signed changes.
The Tencent Hunyuan team released the EvolveScaler paper and project page on X on September 15. The paper was written jointly by researchers at Renmin University of China's Gaoling School of Artificial Intelligence, Tencent's Hunyuan LLM department, and The Chinese University of Hong Kong; it went up on arXiv on September 8, with a second version on the 9th. It addresses one problem: whether a model can correctly reconstruct the currently valid state when, within a long conversation, later records revise or cancel earlier ones. The researchers call this information evolution, and instead of writing the data in natural language first, they define it in code and then render it as natural language to build training and evaluation data. Measuring 14 frontier and open-source models on that data, the median avg@5 fell to 11.3 on the hardest tier.
The Hunyuan team's X post explains the problem with a game log. "Read a 40-day RPG log. Now answer one question: if you skip the mini-boss on Day 7, do you still beat the final boss?" The answer is not in the log; you have to replay the world. Records get retracted, corrected, and backfilled, and the world keeps changing after you read it. The paper defines such a context not as a long document to read but as an event history to replay. Its starting point is that two conversations of the same length and the same number of turns differ sharply in difficulty if one merely accumulates valid information while the other repeatedly overturns earlier records.
The method is a reversal of the usual order. A human-authored operational specification comes first, with ten fields: entities, clock, admissible event schemas, tracked state variables, transition constraints, rules for which records are valid, state-preserving noise, difficulty controls, question operators, and an answer program. A strong LLM takes that specification and synthesizes a self-contained Python simulator; executing it produces a natural-language multi-turn event history, and deterministically replaying the same history computes the reference answer and an atomic checklist. The paper's authors describe the division of labor this way: "The LLM proposes the executable mechanism, while the replayed program state determines the supervision." The Hunyuan team compressed the same point in its X post: "Code guarantees the logic. Language delivers the mess." Constraints are compiled into the code as guards and runtime assertions, so an inventory-decrement event passes only when stock remains, and a refund only when it references a valid earlier payment and does not exceed its unsettled amount.
The scale is spelled out in numbers. More than twenty annotators wrote 117 task prototypes across 12 themes, and the 117 validated simulators support 159 final-question operators. The operators fall into seven families under four groups: ranking and comparison, aggregation and profiling, audit and counterfactual reasoning, and localization. Five difficulty tiers scale the events per sample from about 7 to about 1,200, yielding roughly 35,100 training examples and 585 validated evaluation instances. Each simulator contributes 60 training instances and one held-out instance per tier, so each tier holds 7,020 training examples and 117 evaluation instances. The specifications contain 3,410 named entities, 928 event schemas, 549 tracked quantities, 1,019 validity rules, and 442 distractor templates, with an invalid-record rate of 10 to 14 percent. When the annotators inspected a random sample of 117 instances, the paper says, they found zero logical errors.
What stands out in the evaluation table is the drop. The median avg@5 across the 14 models goes from 71.2 on the shortest tier to 67.8, 50.5, 21.9, and finally 11.3 on the longest. The top score belongs to GPT-5.5 xhigh, with an overall pass@5 of 82.7 and avg@5 of 72.8, and it held an avg@5 of 59.3 even on the longest tier. On that tier, only the three GPT-5.5 variants exceed an avg@5 of 30, with DeepSeek V4 Preview Pro close behind at 29.7, Gemini 3.1 Pro at 27.5, Tencent's own Hy3 high at 19.8, and GLM 5.2 at 12.5. Six models fell below 10. The gaps between models also widen by tier. GPT-5.5 xhigh and GLM 5.1 were 3.6 points apart on the shortest tier, 78.5 to 74.9, but 49.7 points apart on the longest, 59.3 to 9.6. Grading is done by gpt-oss-120b using only the checklist. It does not see the preceding dialogue; it receives only the final question, the answer, and the checklist and judges whether each item is satisfied.
Read through an engineer's eyes, this table shows a number different from a context-window spec. Every input fit within each model's supported window and was processed without truncation, the paper says. So what fell is not the length a model can read but its ability to recompute the current value while filtering out canceled orders, corrected receipts, and records posted late. Most real work contexts look like this: the state-change history of a ticketing system, a ledger mixing orders and refunds, a code change that has gone through repeated reviews. METAL has reported on Real-SWE, a benchmark built from enterprise codebases, and on Alibaba's e-commerce benchmark for long-horizon autonomous work. EvolveScaler sits between them as a ruler that isolates one capability: state reconstruction.
The second result is that the data works for training. The researchers continued training an internal A3B model with GRPO on 6,000 EvolveScaler examples, then measured it on eight external benchmarks absent from the training data. The average rose from 33.17 to 38.42, a gain of 5.25 points, and it beat the base model on all eight. The first 200 examples alone lifted the average by 2.43 points, followed by gains of 1.85 and 0.97. The largest improvements were on MARS at 9.50 points, MRCR at 7.59, MultiChallenge at 7.33, and AA-LCR at 6.41. Under the same 6,000-example budget, a mix of 2,000 examples each from the short, medium, and long tiers beat any single-tier composition on all eight benchmarks.
There is also a breakdown of wrong answers. Labeling each failed response by the first checklist item it violated, the most common failure in all ten models was combining signed changes incorrectly. Its share ranged from 46.2 percent for GPT-5.5 xhigh to 55.9 percent for Doubao 1.8 high. Retrieval and counting errors, where relevant records were omitted or irrelevant ones included, came next at 16.8 to 21.1 percent, and ranking and tie-breaking errors were at least 19.2 percent, reaching their highest, 29.8 percent, in GPT-5.5 xhigh. Picking the wrong set of valid records never exceeded 5.5 percent. In other words, models can mostly tell which records were canceled; they break down on the addition and subtraction of what remains.
METAL has confirmed the paper PDF is 19 pages. The project page has an interactive demo built around a warehouse shift board: on the left is the natural-language context the model reads, on the right the hidden ledger maintained by code. Drill tasks, retracted scans, old count drafts, and test waves never touch the ledger, and the same log is queried through seven operator families. METAL has reported on Tencent Hunyuan releasing a model that generates and edits voices, and on the open-source release of the Hy4 preview. This paper is not a model release but a release of data for measuring and teaching models, and as of the evening of the 15th in Korea time the X post had 5,634 views. The paper notes that the current framework is limited to discrete, programmatically specified state transitions and lists extending it to partially observed, continuous, and multimodal settings as future work.
The long-context race is shifting from how many tokens a model can read to how many corrections it can withstand. The numbers in this paper are the starting line for that race. Even the most advanced model gets a history of about 1,200 events right only about three times in five, and six of the 14 models get it right less than one time in ten. What remains for those who build data is that this approach, pinning state down in code and layering language on top, worked not only for evaluation but for training.
Comments