Claude and Codex build toy tokenizers in 30 minutes, then fail at production scale

The outcome was identical for both agents. Each produced a working trainer within 30 minutes, passed its own unit tests, and even trained a toy tokenizer on a few megabytes of data. But when fed a real, production-scale corpus, neither trainer survived.

Why a tokenizer trainer, specifically

While researching how vocabulary size affects the performance of on-device (edge) LLMs, Liquid AI needed a byte pair encoding (BPE) trainer — a tokenization method that builds a vocabulary by merging frequently occurring character pairs — capable of processing trillions of tokens on a single machine. But every existing tool fell short. sentencepiece is optimized for non-BPE methods and was slow; Hugging Face's tokenizers library ran out of memory on large corpora; and tiktoken had no training functionality at all.

Liquid AI chose this task as its test case for three reasons. First, it wasn't a matter of porting an existing system to another language — the goal was an actual deployable product. Second, it required two distinct areas of expertise at once: an ML researcher who understands tokenizer training, and a Rust engineer who can write memory-aware, multithreaded code. Liquid AI said it wanted to test "whether an agent can cover a range of expertise that none of our own engineers could handle alone." Third, the output had to load correctly in both tiktoken and Hugging Face's tokenizers as an external validation condition, leaving no room for the agent to fake success.

Designing so goals and verification stay separate

Before the agents wrote a single line of code, the experiment's designers prepared two things. The first was a set of specification documents — AGENTS.md and CLAUDE.md — that defined the goal. These contained only the desired outcomes and constraints, not implementation details: since memory was the biggest constraint, the trainer had to reliably handle corpora far larger than available RAM, and it was enough to note that computation and I/O should be handled with Rust and multithreading.

The second was a verification apparatus the agents could not touch. The agents were given sandboxed access to real production training datasets and an AMD EPYC 9755 server (128 cores, 256 threads, 2TB of memory) capable of processing them. The resulting vocabularies were loaded into both tiktoken and Hugging Face's tokenizers to check whether encode-decode round trips matched, and whether the trainer held up across multiple languages, numbers, currency notations, tabs, newlines, and source code.

The code passed — so why did it break?

The problem was scale. Flaws invisible in a few-megabyte test set surfaced one after another once run against a real corpus. Liquid AI listed seven such issues:

Issue found	How it surfaced	What caught it	Time to fix
File encoding mismatch	Fine in tests, silently misbehaved on the corpus	Real corpus files	Several hours
Lack of memory awareness	Ran out of memory at about 1% into the target corpus	Full-scale run	2–3 days
Poor parallelization	Throughput fell short even with all cores busy	Full-scale run	1–2 days
Slow pre-tokenization	Regex backtracking caused a sharp processing slowdown	Profiler	Several hours
Rank-ordering error	Vocabulary loaded but encoded differently than intended	External validation tool	Several hours
Duplicate merges	Vocabulary count fell short, shifting all subsequent ranks	External validation tool	Several hours
Number-encoding error	The two libraries produced mismatched results only for numbers	External validation tool	One-line fix

What changed once the loop was introduced

At this point, Liquid AI changed its approach. It introduced a cycle in which the agent runs against real data, reports the symptom when it hits a wall, diagnoses and fixes the cause itself, and runs again. Liquid AI said that through repeated cycles of this loop, both agents worked through the issues listed in the table above, one by one. The resulting BPE tokenizer trainer, toktoktok, is now open-sourced on Liquid AI's GitHub account.

In the industry, this kind of iterative structure is known as an agent loop. Each cycle has four steps: run against real data, report the symptom exactly as it appears — what broke and how — identify the cause rather than just the symptom, fix it, and go back to the start. The gap in fix times in the table above, ranging from "several hours" to "2–3 days," reflects how many times this cycle had to turn for each defect.

In this structure, the agent is not the one doing the grading. Looking again at the "What caught it" column in the table above, not a single defect was caught by the agent's own unit tests. All seven were caught by real corpus files, full-scale runs, the profiler, and external validation tools. The fact that the toy trainer built in 30 minutes passed its own tests tells the same story from the other side. For the loop to work, there must first be a scorecard the agent cannot touch.

How to set up a loop yourself

The structure of this experiment can be replicated without any special equipment. Both Claude Code and Codex already offer execution modes that run without a human watching. The process breaks down into four steps.

First, keep the goal and the scorecard in separate files. The specification should state only the outcome and constraints, not the implementation method. Put the constraint most likely to break first at the top — in this experiment, that was memory. Since Claude Code doesn't read AGENTS.md directly, follow Anthropic's documentation and put @AGENTS.md on the first line of CLAUDE.md, or symlink the two, so both tools see the same specification.

Second, put the scorecard outside the agent's reach. In Claude Code, adding the verification script's path to permissions.deny in .claude/settings.json blocks editing outright. This carries more force than simply writing "don't touch this file" in the specification. The same documentation makes clear that CLAUDE.md is reference context, not an enforced setting.

Third, move the loop's control to the shell. If you tell the agent to "repeat until it works," it will stop the moment it decides on its own that it's done. It's more reliable to have a loop in the shell that runs the verification script first, and on failure, feeds the entire log back to the agent for another round. Pass the log along unabridged, without summarizing it — clues like "the two libraries only diverge on numbers" are the first thing to disappear in a summary.

Task	Claude Code	Codex
Run one round without a human	claude -p "instruction"	codex exec "instruction"
Allow file edits	--permission-mode acceptEdits	--sandbox workspace-write
Disable approval prompts	Specify tools via --allowedTools	--ask-for-approval never
Carry context from the previous round	--resume session-ID	codex exec resume --last
Cap on rounds/cost	--max-turns / --max-budget-usd	Handle directly in the shell loop

Codex's --full-auto has been deprecated, replaced by --sandbox workspace-write. If approval settings aren't disabled, the run fails outright the moment an approval prompt appears, so this must always be specified for non-interactive execution.

Fourth, decide the stopping conditions in advance. Set a ceiling on both rounds and cost, and with Claude Code you can even use a Stop hook to prevent the agent from stopping until verification passes. If the hook script returns exit code 2, the stop is blocked and whatever is written to stderr is passed straight back to the agent. Be sure to first check the stop_hook_active value coming in through the hook input, and release the agent if it's already looping because of the hook. Without this safeguard, the agent can get stuck on a single defect it can't fix.

If the same log repeats three times, the answer isn't to allow more rounds. That's a sign the specification is ambiguous or the scorecard isn't identifying the cause — and at that point, a human needs to rewrite the specification.

Editor's view

What makes this experiment interesting is that it didn't ask whether the agent writes good code. What Liquid AI actually tested was whether an agent can pass production verification without a human — and that's where the two paths diverged. The toy trainer built in 30 minutes looks like a success at a glance, since it passed its own tests, but its real failure only showed up once scale entered the picture. This experiment suggests that many of the "success stories" commonly seen in benchmarks or demos may never leave this toy stage.

Applying similarly sized coding-agent tasks to real work tends to produce the same conclusion every time: the first result looks plausible, but hidden assumptions break the moment it meets real data scale or edge cases. The fact that the memory shortage in this experiment surfaced at just 1% into the target corpus is evidence of that — an error that a small test could never have caught.

The practically useful takeaway is the design principle of separating goals from verification. Specifying only the outcome and constraints rather than the implementation, and verifying with external libraries the agent cannot touch, is close to the minimum bar teams should reference when handing real work over to agents. It implies that before humans can step back from code review, the verification itself needs to be automated, and repeated execution needs to be allowed.

Similar "loop-based" agent evaluations are likely to emerge from other companies in the coming weeks. As the race over frontier models' coding benchmark scores continues, evaluation methods like this one — distinguishing "toy" from "production" — could establish themselves as a new standard for testing benchmark credibility.