Claude, Codex build toy tokenizers in 30 minutes but fail at production scale

The outcome was identical for both agents. Each produced a working trainer within 30 minutes, passed its own unit tests, and even trained a toy tokenizer on a few megabytes of data. But when fed an actual production-scale corpus, neither trainer survived.

Why a tokenizer trainer, specifically

While researching how vocabulary size affects performance in on-device (edge) LLMs, Liquid AI needed a byte pair encoding (BPE) trainer — a tokenization method that builds a vocabulary by merging frequently occurring character pairs — capable of processing trillions of tokens on a single machine. Existing tools all fell short: sentencepiece was optimized for non-BPE methods and ran slowly, Hugging Face's tokenizers library ran out of memory on large corpora, and tiktoken had no training functionality to begin with.

Liquid AI chose this task as its test case for three reasons. First, it was a genuine deployment target, not a porting exercise of an existing system into another language. Second, it required two distinct areas of expertise simultaneously — an ML researcher who understands tokenizer training, and a Rust engineer who can write memory-aware, multithreaded code. Liquid AI said it wanted to see "whether an agent could cover a breadth of expertise that none of our own engineers could handle alone." Third, the output had to load directly into tiktoken and Hugging Face tokenizers as an external validation condition, leaving the agent no room to fake success.

Separating the goal from the verification

Before either agent wrote a single line of code, the experiment's designers prepared two things. The first was a specification document — AGENTS.md and CLAUDE.md — that defined the goal. It contained only outcomes and constraints, not implementation methods: since memory was the tightest constraint, the trainer had to reliably handle corpora far larger than available RAM, and compute and I/O simply needed to be handled with Rust and multithreading.

The second was a verification mechanism the agents could not touch. The agents were given sandboxed access to an actual production training dataset and to a server (an AMD EPYC 9755 with 128 cores/256 threads and 2TB of memory) capable of processing it. The trained vocabulary was loaded into both tiktoken and Hugging Face tokenizers to check whether encode-decode round trips matched, and whether results held up across multiple languages, numbers, currency notation, tabs, newlines, and source code.

The code passed, so why did it fail

The problem was scale. Flaws that never surfaced on a test dataset of a few megabytes emerged one after another once real corpora were used. Liquid AI summarized seven issues it found:

Issue found	How it surfaced	What caught it	Time to fix
File encoding mismatch	Fine in tests, silently broke on the corpus	Actual corpus files	Several hours
Lack of memory awareness	Ran out of memory at about 1% into the target corpus	Full-scale run	2–3 days
Poor parallelization	Throughput fell short even with all cores busy	Full-scale run	1–2 days
Pre-tokenization slowdown	Regex backtracking caused a sharp drop in processing speed	Profiler	Several hours
Rank ordering errors	Vocabulary loaded but encoded differently than intended	External verification tool	Several hours
Duplicate merges	Vocabulary count fell short, shifting all subsequent ranks	External verification tool	Several hours
Numeric encoding error	The two libraries disagreed only on numbers	External verification tool	One-line fix

What changed once the loop was introduced

At this point, Liquid AI changed its approach. It introduced a cycle in which the agent ran the code on real data, reported symptoms when it hit a wall, diagnosed and fixed the issue itself, and then ran the code again. Liquid AI said that through multiple iterations of this cycle, both agents worked through the problems listed in the table above one by one. The resulting BPE tokenizer trainer, toktoktok, is now open-sourced on Liquid AI's GitHub account.

Editor's view

What makes this experiment interesting is that it didn't ask whether an agent writes good code. What Liquid AI actually tested was whether an agent could pass production-level verification without a human in the loop — and that's where the split happened. The toy trainer produced in 30 minutes looked like a success at first glance, since it passed its own tests, but the real failure only surfaced once scale entered the picture. This experiment suggests that many of the success stories commonly seen in benchmarks or demos may still be stuck at this "toy stage."

Applying similarly sized coding-agent tasks to real work tends to lead to the same conclusion every time — the first output looks plausible, but hidden assumptions break the moment it meets real data scale or edge cases. The fact that the memory failure in this experiment hit at just 1% into the target corpus is proof of that: an error that a small test would never catch.

The practically meaningful takeaway is the design principle of "separating the goal from the verification." Specifying only outcomes and constraints instead of implementation methods, and verifying with external libraries the agent cannot touch, is a minimal condition worth referencing for teams considering handing real work to agents. It implies that rather than having humans substitute for code review, verification itself needs to be automated and structured to allow repeated execution first.

It's likely that similar "loop-based" agent evaluations will emerge from other companies in the coming weeks. Amid ongoing competition over coding benchmark scores among frontier models, an evaluation approach like this one — distinguishing "toy" from "production" — could establish itself as a new standard for testing benchmark reliability.