liquid4all/toktoktok

★ 23RustApache-2.0

An AI coding agent wrote this entire tokenizer trainer, and no human read the code

toktoktok is an experiment by Liquid AI: they gave coding agents a short spec of outcomes and constraints, sandboxed access to real data, and an external check the agents couldn't game, then let the agents write everything from scratch. The result is a BPE (Byte Pair Encoding) tokenizer trainer whose output loads directly into both OpenAI's tiktoken and Hugging Face tokenizers. It can train on trillions of tokens on a single machine while staying inside a memory budget you set.

What it does

Liquid AI needed a BPE tokenizer trainer for their research on vocabulary size in edge LLMs, but existing tools were too slow, ran out of memory, or couldn't train at all — so instead of building it themselves, they handed the problem to coding agents
The agents worked from AGENTS.md, a short specification, with sandboxed access to production data and a large machine, verified only by an external harness requiring the trained vocabulary to produce identical token IDs in both tiktoken and Hugging Face tokenizers
Reservoir sampling keeps training inside a declared memory budget even when the corpus is far larger than RAM, so huge corpora stay representative without being loaded whole
Multi-phase training lets you allocate vocabulary budget explicitly across languages or domains (e.g. English, code, multilingual) instead of letting the biggest corpus dominate the merges, and warm start lets you extend an existing tokenizer without changing its existing token IDs
The final vocabulary is always 256 base bytes plus 1,161 hardcoded merges (for numbers, whitespace, operators, etc.) plus the merges trained in each phase

Why it matters

It's a concrete test case of whether coding agents can solve a production-grade engineering problem end to end without a human reading the code, which matters for anyone weighing how much to trust AI-written infrastructure. For LLM builders, it's also a practical, reproducible tool for training and extending tokenizer vocabularies at scale.

Terms in this repo

BPE (Byte Pair Encoding) · a tokenizer training method that repeatedly merges the most frequent pair of symbols into a single token
tiktoken · OpenAI's BPE tokenizer library
reservoir sampling · a technique for randomly sampling a representative subset from a stream of data too large to hold in memory
warm start · continuing training from an existing tokenizer's vocabulary so its prior token IDs stay unchanged
coding agent · an AI system that can write and modify code autonomously without human intervention

Open on GitHub

Coverage

클로드·코덱스, 토크나이저 장난감은 30분 만에 실전은 실패

Trending repos

All repos →