Existing Token Benchmarks Cannot Rank Coding-Agent Languages

Revisiting the Belief That Dynamic Languages Have an Edge

Blogger Dan Luu (danluu) published a post examining which programming languages are advantageous for coding agents. The existing benchmark he cited found a 2.6x token gap between C and Clojure, and a later attempt with the array language J reportedly averaged 70 tokens — about half of Clojure's 109 tokens. The author quoted the original source as saying, "There was a fairly significant 2.6x gap between C and Clojure." This result spread widely enough to be cited even in Google's AI search summaries.

Why Token Count Becomes a Language Selection Criterion

Coding agents operate by having a large language model (LLM) read and write code to complete tasks. Longer code means more tokens — the basic unit LLMs use to process text — which affects processing cost and response speed. Dynamically typed languages like Python or Clojure tend to produce shorter code because they don't require variable types to be declared in advance, whereas statically typed languages like Rust, Go, and C++ tend to produce longer code because types must be explicitly specified. This difference is the basis for claims that dynamic languages consume fewer LLM tokens.

The problem is that most of this conclusion rests on extremely simple problems, such as those from Rosetta Code. The author pointed out that problems solvable in as few as 70 or 109 tokens can hardly be called real problems. Methodological flaws were also found in other benchmarks. In one test, an attempt to execute a nonexistent path failed, and a subsequent agent run symlinked that path to its own executable — after which all further scoring was based on that agent's executable rather than the intended target language. The author explained that this bug, not actual failures, was the real reason Rust appeared to fail repeatedly.

What Actually Changes

The author validates his own judgment by publishing predictions before seeing results, and in this case he put 95% confidence on the gap between dynamic and static languages disappearing as problem size increases. The pattern where gains seen in simple tasks shrink or reverse in complex tasks has repeatedly shown up across his earlier series of evaluations.

This carries a practical implication for teams designing coding agents: rather than choosing a language based solely on token savings, they should examine whether performance holds up on tasks of real scale. On August 5, Prime Intellect unveiled a self-improving coding harness called "Prime Agent," which it said aims for both token efficiency and expressiveness — showing that token cost remains a key variable across language and harness design more broadly.