이미지: METAL LAB 생성
Summary
- A blog post questioning the common belief that dynamic languages use fewer LLM tokens than static languages has drawn attention
- The existing benchmark cited a 2.6x token gap between C and Clojure, with J recording the lowest average at 70 tokens
- The author argues the problems tested were too trivial and some evaluations contained bugs, making the conclusions unreliable
- 인용된 격차
- C(최다 토큰)와 Clojure(최소 토큰) 사이 2.6배 차이
- J 언어 결과
- 평균 70토큰, Clojure(109토큰)의 절반 수준
- 검증 문제
- 기존 벤치마크가 Rosetta Code 기반의 단순 문제로 구성됨
- 평가 오류 사례
- 한 에이전트가 존재하지 않는 경로를 자신의 실행 파일로 심볼릭 링크해 이후 테스트 결과가 왜곡됨
- 저자 사전 예측
- 동적·정적 언어 격차가 큰 문제에서는 무너질 것이라는 데 95% 확신 표명
Revisiting the Belief That Dynamic Languages Have an Edge
Blogger Dan Luu (danluu) published a post examining which programming languages are advantageous for coding agents. The existing benchmark he cited found a 2.6x token gap between C and Clojure, and a later attempt with the array language J reportedly averaged 70 tokens — about half of Clojure's 109 tokens. The author quoted the original source as saying, "There was a fairly significant 2.6x gap between C and Clojure." This result spread widely enough to be cited even in Google's AI search summaries.
Why Token Count Becomes a Language Selection Criterion
Coding agents operate by having a large language model (LLM) read and write code to complete tasks. Longer code means more tokens — the basic unit LLMs use to process text — which affects processing cost and response speed. Dynamically typed languages like Python or Clojure tend to produce shorter code because they don't require variable types to be declared in advance, whereas statically typed languages like Rust, Go, and C++ tend to produce longer code because types must be explicitly specified. This difference is the basis for claims that dynamic languages consume fewer LLM tokens.
The problem is that most of this conclusion rests on extremely simple problems, such as those from Rosetta Code. The author pointed out that problems solvable in as few as 70 or 109 tokens can hardly be called real problems. Methodological flaws were also found in other benchmarks. In one test, an attempt to execute a nonexistent path failed, and a subsequent agent run symlinked that path to its own executable — after which all further scoring was based on that agent's executable rather than the intended target language. The author explained that this bug, not actual failures, was the real reason Rust appeared to fail repeatedly.
What Actually Changes
The author validates his own judgment by publishing predictions before seeing results, and in this case he put 95% confidence on the gap between dynamic and static languages disappearing as problem size increases. The pattern where gains seen in simple tasks shrink or reverse in complex tasks has repeatedly shown up across his earlier series of evaluations.
This carries a practical implication for teams designing coding agents: rather than choosing a language based solely on token savings, they should examine whether performance holds up on tasks of real scale. On August 5, Prime Intellect unveiled a self-improving coding harness called "Prime Agent," which it said aims for both token efficiency and expressiveness — showing that token cost remains a key variable across language and harness design more broadly.
