이미지: Anthropic 화면 갈무리
Summary
- Anthropic's Frontier Red Team tested behavioral patterns in multi-agent systems where agents interact with one another
- In a vulnerability-hunting experiment with 45 collaborating agents, Claude Mythos Preview found far more vulnerabilities than an independent, parallel approach — but token costs also rose sharply
- In a game-development experiment, changing the prompting approach didn't improve results, and newer models tended to collaborate less in order to avoid conflicts
- 실험1 구성
- 45개 에이전트, 각자 VM, 공유 포럼, 15개 오픈소스 프로젝트, 중재 에이전트 1개
- Mythos Preview 독립 병렬
- 취약점 21개 / 토큰 650만
- Mythos Preview 협업 군집
- 취약점 266개 / 토큰 2700만
- 두 방식 공통 발견
- 겹치는 취약점은 12개뿐
- 실험2 대상 모델
- Sonnet 4.6·5, Opus 4.6·4.8, Mythos Preview / 각 12시간 실행
- 실험2 결과
- 초기 모델은 PR 충돌로 병합률이 낮고, 최신 모델은 협업 자체를 줄여 충돌을 회피
When 45 agents dig through code at once
Anthropic's Frontier Red Team has released the results of an experiment testing how AI agents behave when interacting with one another. The team's premise is that as agents take on more work across codebases, markets, and various social systems, agent-to-agent interactions could explode in volume even before human-to-human interactions do. The problem is that the world doesn't yet know the conditions under which such interactions work well.
To test this, the team ran an open-source vulnerability detection experiment. Each of 45 agents was given its own virtual machine (VM) and a shared forum, along with the identical instruction to find vulnerabilities across 15 open-source projects. The agents peer-reviewed each other's findings, while a separate arbitration agent made the final call on whether a submitted vulnerability was novel and valid. This approach ties back to how Anthropic scans open-source code in Project Glasswing.
Results were compared across two models, Claude Mythos Preview and Opus 4.8. Using Mythos Preview, the simple independent-parallel approach used 6.5 million tokens to find 21 vulnerabilities, while the collaborative-swarm approach used 27 million tokens to find 266. However, roughly half of the vulnerabilities found by the swarm fell outside the core directories that the parallel approach had been instructed to focus on. When narrowed to just the core directories, the two approaches' discovery efficiency per token was similar. Only 12 vulnerabilities were found by both approaches.
What building a game revealed about the limits of collaboration
Collaboration becomes much harder in tasks with high interdependency. To test this, the team assigned multiple agent swarms the task of building a text-based web fantasy game. Each agent had access to its own VM, a shared forum, and a self-hosted repository, and the swarms worked for 12 hours. The team varied both the model generation and the number of agents, and tested three prompting styles: a basic approach, a role-assigned approach (splitting agents into programming, art, and testing teams), and a CEO-hierarchy approach.
Changing the prompting approach, however, didn't produce meaningfully different results. Across all three styles, the finished games didn't run at human-usable speed, the interfaces were hard to make sense of, and the learning curve was steep. The team noted that current models still lack taste in this kind of creative domain and require substantial human intervention.
Even with similar outputs, collaboration patterns diverged sharply by model generation. Sonnet 4.6 and Opus 4.6 appeared to collaborate by committing code to the same files, but the rate of pull requests (PRs) that actually got merged was low — meaning conflicting code was left abandoned. More recent models, Opus 4.8 and Mythos Preview, solved this merge problem, but did so by collaborating with each other far less.
What the two experiments show
| Item | Independent parallel approach | Collaborative swarm approach |
|---|---|---|
| Vulnerabilities found (Mythos Preview) | 21 | 266 |
| Tokens used | 6.5 million | 27 million |
| Overlapping findings | 12 (shared) | 12 (shared) |
| Model generation | PR merge tendency | Code-sharing pattern |
|---|---|---|
| Sonnet 4.6, Opus 4.6 | Low merge rate | Conflicts left unresolved |
| Opus 4.8, Mythos Preview | Stable merge rate | Minimized collaboration itself |
Placing the two tables side by side reveals an interesting contrast. For tasks that can be split into independent chunks, like vulnerability hunting, swarm collaboration increased the number of findings. But for tasks that depend on each agent's output, like a shared code repository, smarter models "adapted" by reducing collaboration altogether to avoid conflicts.
Editor's view
What makes this experiment interesting isn't the results themselves but the shape of the failure. In the game-development experiment, the fact that newer models improved on the "merge rate" metric — but did so by avoiding collaboration rather than genuinely improving their ability to collaborate — is the sharpest takeaway. Looking only at benchmark numbers, it looks like progress; in reality, it's closer to sidestepping the problem. This kind of metric illusion is a pattern that could keep showing up across agent evaluation more broadly.
Discussions of multi-agent systems have mostly framed the issue as one of scale — how many agents can run in parallel. Kimi Agent Swarm, released on August 8, focused on the same thing, deploying up to 100 sub-agents in parallel. But this Anthropic experiment takes a different angle, looking not at scale but at how collaboration fails. It shows that increasing the number of agents and getting agents to genuinely collaborate well on interdependent tasks are entirely different problems.
In practice, the two cases should be handled separately. For tasks that can be split into independent units, like vulnerability scanning or data labeling, agent swarms are already worth deploying — though it's worth factoring in that token costs can jump more than fourfold compared to a parallel approach. For tasks where multiple agents need to edit the same codebase or document together, it's safer to assume that, with the current generation of models, humans still need to step in at every merge point.
Over the coming months, frontier labs are likely to shift their training signals toward teaching genuine coordination rather than conflict avoidance. Instead of surface-level metrics like merge rate, measures of collaboration quality — such as the rate of actual code sharing or real contribution — are likely to start appearing in future model cards.
