Model or Harness? An Interaction-Centric Taxonomy for Localizing Agent Failures

arXiv:2607.288022026-07-29

A classification system that pinpoints whether an AI agent's failure is the model's fault, the tooling's fault, or the environment's fault

When an AI agent (a coding assistant, a personal assistant, a multi-agent system) fails, the visible outcome alone often doesn't tell you what to fix. This paper represents each failure as an interaction between the model and another component, and marks which side is at fault, producing a taxonomy of 41 failure modes. Testing four frontier models as independent judges against human labels, the best judge reached Cohen's kappa of 0.76 on category agreement.

METAL LAB explanatory visual

Labeling failures as an interaction edge plus a fault side

Evidence statusMeasured results reported

Define componentsAn agent is modeled as model, owner, grader, third party, harness (context, memory, tool), and environment (local, external).
Interaction edgeEach failure is anchored to a specific relationship between two components, such as model-tool or model-memory, as the unit of analysis.
Root-cause tracingWhen errors cascade, the label is assigned to the earliest failure from which execution does not recover, not to later symptoms.
41 failure modesEach edge is marked with which side is at fault, yielding 41 failure modes, 36 model-side and 5 assigned to surrounding components.
AI judge validationGPT-5.5 and Claude Opus 4.6/4.7/4.8 were tested as independent judges on 40 worked examples, measured against human labels with Cohen's kappa.

An explanatory diagram made by METAL LAB, not a figure supplied by the paper's authors.

What they did

The paper argues that reducing agent failures to system-level outcomes hides where the fault originated, so the same visible failure could call for model retraining, harness engineering, or environment redesign depending on its true cause.
It represents an agent as a set of interacting components, model, owner, grader, third party, harness (context, memory, tool), and environment (local, external), and labels each failure as an edge between two components plus which side is at fault, yielding 41 failure modes.
When multiple errors cascade, the authors trace backward to the earliest failure from which execution does not recover and assign the label there, rather than to later downstream symptoms.
They validated the taxonomy on 40 worked examples drawn from public benchmarks, model system cards, published reports, and logged agent trajectories, using GPT-5.5 and Claude Opus 4.6, 4.7, and 4.8 as independent judges measured by Cohen's kappa against human labels.
For category-level agreement (interaction edge plus fault side), GPT-5.5 reached kappa=0.76 with the human annotator, Claude Opus 4.6 and 4.7 each reached 0.71, Opus 4.8 reached 0.70, and the highest pairwise agreement among the judges themselves was kappa=0.84 between Opus 4.6 and 4.8.

Figure 1: Radial interaction map. The model is the hub, the User, Harness, and Environment families form the inner ring, and their components appear on the outer ring. Each failure is represented by an edge between two components. Since a model interacts with another model through its own harness, model–model interactions are grouped under Harness alongside Context, Memory, and Tool. The outer node is another model, labeled according to its role as a peer or subagent.

Table 1: Component vocabulary used to localize agent failures. Each row defines a component that can form an endpoint of an interaction edge. In multi-agent settings, peer and subagent describe the role of the other model. In subagent interactions, the focal model acts as the orchestrator, while in peer interactions, neither model directs the other.
Component	Definition
Model	The policy that processes observations and produces outputs or actions.
Owner	The human or upstream system that gives the agent its task and defines what counts as success.
Grader	The mechanism used to evaluate whether the agent completed the task successfully; it is usually not visible to the agent.
Third party	An actor encountered during execution that does not act on behalf of the owner. The actor can be a human, organization, or agent, and the interaction may be adversarial, persuasive, or cooperative.
Context	The information available to the model during the current interaction, including instructions, conversation history, observations, and summaries.
Memory	A persistent store that outlives the active context, within or across sessions.
Tool	The bidirectional interface through which the model exchanges requests, messages, actions, observations, and responses with other components. This includes callable tools, communication channels, and wrappers that relay inputs and outputs.
Local env.	The agent’s immediate execution environment, such as the operating system, shell, filesystem, and runtimes.
External env.	Systems outside the agent’s immediate execution environment, such as remote services, websites, APIs, databases, and model-provider infrastructure.

Figure 2: Interaction-centric taxonomy of 41 failure modes. Failures are organized by the family of the component interacting with the model: User, Harness, or Environment, and then by the specific component within that family. Each branch represents an interaction edge between the model and that component. The hierarchy is organizational, where the model and the interacting component form the two endpoints of each edge. The leaves show the failure modes arising from each interaction, and shading indicates which endpoint is at fault. Of the 41 role-specific failure modes, 36 are assigned to a model and five to surrounding components.

Table 2: Agreement of each judge with the human labels on the 40 worked examples. Category scores require the correct interaction edge and fault side. Failure-mode scores additionally require the correct named failure. Acc denotes exact-match accuracy and F1 is macro-averaged.
	Category	Failure mode
Model	Acc	F1	Acc	F1
GPT-5.5	0.80	0.69	0.72	0.64
Claude-Opus-4.6	0.75	0.61	0.70	0.57
Claude-Opus-4.7	0.75	0.63	0.62	0.53
Claude-Opus-4.8	0.75	0.62	0.68	0.58

Figure 3: Pairwise Cohen’s κ among the four LLM judges and the human annotator, with category labels on the left and complete failure-mode labels on the right. The bold outline marks agreement with the human annotator.

Table 3: Failure-mode agreement for the four judges on the 40 worked examples. Under Predicted cat., the judge predicts both the category and failure mode; under Gold cat., it selects the failure mode given the human-assigned category. Acc is exact-match accuracy, and F1 is macro-averaged.
	Predicted cat.	Gold cat.
Model	Acc	F1	Acc	F1
GPT-5.5	0.72	0.64	0.72	0.62
Claude-Opus-4.6	0.70	0.57	0.80	0.70
Claude-Opus-4.7	0.62	0.53	0.70	0.58
Claude-Opus-4.8	0.68	0.58	0.78	0.69

Table 4: Selective-voting ensemble of the four judges at increasing category-agreement thresholds. At each threshold, the ensemble assigns a category label only when the required number of judges agree and abstains otherwise. Coverage is the proportion of all examples that receive a label. Precision is computed over labeled examples, whereas recall is computed over the full evaluation set. After selecting a category, the ensemble assigns the failure mode by majority vote among only the judges that predicted that category.
		Category	Failure mode
Agreement	Cov	P	R	F1	P	R	F1
≥2 of 4	1.00	0.78	0.78	0.78	0.70	0.70	0.70
≥3 of 4	0.90	0.83	0.75	0.79	0.75	0.68	0.71
4 of 4	0.68	0.96	0.65	0.78	0.89	0.60	0.72

Table 5: Safety-risk categories used to annotate the worked examples. Each example is assigned the single most salient harm associated with its root-cause failure.
Risk category	Source	Trajectory-observable harm	Examples
Excessive Agency	OWASP LLM06	acted beyond granted permission, or took an unconfirmed risky/irreversible step	E2, E4, E6, E39
Unbounded Consumption	OWASP LLM10	looped or exhausted its budget without making progress	E19, E32, E33
Rogue Agents	OWASP ASI10	gamed its own grader / reward-hacked, deviating from the set objective	E12, E13
Agent Goal Hijack	OWASP ASI01	untrusted third-party input hijacked the agent’s goal or control flow	E15, E16
Misinformation	OWASP LLM09	fabricated content presented as genuine, completed work	E11, E28, E31
Sensitive Information Disclosure	OWASP LLM02	exposed or over-shared private data	E10

Findings

On category-level recovery (interaction edge plus fault side), GPT-5.5 reached the highest agreement with the human annotator at Cohen's kappa=0.76, with Claude Opus 4.6 and 4.7 each at 0.71 and Opus 4.8 at 0.70.
Agreement among the four judges themselves was comparable to their agreement with the human annotator, with the highest pairwise value at kappa=0.84 between Claude Opus 4.6 and 4.8.
Agreement on the complete, specific failure-mode label was lower across all pairs than agreement on the broader category label.
Of the 41 role-specific failure modes, 36 were assigned to the model as the fault side and 5 to surrounding components.

Where it can be used

Analyzing failure logs across coding assistants, long-horizon personal assistants, and multi-agent systems using a shared checklist for deciding whether to retrain the model, fix tool integration, or redesign the environment.
Using AI models as judges to build a pipeline that classifies failure causes at scale across large sets of agent execution logs.
Tagging failure examples with safety and security impact by cross-referencing the taxonomy with OWASP LLM risk categories.

Limits and open work

The 40 worked examples were selected to illustrate a range of taxonomy categories, not to estimate how frequently each failure mode occurs in practice.
Agreement on the exact failure-mode label was consistently lower than category-level agreement, so automating fine-grained labeling needs further validation.
When source material is incomplete (an issue report or blog post rather than a full execution trace), both judges and humans can disagree about where the root cause actually lies.
Root-cause attribution itself remains genuinely difficult in some cases, for instance when tracing a causal path back through a delayed or missing downstream event, leaving a risk that automated judges label surface symptoms rather than true causes.

Why it matters

Teams building agent systems need a shared vocabulary for deciding whether a failure log calls for retraining the model, fixing the harness code, or redesigning the evaluation environment. The finding that AI judges can reproduce human labels reasonably well suggests this classification work itself could eventually be automated at scale.

Terms in this paper

harness · the software scaffolding that manages a model's context, memory, and tool access, i.e. the code that runs the agent
Cohen's kappa · a statistic measuring how much two raters (e.g. a human and an AI) agree, beyond what chance agreement alone would predict
interaction edge · the connection between the model and another component (user, tool, environment, etc.) that a failure is localized to
agent-as-a-judge · an evaluation method where an AI agent independently investigates evidence and reconstructs an account before rendering a judgment, rather than just reading a candidate answer

Original abstract (English)

Existing evaluations often reduce agent failures to system-level outcomes, obscuring where the fault originated and which intervention would improve the agent system. This creates a repair-assignment problem: the same visible failure may call for model post-training, harness engineering, environment redesign, or benchmark repair depending on its source. Because agent behavior emerges from interactions among models, harnesses, users, tools, memory, and environments, outcome-level labels are often insufficient for improvement. Most failure taxonomies do little to resolve this problem because they are benchmark-specific and lack a shared structure. We introduce an interaction-centric taxonomy that localizes failures to the interactions in which they originate and identifies the responsible component. It organizes 41 failure modes by assigning each to an edge between two components and a fault side indicating where the repair belongs. This makes the taxonomy actionable: model-side failures identify targets for post-training, harness-side failures point to scaffolding and tool-integration fixes, and environment or grader failures reveal evaluation conditions requiring redesign. The schema applies across agent architectures, from coding assistants to long-horizon personal assistants and multi-agent systems. We ground the taxonomy in worked examples from public benchmarks, model system cards, published reports, and logged agent trajectories, and evaluate its reproducibility using independent reasoning agents as judges. Across four frontier models, the strongest judge reaches Cohen's $\kappa=0.76$ against human category labels, suggesting that the categories capture shared structure rather than annotator-specific preferences.

Authors · Harsh Raj

Read on arXiv

Latest papers

All papers →

Latest from METAL LAB

Figures: Harsh Raj et al., arXiv:2607.28802, CC BY 4.0