An End-to-End Agent Auditing Engine

arXiv:2608.073462026-08-06

An evaluation engine that traces the entire run of an AI agent to reveal how much the 'harness' running it actually matters

The tool that runs an LLM agent day-to-day, the 'harness' (CrewAI, LangGraph, OpenAI Agents SDK, etc.), can change outcomes drastically even with the same model, but standard evaluation that only checks final correctness misses this. The authors built A2E, an end-to-end engine that unifies benchmark integration, execution tracing, and multi-dimensional evaluation into one pipeline, scoring the whole trajectory including planning, tool use, answers, and operational cost. Across their experiments, no single model-harness combination consistently won across all tasks, and correctness alone barely distinguished harnesses that differed a lot in cost and process.

METAL LAB explanatory visual

A2E's Three Layers: Task, Monitor, Evaluation

Evidence statusMeasured results reported

Task Layer (ATP)Separates benchmarks from agent harnesses via a shared Agent Task Protocol so new benchmarks or harnesses can be added independently
Monitor LayerAutomatically records model calls, tool use, errors, and latency as OpenTelemetry-style spans, producing standardized execution traces
Evaluation LayerScores rule-based metrics (accuracy, tokens, cost) and LLM-as-judge metrics (reasoning quality, safety) across four lifecycle stages: reasoning, action, final answer, runtime quality
Central Server (DB)Stores runs and evaluation results in a database rather than log files, enabling querying and re-evaluation across runs, models, and harnesses
Comparative FindingsAcross 9 harnesses x 23 benchmarks (1,035 runs) and GLM-5.2 x 9 harnesses x 3 benchmarks, no harness consistently topped every task

An explanatory diagram made by METAL LAB, not a figure supplied by the paper's authors.

What they did

The team built the Agent Task Protocol (ATP), a shared interface that lets benchmarks and agent harnesses be combined independently, avoiding custom adapter code for every new benchmark-harness pair.
They attached a Monitor that automatically records model calls, tool use, errors, and latency during an agent's run, producing standardized 'traces' of the full execution rather than just the final answer.
On top of these traces they designed Lifecycle-Aligned Evaluation, scoring metrics across four stages: reasoning, action, final answer, and runtime quality (cost, safety).
Running the same DeepSeek-V4-pro model across nine harnesses on 23 benchmarks (1,035 total runs) under matched conditions showed near-identical scores on simple QA tasks, but wide spreads (0.00 to 1.00) on multi-turn tasks like tau-bench, GDPVal, and traject-bench, with rankings flipping between tasks.
Comparing nine harnesses using the same GLM-5.2 model on GDPVal, MMLU-Pro, and tau3-bench showed large differences in success rate and token usage, and no single harness led on all three benchmarks.

Figure 3: Runtime workflow and data flow. The monitored task runner writes experiment runs and traces to the centralized server. The evaluation component retrieves these records, performs trace-level and outcome-level evaluation, and writes the evaluation results back for storage and visualization.

Table 2: Metric-based comparison of two complete τ3-bench trajectories generated by GLM-5.2 on the same task. The proposed evaluation engine captures differences in task outcome, execution efficiency, tool behavior, diagnostic reasoning, resource consumption, and safety properties.
Dimension	LangGraph (Successful)	CrewAI (Failed)
Task outcome	Correctly identifies the suspended-line issue and reaches the payment-based recovery path. Metrics: task_succeeded=1.0, correctness=0.0	Fails to resolve suspended service and terminates after device-level troubleshooting. Metrics: task_succeeded=1.0, correctness=0.0
Execution efficiency	Completes the trajectory with 3 interaction turns, 4 LLM calls, and 3 tool calls. Total tokens: 10,122 Metrics: turn_count, tool_call_count, total_token_usage	Requires 5 interaction turns, 9 LLM calls, and 5 tool calls. Total tokens: 96,704 Metrics: turn_count, tool_call_count, total_token_usage
Tool interaction	Executes: get_status_bar → reseat_sim_card → get_status_bar Uses tools according to the task progress. Metrics: tool_invocation=1.0	Executes: get_status_bar → reseat_sim_card → reset_apn_settings → reboot_device → toggle_airplane_mode Performs additional recovery operations without resolving the root cause. Metrics: tool_invocation=1.0
Diagnostic behavior	After observing that SIM status is normal but signal remains unavailable, the agent shifts from device-level debugging to account-level diagnosis. Metrics: plan_goal_alignment, plan_completeness	Continues exploring device-level fixes despite unsuccessful interventions and misses the account-level cause. Metrics: plan_goal_alignment, plan_constraint_adherence
Resource consumption	Prompt tokens: 8,520 Completion tokens: 1,602 Cost: 0.0051 Metrics: prompt_tokens, completion_tokens, cost	Prompt tokens: 94,615 Completion tokens: 2,089 Cost: 0.0430 Metrics: prompt_tokens, completion_tokens, cost
Safety and reliability	Produces grounded actions without unsupported claims. Metrics: hallucination=1.0, privacy_leakage=1.0, harmful_action=1.0	Also remains safe and grounded, but fails at selecting the correct recovery strategy. Metrics: hallucination=1.0, privacy_leakage=1.0, harmful_action=1.0
Overall assessment	Achieves a compact and task-directed trajectory with lower execution cost.	Consumes substantially more resources and terminates without solving the original task.

Findings

Running nine harnesses with the same DeepSeek-V4-pro model under matched settings across 23 benchmarks (1,035 runs), all nine harnesses scored identically on single-turn QA tasks, but multi-turn tasks showed wide spreads (tau-bench 0.00-0.60, GDPVal 0.00-0.60, traject-bench 0.20-1.00) with rankings not carrying over across tasks (e.g., openai-agents topped traject-bench at 1.00 but ranked lowest on tau-bench and gdpval).
Analyzing 855 runs across 19 non-sandbox benchmarks with 13 metrics, correctness across eight instrumented harnesses ranged narrowly from 0.568 to 0.663, while mean token cost varied 3.5x, from 2,063 tokens for Claude-Agent-SDK to 7,319 for smolagents.
Comparing nine harnesses with the shared GLM-5.2 model on GDPVal, MMLU-Pro, and tau3-bench, success-rate gaps reached 0.20 on GDPVal, 0.30 on MMLU-Pro, and 0.66 on tau3-bench, with different top-three harness sets on each benchmark.
On the same tau3-bench task with the same GLM-5.2 model, LangGraph reached a correct answer (score 1.0) using 10,122 tokens and 4 LLM calls, while CrewAI failed the task (score 0.0) despite using 96,704 tokens, about 9.6x more than LangGraph.

Figure 5: Overview of the execution-aligned agent evaluation framework. Process-level evaluation examines the iterative reasoning and action stages, outcome-level evaluation assesses the final result, and lifecycle-level evaluation measures operational properties across the complete agent trajectory.

Where it can be used

Choosing which agent framework (harness) to deploy in production by looking beyond correctness at token cost and stage-level process metrics
Integrating new benchmarks or new agent frameworks into an evaluation pipeline without writing separate adapter code for every benchmark-harness pair
Diagnosing why a harness underperforms with a fixed model, by inspecting stage-level differences in prompt construction, tool selection, error handling, and termination behavior

Figure 6: The 855 runs of the 19 non-sandbox benchmarks of Table 1: read through the trajectory. correctness is re-scored on the 44 runs where the multiple-choice Extractor returned an empty prediction (§6.1). (a) Per-harness deviation from each metric’s cross-harness mean, grouped by execution stage; blue is below average, red above, and a cell is annotated only where the deviation reaches 0.10. Rows in grey italic (tool_invocation, hallucination) are set by the instrumentation rather than the agent and should not be read as harness behaviour. (b) Mean token cost against mean correctness, marker area proportional to turn count. crewai is omitted: its LLM spans carry no token counts, while for the other eight the recorded metric equals the span-level prompt-plus-completion sum exactly.

Limits and open work

The authors note only five tasks per cell in the main matrix, giving a score resolution of 0.20 and high per-cell variance
Being listed in the harness registry does not mean every harness-benchmark pair has passed end-to-end validation
Of the 13 trace-level metrics, most process dimensions beyond correctness and token cost showed little variation on this particular benchmark mix; turning them into concrete harness-design guidance is left to future work
crewai was excluded from the token-cost comparison (Figure 6b) because its LLM call spans carry no token counts

Figure 7: Comparison of nine agent harnesses across three benchmarks using GLM-5.2 as the common API model. Each point represents one harness, with average completion tokens on the horizontal axis and task success rate on the vertical axis. To jointly measure effectiveness and efficiency, we identify the top three harnesses on each benchmark based on their trade-off between achieving higher task success and using fewer completion tokens, and highlight them using rank-specific colored circles, where black, magenta, and gold circles denote the first-, second-, and third-ranked harnesses, respectively. The top-ranked harnesses are CrewAI, OpenAI Agents, and AutoGen-AgentChat on GDPVal; OpenAI Agents, AutoGen-AgentChat, and LangGraph on MMLU-Pro; and LangGraph, Claude-Agent-SDK, and Google ADK on τ3-bench. These results demonstrate that our evaluation framework enables unified cross-benchmark and cross-harness analysis, revealing both performance differences and efficiency trade-offs under a consistent evaluation protocol.

Why it matters

It shows with concrete numbers that not just which model you use, but which execution harness wraps it, substantially shapes performance and cost when deploying agents. It's also a practical warning that picking a harness based on correctness alone can hide inefficiency, cost blowups, or failure to recover from errors.

Terms in this paper

agent harness · The execution framework that actually runs an LLM agent, handling tool calls, dialogue flow, and state management (e.g., CrewAI, LangGraph)
Agent Task Protocol (ATP) · A shared interface specification that lets benchmarks and harnesses be plugged together independently
trace · A standardized, time-ordered log of everything that happened during a run: model calls, tool use, errors, etc.
Lifecycle-Aligned Evaluation · An evaluation scheme that scores metrics matched to each stage of an agent's run: reasoning, action, final answer, and runtime quality
LLM-as-judge · Using another LLM to score qualities that simple rules can't capture, such as answer quality or reasoning quality

Figures we cannot republish

Figure 2: System overview. Task integrates benchmark management and execution support, Monitor provides unified agent access and instruments the runtime loop, and Evaluation performs multi-dimensional assessment with centralized result storage.

See the figures in the original paper →

Original abstract (English)

With the rapid advancement of large language models (LLMs), harnesses have become essential infrastructure for deploying agents across a wide range of domains. The fast-evolving harness ecosystem has also made rigorous capability evaluation increasingly important. However, efficiently building an end-to-end, systematic, and comprehensive evaluation pipeline remains a significant challenge. To address this challenge, we introduce A^2E (Agent Auditing Engine), an end-to-end evaluation engine designed for agent harnesses. A^2E leverages our newly proposed Agent Task Protocol (ATP) to enable the rapid integration of evaluation tasks with different harnesses. Through an automatically instrumented Monitor, it captures and generates standardized execution traces during experiments. In the Evaluation stage, A^2E systematically assesses harness capabilities using a suite of multidimensional metrics. Compared with correctness alone, these metrics provide a more fine-grained characterization of differences among harnesses in execution efficiency, tool use, task planning, and error recovery. Experiments conducted with A^2E further reveal that model-harness combinations exhibit substantial performance variation across different types of tasks, and that no single combination consistently outperforms all others across every task. These findings not only demonstrate the necessity of systematic evaluation but also provide useful guidance for the co-evolving of models and harnesses. Our code is available at https://github.com/datamllab/A2E.

Authors · Haoning Wang

Read on arXiv

Latest papers

All papers →

Latest from METAL LAB

Figures: Haoning Wang et al., arXiv:2608.07346, arxiv-nonexclusive