研究评测了12个模型,包括GPT-5.2、GPT-5.4、o3-pro、Claude Sonnet 4.6、Claude Opus 4.6、Grok-4.3等商用模型,以及DeepSeek-V4-Pro、GLM-5.1、Kimi-K2.6、Mistral-Large-3、Qwen3.5-9B、Qwen3.6-27B等开放权重模型,每个任务重复试验20次。
Figure 1: ThinkingBox overview and discovery–reliability gap. (A) ThinkingBox runs multi-turn interactions among a simulated user, an LLM agent, and isolated MCP-compatible tools, then evaluates terminal state, side effects, and dialogue with executable judges. More detailed pipeline can be found in Figure 2 (B) Across 507 tasks and 20 attempts per task, pass@20 is much higher than pass@1, while all-20 success is far lower, showing that successful trajectories are often discoverable but not reliably repeatable. Full results can be found in Table 4
Table 1: Comparison with representative agent and tool-use benchmarks. Thinkingbox-bench focuses on stateful business-domain workflows where the final verdict checks backend state, side effects, and dialogue outcome, while exposing tasks through MCP-compatible servers.
Benchmark
Primary domain
Tools/APIs
User dialogue
Stateful backend
Side-effect checks
MCP servers
SWE-bench (9)
Code repair
×
×
✓
×
×
BFCL (23)
Function calling
✓
×
×
×
×
ToolBench / API-Bank (25; 14)
API tool use
✓
×
×
×
×
WebArena / OSWorld (43; 36)
Web/desktop control
×
×
✓
×
×
AppWorld (30)
App APIs / coding agents
✓
×
✓
✓
×
MCP-Atlas (2)
Real MCP servers
✓
×
×
×
✓
τ-bench / τ2-bench (40; 3)
Domain APIs
✓
✓
✓
△
×
Thinkingbox-bench
Business tool workflows
✓
✓
✓
✓
✓
Figure 2: Overview of Thinkingbox. The sandbox orchestrates the interaction among a simulated user, LLM agents, isolated domain tools, side-effect extraction, and executable judges. The same trajectory-level verdict supports benchmark evaluation, failure analysis, and agent training.
Table 2: Key statistics for the ThinkingBox-Bench domains.
Retail
Booking
Insurance
Neobank
Consulting
Tasks
98
104
100
104
101
Backend systems
11
8
7
3
18
Databases (tables / rows)
22 / 86
17 / 98
14 / 72
20 / 151
30 / 231
Agent tools (write / read)
16 / 17
10 / 28
14 / 19
13 / 19
13 / 14
Policy (words)
945
3,684
2,471
3,392
1,747
Knowledge base (documents)
9
11
8
8
9
Actions per task
4.4 (1–10)
8.8 (4–19)
4.7 (1–11)
6.7 (3–12)
5.8 (1–13)
Evaluation
DB state
DB + rubrics
DB state
DB + rubrics
DB state
Figure 3: Pass@k progression of models used for evaluation
Table 3: Representative Thinkingbox-bench scenario patterns. Each scenario is instantiated as a concrete task with an initial backend state, available MCP-compatible tools, and executable checks.
Domain
Scenario
Required agent behavior
Executable checks
Retail / e-commerce
User asks to change or refund part of an order.
Identify the correct order/item, verify eligibility, request missing confirmation, and update only the relevant record.
Correct order state; refund/order side effect; no unrelated customer or item modified.
Travel / hospitality
User requests a booking change under date, room, or policy constraints.
Check reservation, availability, and change policy before modifying booking or explaining denial.
Correct reservation state; price/fee side effect if applicable; policy-compliant dialogue.
Auto insurance
User reports or updates a claim.
Verify policy and vehicle/incident details, collect missing information, and create or update the claim.
Correct ticket/claim state; no coverage mutation unless allowed.
Neobank support
User asks to dispute, freeze, or modify an account/card action.
Authenticate relevant account context, distinguish reversible and irreversible actions, and apply only valid updates.
Correct account/card state; required dispute/freeze side effect; no unrelated account changed.
Consulting IT/HR
Employee asks for access, HR, or internal support changes.
Verify role, approval, or employee record, then create ticket or update access according to policy.
Correct ticket/access state; approval and provisioning side effects; no unrelated records modified
Figure 4: pass^k progression of models used for evaluation
Table 4: Thinkingbox-bench pass@1 (%) by domain. Each task is evaluated with N=20 repeated trials, and scores are micro-averaged over trials and tasks. Size is reported as total/activated parameters for MoE models. The o3-pro row excludes 636 system/harness-error trials; its valid denominators therefore vary by domain. Task-cluster bootstrap intervals are reported in Appendix D.1.
Model
Size
Retail (98)
Auto (100)
Booking (104)
Bank (104)
Consulting (101)
Average
Proprietary models
GPT-5.4
–
76.33
62.65
68.13
65.34
54.60
65.36
GPT-5.2
–
70.20
22.40
53.70
51.15
34.06
46.28
o3-pro
–
37.94
2.96
24.16
24.37
14.75
20.60
Claude Sonnet 4.6
–
68.93
58.20
60.38
53.99
51.14
58.45
Claude Opus 4.6
–
74.90
14.65
28.89
38.03
34.21
37.91
Grok-4.3
–
43.93
2.60
15.14
1.78
9.55
14.38
Open-weights models
DeepSeek-V4-Pro
1.6T/49B
68.21
29.65
43.13
44.86
31.04
43.26
GLM-5.1
744B/40B
58.67
25.70
35.43
13.27
34.06
33.19
Kimi-K2.6
1T/32B
53.72
24.50
39.52
33.65
37.33
37.66
Mistral-Large-3
675B/41B
11.28
1.30
8.99
1.15
0.74
4.66
Qwen3.6-27B
27B
43.11
29.00
46.39
27.84
18.37
32.94
Qwen3.5-9B
9B
19.15
0.45
4.52
1.06
2.34
5.41
Figure 5: Domain failure-rate heatmap (%) on Thinkingbox-bench. Each cell is 100−pass@1 within the domain; darker red indicates higher failure rate.
Table 5: Failure mode breakdown (%) on Thinkingbox-bench. For each model, values indicate the percentage distribution of dominant failure types across failed trials.
Model
Tool Usage
No State-Changing Action
Incomplete User Resolution
Wrong State Update
GPT-5.4
89.6
1.6
0.5
8.3
Claude Sonnet 4.6
84.0
2.8
4.2
8.9
GPT-5.2
80.1
0.7
0.9
18.3
DeepSeek-V4-Pro
69.8
0.8
24.7
4.7
Claude Opus 4.6
78.1
3.0
10.2
8.7
Kimi-K2.6
85.2
1.3
12.1
1.4
GLM-5.1
88.1
1.3
3.5
7.0
Qwen3.6-27B
75.0
0.6
13.1
11.2
o3-pro
67.4
3.9
0.9
27.8
Grok-4.3
69.2
6.8
1.5
22.5
Mistral-Large-3
65.8
4.6
15.2
14.4
Average
77.5
2.5
7.9
12.1
Table 6: Composition of the 507-task evaluation set. Every case has executable backend-state checks. “Backend only” cases use no additional response rubric, whereas “Backend + rubric” cases also impose a binary final-response requirement.
Domain
Backend only
Backend + rubric
Task families represented in the final set
Retail / e-commerce
98
0
Delivery delay and exception handling; delivered-but-missing and return-to-sender cases; returns, refunds, exchanges, and warranty claims; installation scheduling and cancellation; order cancellation, promotions, and membership changes.
Travel / hospitality
89
15
Individual, corporate, and group booking changes; payment recovery; cancellations and refunds; corporate invoices and account benefits; group billing and services; hotel-partner verification and discrepancies; special requests and post-stay complaints.
Auto insurance
100
0
Billing extensions and arrangements; proof-of-insurance documents; adding, removing, or updating drivers and vehicles; first notice of loss and claim intake; listed-driver requests; reinstatement and policy cancellation.
Neobank internal IT support
89
15
Employee access and approval requests; password and account-security actions; production-incident access; hardware troubleshooting, assignment, replacement, and procurement; software and license requests; policy-information questions.
Consulting IT / HR support
101
0
Client-system and document access; software provisioning; expenses; hardware requests; employee onboarding; training enrollment; engagement and approval checks; corporate-travel policy and escalation.
Total
477
30
507 executable cases in total.
Table 7: Artifacts in a final Thinkingbox-bench case and the corresponding construction or review question.
Artifact
Contents
Review question
User goal g
Initial request plus facts the simulated user can provide during follow-up
Is the request natural, internally consistent, and resolvable without access to hidden evaluator information?
Initial state b0
Synthetic records in the domain backend, including existing tickets and related business objects
Do all referenced identifiers resolve, and do cross-system records agree before the agent acts?
Policy context
Domain operating manual and the fixed evaluation time
Does the policy determine eligibility, approvals, disclosures, and allowed actions without exposing the golden result?
Tools 𝒯
MCP-compatible read and write operations over the isolated domain services
Can the required evidence be retrieved and the intended outcome be executed using available tools?
User simulation 𝒰
Task-specific user role used for on-policy follow-up dialogue
Does the user provide only task-consistent facts and allow necessary clarification?
Checks 𝒞
Expected backend state and, for designated cases, final-response requirements
Does the evaluator accept the intended outcome and reject missing, wrong, or extra effects?
Table 8: Representative evaluator targets drawn from the final task families. These are outcome conditions, not prescribed action sequences.
Case type
Required outcome
Incorrect effects rejected
Retail return or delivery exception
Correct order, return/refund/replacement, and support-ticket state under the applicable policy
Wrong order or item, ineligible refund, duplicate ticket, incorrect ticket status, or missing compensation record
Booking modification or cancellation
Correct booking dates, room/board attributes, charges or refund, and ticket or hotel escalation when required
Modification without availability or policy support, wrong fee, partial group update, or confidential partner information disclosed
Insurance billing, policy, or claim request
Correct policy-linked extension, driver/vehicle change, claim, document, cancellation, or reinstatement state
Identity or eligibility bypass, wrong effective date, unintended coverage change, or incorrect ticket resolution
Internal access, hardware, or software request
Correct employee, approval, access, asset, procurement, notification, and ticket records
Excess privilege, bypassed approval, wrong assignee or device, duplicate request, or incomplete multi-system update
Consulting operations request
Correct engagement-linked access, expense, onboarding, training, hardware, or travel outcome
Table 9: Azure API inference parameters for the agent, fixed GPT-5.4-mini simulated user, and fixed GPT-5.4-mini response judge.
Parameter
Agent
Simulated User
Response Judge
temperature
1.0
0.3
0.0
max_completion_tokens
4096
4096
128
is_reasoning
True
False
False
reasoning_effort
medium
none
none
top-p
1.0
1.0
1.0
frequency penalty
0.0
0.0
0.0
number of completions
1
1
1
timeout (seconds)
600
600
600
Table 10: vLLM serving parameters for the locally hosted Qwen-series models.
Parameter
Value
port
8000
data parallel size
8
tensor parallel size
1
maximum model length
65,536
reasoning parser
qwen3
automatic tool choice
True
tool-call parser
qwen3_coder
API interface
Chat Completions
API endpoint
/v1/chat/completions
Table 11: Repeated-trial discovery and reliability on Thinkingbox-bench. CI denotes a 95% task-cluster bootstrap interval. pass^20 requires all 20 attempts to pass, while pass@20 requires at least one. 0/20 and 20/20 means the number of tasks that have never been passed, and has always passed during all 20 attempts. We include Qwen3-8B (38) and MiniMax-M2.5 (18).
Model
pass@1 [95% CI]
pass^20
pass@20
0/20 tasks
20/20 tasks
GPT-5.4
65.36% [62.23%, 68.51%]
25.25%
91.12%
45
128
Claude Sonnet 4.6
58.45% [55.23%, 61.66%]
20.12%
88.56%
58
102
GPT-5.2
46.28% [43.19%, 49.39%]
8.68%
84.81%
77
44
DeepSeek-V4-Pro
43.26% [40.27%, 46.20%]
3.55%
84.62%
78
18
Claude Opus 4.6
37.91% [34.48%, 41.40%]
13.81%
70.02%
152
70
Kimi-K2.6
37.66% [35.02%, 40.30%]
3.16%
84.22%
80
16
GLM-5.1
33.19% [30.23%, 36.14%]
2.76%
69.82%
153
14
Qwen3.6-27B
32.94% [30.20%, 35.75%]
2.37%
78.50%
109
12
o3-pro
20.60% [18.27%, 23.02%]
1.65%
61.54%
195
4
Grok-4.3
14.38% [12.34%, 16.50%]
0.00%
45.96%
274
0
Qwen3.5-9B
5.84% [5.39%, 6.31%]
0.03%
30.77%
351
0
Mistral-Large-3
4.66% [3.66%, 5.75%]
0.00%
25.44%
378
0
MiniMax-M2.5
0.19% [0.00%, 0.53%]
0.00%
0.59%
504
0
Qwen3-8B
0.13% [0.01%,0.34%]
0.00%
0.99%
502
0
Table 12: Retrospective evaluator ablation study. The first block counts failed trials that nevertheless appear complete under weaker response- or action-level evaluators. The second block reports evidence exposed by our executable database comparison.
Signal among executable-check failures
Trials
Share of failures
Failed trials accepted by weak observable evaluators
Clean termination
67,763
84.86%
Clean termination + state-changing tool call
64,586
80.88%
Above + no explicit error in final tool response
53,697
67.24%
Evidence reported by executable state/side-effect checks (ours)
Database hash mismatch
79,015
98.95%
Wrong field value
61,973
77.61%
Collection-length mismatch
45,510
56.99%
Missing expected state or side effect
20,250
25.36%
Extra unintended state or side effect
34,575
43.30%
Table 13: Failure distribution (%) by domain on Thinkingbox-bench. Each row reports percentages over failed trials in that domain.
Domain
Tool Usage
No State-Changing Action
Incomplete User Resolution
Wrong State Update
Retail
70.7
4.3
12.9
12.1
Travel
81.2
1.0
13.3
4.5
Auto insurance
48.2
3.8
15.1
32.9
Neobank internal IT
79.0
6.0
8.4
6.6
Consulting IT/HR
71.4
5.4
10.2
13.0
Table 14: Average trajectory-level interaction counts on Thinkingbox-bench. All columns are mean counts per trial, computed from parsed traces. Write calls are tool calls whose names indicate state-changing actions such as create, update, cancel, refund, or modify.
Model
Avg. msgs. / trial
Avg. tool calls / trial
Avg. write calls / trial
Avg. tool errors / trial
GPT-5.4
29.80
11.05
3.91
1.98
Claude Sonnet 4.6
34.41
10.16
3.62
1.68
GPT-5.2
36.40
11.71
4.74
2.10
DeepSeek-V4-Pro
38.94
11.68
3.78
2.34
Claude Opus 4.6
31.20
9.99
3.21
1.57
Kimi-K2.6
35.62
11.70
3.60
2.91
GLM-5.1
44.86
11.99
3.91
2.58
Qwen3.6-27B
28.40
11.11
3.82
1.68
o3-pro
30.58
7.52
2.78
1.35
Grok-4.3
24.15
8.74
3.13
2.06
Mistral-Large-3
25.72
9.22
3.15
1.81
MiniMax-M2.5
25.50
5.56
1.63
0.99
Qwen3.5-9B
31.31
11.75
3.61
0.85
Qwen3-8B
19.76
5.29
2.07
0.66
Table 15: Average token usage per model turn and average model turns per trial by model and domain. Each cell reports “Tokens/Turn x Turns”, with tokens in thousands; both averages use trials with token metadata. Tokens include input context and output tokens for each model invocation, with cached input counted once.
Recent agent benchmarks increasingly ground evaluation in executable environments, from code repair to web navigation, app APIs, and function calling. Yet completing consequential work beyond code requires more than producing a plausible response or valid tool call: agents must gather missing information over multiple turns, follow domain policies, coordinate dependent tools, and realize the correct persistent state transition without collateral effects. In this paper, we introduce Thinkingbox, a sandbox for tool-agent-user interaction that provides isolated MCP-compatible tool sessions, complete execution traces, and outcome evaluation over terminal backend state. Built on this sandbox, Thinkingbox-bench contains 507 policy-conditioned workflows across numerous scenarios, including retail, hospitality, auto insurance, neobank internal IT, and consulting IT/HR support. Each attempt is evaluated by task-specific executable checks that accept valid trajectories while rejecting wrong, missing, or extra effects; designated tasks additionally check required properties of the final response. Across proprietary and open-weight models, the strongest achieves 65.36% pass@1, but only 25.25% pass^20. Moreover, many failed trials show clean termination and valid state-changing actions, showing that response or tool-call-level signals are not clear proxies for end-to-end task completion. Thinkingbox-bench reveals a large gap between occasionally finding a successful trajectory and reliably completing stateful business tasks. We release both Thinkingbox and Thinkingbox-Bench: https://github.com/microsoft/thinkingbox
作者 · Zhuochun Li, Youngmin Ko, Ali Keramati, Nicola Ferri, Susana Palmaz Lopez Pelaez, Liang-Chun Tsai, Calvin Wang, Mirco Mi
