One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows
arXiv:2608.197412026-08-21
AI 비서가 업무를 한 번 성공했다고 믿을 수 있는 건 아니다 - 507개 업무 시나리오로 확인한 반복 신뢰성의 민낯
마이크로소프트 연구팀이 AI 에이전트가 환불 처리, 예약 변경, 보험 청구 갱신 같은 실제 업무를 끝까지 제대로 해내는지 확인하는 테스트 환경 Thinkingbox와 507개 업무 벤치마크를 공개했다. 같은 작업을 20번 반복시켜 보니 가장 뛰어난 모델도 한 번이라도 성공하는 비율은 91.12%였지만, 20번 모두 성공하는 비율은 25.25%에 그쳤다. 실패한 시도의 상당수가 겉보기엔 깔끔하게 끝났고 도구 호출도 문제없어 보였지만, 실제로는 백엔드 상태가 틀려 있었다.
무엇을 했나
Thinkingbox는 AI 에이전트, 가상 사용자, 업무용 도구(주문 시스템, 예약 시스템 등)가 서로 대화하고 작업하는 과정을 격리된 환경에서 재현하고, 마지막에 데이터베이스 상태와 부작용까지 확인하는 테스트 도구다.
이 위에서 소매, 여행/숙박, 자동차보험, 은행 내부 IT, 컨설팅 IT/HR 등 5개 분야에 걸친 507개의 업무 시나리오(Thinkingbox-bench)를 만들었고, 각 시나리오는 정답 하나가 아니라 여러 방법을 인정하되 잘못되거나 빠지거나 불필요한 결과는 걸러내는 실행 가능한 채점 기준을 갖췄다.
GPT-5.2, GPT-5.4, o3-pro, Claude Sonnet/Opus 4.6, Grok-4.3 등 상용 모델과 DeepSeek-V4-Pro, GLM-5.1, Kimi-K2.6, Qwen 계열 등 공개 가중치 모델 12종을 각 작업당 20번씩 시험했다.
가장 잘한 모델(GPT-5.4)도 한 번 시도했을 때 성공률(pass@1)은 65.36%였고, 20번을 모두 성공시켜야 인정하는 pass^20은 25.25%에 불과해, 우연히 성공하는 것과 매번 안정적으로 해내는 것 사이에 큰 격차가 있었다.
실패 사례를 분석하니 전체 실패의 77.5%가 '도구 오류를 만나고도 복구하지 못함'이었고, 12.1%는 '도구 호출은 성공했지만 잘못된 결과로 상태를 바꿈'이었다. 실패했는데도 대화는 매끄럽게 끝나고 상태 변경 행동도 있어 보이는 경우가 많아, 응답 내용이나 도구 호출 여부만으로는 진짜 성공을 판단할 수 없었다.
Figure 1: ThinkingBox overview and discovery–reliability gap. (A) ThinkingBox runs multi-turn interactions among a simulated user, an LLM agent, and isolated MCP-compatible tools, then evaluates terminal state, side effects, and dialogue with executable judges. More detailed pipeline can be found in Figure 2 (B) Across 507 tasks and 20 attempts per task, pass@20 is much higher than pass@1, while all-20 success is far lower, showing that successful trajectories are often discoverable but not reliably repeatable. Full results can be found in Table 4
Table 1: Comparison with representative agent and tool-use benchmarks. Thinkingbox-bench focuses on stateful business-domain workflows where the final verdict checks backend state, side effects, and dialogue outcome, while exposing tasks through MCP-compatible servers.
Benchmark
Primary domain
Tools/APIs
User dialogue
Stateful backend
Side-effect checks
MCP servers
SWE-bench (9)
Code repair
×
×
✓
×
×
BFCL (23)
Function calling
✓
×
×
×
×
ToolBench / API-Bank (25; 14)
API tool use
✓
×
×
×
×
WebArena / OSWorld (43; 36)
Web/desktop control
×
×
✓
×
×
AppWorld (30)
App APIs / coding agents
✓
×
✓
✓
×
MCP-Atlas (2)
Real MCP servers
✓
×
×
×
✓
τ-bench / τ2-bench (40; 3)
Domain APIs
✓
✓
✓
△
×
Thinkingbox-bench
Business tool workflows
✓
✓
✓
✓
✓
Figure 2: Overview of Thinkingbox. The sandbox orchestrates the interaction among a simulated user, LLM agents, isolated domain tools, side-effect extraction, and executable judges. The same trajectory-level verdict supports benchmark evaluation, failure analysis, and agent training.
Table 2: Key statistics for the ThinkingBox-Bench domains.
Retail
Booking
Insurance
Neobank
Consulting
Tasks
98
104
100
104
101
Backend systems
11
8
7
3
18
Databases (tables / rows)
22 / 86
17 / 98
14 / 72
20 / 151
30 / 231
Agent tools (write / read)
16 / 17
10 / 28
14 / 19
13 / 19
13 / 14
Policy (words)
945
3,684
2,471
3,392
1,747
Knowledge base (documents)
9
11
8
8
9
Actions per task
4.4 (1–10)
8.8 (4–19)
4.7 (1–11)
6.7 (3–12)
5.8 (1–13)
Evaluation
DB state
DB + rubrics
DB state
DB + rubrics
DB state
Figure 3: Pass@k progression of models used for evaluation
Table 3: Representative Thinkingbox-bench scenario patterns. Each scenario is instantiated as a concrete task with an initial backend state, available MCP-compatible tools, and executable checks.
Domain
Scenario
Required agent behavior
Executable checks
Retail / e-commerce
User asks to change or refund part of an order.
Identify the correct order/item, verify eligibility, request missing confirmation, and update only the relevant record.
Correct order state; refund/order side effect; no unrelated customer or item modified.
Travel / hospitality
User requests a booking change under date, room, or policy constraints.
Check reservation, availability, and change policy before modifying booking or explaining denial.
Correct reservation state; price/fee side effect if applicable; policy-compliant dialogue.
Auto insurance
User reports or updates a claim.
Verify policy and vehicle/incident details, collect missing information, and create or update the claim.
Correct ticket/claim state; no coverage mutation unless allowed.
Neobank support
User asks to dispute, freeze, or modify an account/card action.
Authenticate relevant account context, distinguish reversible and irreversible actions, and apply only valid updates.
Correct account/card state; required dispute/freeze side effect; no unrelated account changed.
Consulting IT/HR
Employee asks for access, HR, or internal support changes.
Verify role, approval, or employee record, then create ticket or update access according to policy.
Correct ticket/access state; approval and provisioning side effects; no unrelated records modified
Figure 4: pass^k progression of models used for evaluation
Table 4: Thinkingbox-bench pass@1 (%) by domain. Each task is evaluated with N=20 repeated trials, and scores are micro-averaged over trials and tasks. Size is reported as total/activated parameters for MoE models. The o3-pro row excludes 636 system/harness-error trials; its valid denominators therefore vary by domain. Task-cluster bootstrap intervals are reported in Appendix D.1.
Model
Size
Retail (98)
Auto (100)
Booking (104)
Bank (104)
Consulting (101)
Average
Proprietary models
GPT-5.4
–
76.33
62.65
68.13
65.34
54.60
65.36
GPT-5.2
–
70.20
22.40
53.70
51.15
34.06
46.28
o3-pro
–
37.94
2.96
24.16
24.37
14.75
20.60
Claude Sonnet 4.6
–
68.93
58.20
60.38
53.99
51.14
58.45
Claude Opus 4.6
–
74.90
14.65
28.89
38.03
34.21
37.91
Grok-4.3
–
43.93
2.60
15.14
1.78
9.55
14.38
Open-weights models
DeepSeek-V4-Pro
1.6T/49B
68.21
29.65
43.13
44.86
31.04
43.26
GLM-5.1
744B/40B
58.67
25.70
35.43
13.27
34.06
33.19
Kimi-K2.6
1T/32B
53.72
24.50
39.52
33.65
37.33
37.66
Mistral-Large-3
675B/41B
11.28
1.30
8.99
1.15
0.74
4.66
Qwen3.6-27B
27B
43.11
29.00
46.39
27.84
18.37
32.94
Qwen3.5-9B
9B
19.15
0.45
4.52
1.06
2.34
5.41
Figure 5: Domain failure-rate heatmap (%) on Thinkingbox-bench. Each cell is 100−pass@1 within the domain; darker red indicates higher failure rate.
Table 5: Failure mode breakdown (%) on Thinkingbox-bench. For each model, values indicate the percentage distribution of dominant failure types across failed trials.
Model
Tool Usage
No State-Changing Action
Incomplete User Resolution
Wrong State Update
GPT-5.4
89.6
1.6
0.5
8.3
Claude Sonnet 4.6
84.0
2.8
4.2
8.9
GPT-5.2
80.1
0.7
0.9
18.3
DeepSeek-V4-Pro
69.8
0.8
24.7
4.7
Claude Opus 4.6
78.1
3.0
10.2
8.7
Kimi-K2.6
85.2
1.3
12.1
1.4
GLM-5.1
88.1
1.3
3.5
7.0
Qwen3.6-27B
75.0
0.6
13.1
11.2
o3-pro
67.4
3.9
0.9
27.8
Grok-4.3
69.2
6.8
1.5
22.5
Mistral-Large-3
65.8
4.6
15.2
14.4
Average
77.5
2.5
7.9
12.1
Table 6: Composition of the 507-task evaluation set. Every case has executable backend-state checks. “Backend only” cases use no additional response rubric, whereas “Backend + rubric” cases also impose a binary final-response requirement.
Domain
Backend only
Backend + rubric
Task families represented in the final set
Retail / e-commerce
98
0
Delivery delay and exception handling; delivered-but-missing and return-to-sender cases; returns, refunds, exchanges, and warranty claims; installation scheduling and cancellation; order cancellation, promotions, and membership changes.
Travel / hospitality
89
15
Individual, corporate, and group booking changes; payment recovery; cancellations and refunds; corporate invoices and account benefits; group billing and services; hotel-partner verification and discrepancies; special requests and post-stay complaints.
Auto insurance
100
0
Billing extensions and arrangements; proof-of-insurance documents; adding, removing, or updating drivers and vehicles; first notice of loss and claim intake; listed-driver requests; reinstatement and policy cancellation.
Neobank internal IT support
89
15
Employee access and approval requests; password and account-security actions; production-incident access; hardware troubleshooting, assignment, replacement, and procurement; software and license requests; policy-information questions.
Consulting IT / HR support
101
0
Client-system and document access; software provisioning; expenses; hardware requests; employee onboarding; training enrollment; engagement and approval checks; corporate-travel policy and escalation.
Total
477
30
507 executable cases in total.
Table 7: Artifacts in a final Thinkingbox-bench case and the corresponding construction or review question.
Artifact
Contents
Review question
User goal g
Initial request plus facts the simulated user can provide during follow-up
Is the request natural, internally consistent, and resolvable without access to hidden evaluator information?
Initial state b0
Synthetic records in the domain backend, including existing tickets and related business objects
Do all referenced identifiers resolve, and do cross-system records agree before the agent acts?
Policy context
Domain operating manual and the fixed evaluation time
Does the policy determine eligibility, approvals, disclosures, and allowed actions without exposing the golden result?
Tools 𝒯
MCP-compatible read and write operations over the isolated domain services
Can the required evidence be retrieved and the intended outcome be executed using available tools?
User simulation 𝒰
Task-specific user role used for on-policy follow-up dialogue
Does the user provide only task-consistent facts and allow necessary clarification?
Checks 𝒞
Expected backend state and, for designated cases, final-response requirements
Does the evaluator accept the intended outcome and reject missing, wrong, or extra effects?
Table 8: Representative evaluator targets drawn from the final task families. These are outcome conditions, not prescribed action sequences.
Case type
Required outcome
Incorrect effects rejected
Retail return or delivery exception
Correct order, return/refund/replacement, and support-ticket state under the applicable policy
Wrong order or item, ineligible refund, duplicate ticket, incorrect ticket status, or missing compensation record
Booking modification or cancellation
Correct booking dates, room/board attributes, charges or refund, and ticket or hotel escalation when required
Modification without availability or policy support, wrong fee, partial group update, or confidential partner information disclosed
Insurance billing, policy, or claim request
Correct policy-linked extension, driver/vehicle change, claim, document, cancellation, or reinstatement state
Identity or eligibility bypass, wrong effective date, unintended coverage change, or incorrect ticket resolution
Internal access, hardware, or software request
Correct employee, approval, access, asset, procurement, notification, and ticket records
Excess privilege, bypassed approval, wrong assignee or device, duplicate request, or incomplete multi-system update
Consulting operations request
Correct engagement-linked access, expense, onboarding, training, hardware, or travel outcome
Table 9: Azure API inference parameters for the agent, fixed GPT-5.4-mini simulated user, and fixed GPT-5.4-mini response judge.
Parameter
Agent
Simulated User
Response Judge
temperature
1.0
0.3
0.0
max_completion_tokens
4096
4096
128
is_reasoning
True
False
False
reasoning_effort
medium
none
none
top-p
1.0
1.0
1.0
frequency penalty
0.0
0.0
0.0
number of completions
1
1
1
timeout (seconds)
600
600
600
Table 10: vLLM serving parameters for the locally hosted Qwen-series models.
Parameter
Value
port
8000
data parallel size
8
tensor parallel size
1
maximum model length
65,536
reasoning parser
qwen3
automatic tool choice
True
tool-call parser
qwen3_coder
API interface
Chat Completions
API endpoint
/v1/chat/completions
Table 11: Repeated-trial discovery and reliability on Thinkingbox-bench. CI denotes a 95% task-cluster bootstrap interval. pass^20 requires all 20 attempts to pass, while pass@20 requires at least one. 0/20 and 20/20 means the number of tasks that have never been passed, and has always passed during all 20 attempts. We include Qwen3-8B (38) and MiniMax-M2.5 (18).
Model
pass@1 [95% CI]
pass^20
pass@20
0/20 tasks
20/20 tasks
GPT-5.4
65.36% [62.23%, 68.51%]
25.25%
91.12%
45
128
Claude Sonnet 4.6
58.45% [55.23%, 61.66%]
20.12%
88.56%
58
102
GPT-5.2
46.28% [43.19%, 49.39%]
8.68%
84.81%
77
44
DeepSeek-V4-Pro
43.26% [40.27%, 46.20%]
3.55%
84.62%
78
18
Claude Opus 4.6
37.91% [34.48%, 41.40%]
13.81%
70.02%
152
70
Kimi-K2.6
37.66% [35.02%, 40.30%]
3.16%
84.22%
80
16
GLM-5.1
33.19% [30.23%, 36.14%]
2.76%
69.82%
153
14
Qwen3.6-27B
32.94% [30.20%, 35.75%]
2.37%
78.50%
109
12
o3-pro
20.60% [18.27%, 23.02%]
1.65%
61.54%
195
4
Grok-4.3
14.38% [12.34%, 16.50%]
0.00%
45.96%
274
0
Qwen3.5-9B
5.84% [5.39%, 6.31%]
0.03%
30.77%
351
0
Mistral-Large-3
4.66% [3.66%, 5.75%]
0.00%
25.44%
378
0
MiniMax-M2.5
0.19% [0.00%, 0.53%]
0.00%
0.59%
504
0
Qwen3-8B
0.13% [0.01%,0.34%]
0.00%
0.99%
502
0
Table 12: Retrospective evaluator ablation study. The first block counts failed trials that nevertheless appear complete under weaker response- or action-level evaluators. The second block reports evidence exposed by our executable database comparison.
Signal among executable-check failures
Trials
Share of failures
Failed trials accepted by weak observable evaluators
Clean termination
67,763
84.86%
Clean termination + state-changing tool call
64,586
80.88%
Above + no explicit error in final tool response
53,697
67.24%
Evidence reported by executable state/side-effect checks (ours)
Database hash mismatch
79,015
98.95%
Wrong field value
61,973
77.61%
Collection-length mismatch
45,510
56.99%
Missing expected state or side effect
20,250
25.36%
Extra unintended state or side effect
34,575
43.30%
Table 13: Failure distribution (%) by domain on Thinkingbox-bench. Each row reports percentages over failed trials in that domain.
Domain
Tool Usage
No State-Changing Action
Incomplete User Resolution
Wrong State Update
Retail
70.7
4.3
12.9
12.1
Travel
81.2
1.0
13.3
4.5
Auto insurance
48.2
3.8
15.1
32.9
Neobank internal IT
79.0
6.0
8.4
6.6
Consulting IT/HR
71.4
5.4
10.2
13.0
Table 14: Average trajectory-level interaction counts on Thinkingbox-bench. All columns are mean counts per trial, computed from parsed traces. Write calls are tool calls whose names indicate state-changing actions such as create, update, cancel, refund, or modify.
Model
Avg. msgs. / trial
Avg. tool calls / trial
Avg. write calls / trial
Avg. tool errors / trial
GPT-5.4
29.80
11.05
3.91
1.98
Claude Sonnet 4.6
34.41
10.16
3.62
1.68
GPT-5.2
36.40
11.71
4.74
2.10
DeepSeek-V4-Pro
38.94
11.68
3.78
2.34
Claude Opus 4.6
31.20
9.99
3.21
1.57
Kimi-K2.6
35.62
11.70
3.60
2.91
GLM-5.1
44.86
11.99
3.91
2.58
Qwen3.6-27B
28.40
11.11
3.82
1.68
o3-pro
30.58
7.52
2.78
1.35
Grok-4.3
24.15
8.74
3.13
2.06
Mistral-Large-3
25.72
9.22
3.15
1.81
MiniMax-M2.5
25.50
5.56
1.63
0.99
Qwen3.5-9B
31.31
11.75
3.61
0.85
Qwen3-8B
19.76
5.29
2.07
0.66
Table 15: Average token usage per model turn and average model turns per trial by model and domain. Each cell reports “Tokens/Turn x Turns”, with tokens in thousands; both averages use trials with token metadata. Tokens include input context and output tokens for each model invocation, with cached input counted once.
Model
Retail
Travel
Auto insurance
Neobank internal IT
Consulting IT/HR
GPT-5.4
14.2k × 8.1
27.4k × 7.2
14.6k × 9.6
21.3k × 6.7
18.2k × 7.4
Claude Sonnet 4.6
14.4k × 8.3
29.8k × 7.3
16.7k × 8.9
25.2k × 7.5
16.9k × 7.7
GPT-5.2
12.9k × 12.7
30.0k × 9.0
14.2k × 12.9
21.6k × 8.2
17.8k × 16.8
DeepSeek-V4-Pro
19.8k × 8.4
31.9k × 8.0
21.3k × 9.7
25.8k × 7.9
24.3k × 8.6
Claude Opus 4.6
14.9k × 8.7
27.3k × 7.6
15.8k × 10.1
25.1k × 7.0
17.1k × 7.6
Kimi-K2.6
13.1k × 10.0
23.3k × 8.6
13.6k × 11.9
19.2k × 9.0
17.3k × 10.2
GLM-5.1
17.7k × 10.0
28.5k × 9.8
20.1k × 12.3
25.2k × 10.6
22.5k × 11.6
Qwen3.6-27B
18.5k × 13.6
27.2k × 13.3
19.9k × 13.5
24.0k × 11.8
22.0k × 16.2
o3-pro
12.3k × 10.1
24.0k × 10.1
14.5k × 10.2
18.8k × 9.2
15.6k × 11.9
Grok-4.3
16.6k × 10.9
28.7k × 9.7
18.4k × 9.6
24.8k × 9.8
21.3k × 15.8
Mistral-Large-3
16.0k × 12.5
30.3k × 13.1
18.1k × 12.6
23.2k × 9.6
19.6k × 10.8
MiniMax-M2.5
17.0k × 8.8
16.4k × 6.2
13.9k × 5.7
14.9k × 6.0
17.1k × 7.2
Qwen3.5-9B
17.7k × 13.5
29.1k × 14.53
19.9k × 14.2
24.0k × 13.7
22.9k × 18.3
Qwen3-8B
15.7k × 8.87
23.3k × 11.6
18.7k × 8.9
20.8k × 8.29
19.9k × 9.11
왜 중요한가
고객센터 자동화나 사내 IT 지원처럼 실제 시스템 상태를 바꾸는 AI 에이전트를 도입하려는 기업이라면, 한두 번의 데모 성공만으로 신뢰해서는 안 된다는 근거를 제공한다. 도구 호출이 문법적으로 맞고 대화가 자연스럽게 끝났다고 해서 실제 업무가 올바르게 처리됐다고 볼 수 없다는 점을 구체적 수치로 보여준다.
이 논문의 용어
MCP · AI 에이전트가 외부 도구·서비스를 표준화된 방식으로 호출하도록 만든 연결 규격
pass@k · 같은 작업을 k번 시도했을 때 적어도 한 번 성공할 확률
pass^k · 같은 작업을 k번 시도했을 때 전부 성공해야 인정하는 엄격한 성공률
POMDP · 에이전트가 전체 상황을 다 볼 수 없는 상태에서 순차적으로 결정을 내리는 상황을 수학적으로 표현한 모델
부작용(side effect) · 작업 도중 의도치 않게 발생한 추가적인 데이터베이스 변경이나 결과
논문 원문 초록 (영문)
Recent agent benchmarks increasingly ground evaluation in executable environments, from code repair to web navigation, app APIs, and function calling. Yet completing consequential work beyond code requires more than producing a plausible response or valid tool call: agents must gather missing information over multiple turns, follow domain policies, coordinate dependent tools, and realize the correct persistent state transition without collateral effects. In this paper, we introduce Thinkingbox, a sandbox for tool-agent-user interaction that provides isolated MCP-compatible tool sessions, complete execution traces, and outcome evaluation over terminal backend state. Built on this sandbox, Thinkingbox-bench contains 507 policy-conditioned workflows across numerous scenarios, including retail, hospitality, auto insurance, neobank internal IT, and consulting IT/HR support. Each attempt is evaluated by task-specific executable checks that accept valid trajectories while rejecting wrong, missing, or extra effects; designated tasks additionally check required properties of the final response. Across proprietary and open-weight models, the strongest achieves 65.36% pass@1, but only 25.25% pass^20. Moreover, many failed trials show clean termination and valid state-changing actions, showing that response or tool-call-level signals are not clear proxies for end-to-end task completion. Thinkingbox-bench reveals a large gap between occasionally finding a successful trajectory and reliably completing stateful business tasks. We release both Thinkingbox and Thinkingbox-Bench: https://github.com/microsoft/thinkingbox
저자 · Zhuochun Li, Youngmin Ko, Ali Keramati, Nicola Ferri, Susana Palmaz Lopez Pelaez, Liang-Chun Tsai, Calvin Wang, Mirco Mi
