Business Arena: Benchmarking LLM Agents in a Realistic Marketplace

arXiv:2608.086212026-08-08

Fifteen frontier AI agents were put in charge of running an entire cross-border shop, and the best ended up with nine times more money than the worst

Business Arena is a testbed where an AI agent runs a cross-border wholesale shop end to end, grounded in real Alibaba.com sourcing data and real tariff and demand conditions. Across 15 frontier models evaluated under identical conditions, mean final net worth ranged nine-fold, from roughly $20,856 to $188,488, and more than half of all runs actually lost money. Even the best model fell well short of human-designed expert strategies, so the researchers used skill-level metrics and action-level attribution to explain why each model succeeded or failed rather than relying on the final score alone.

METAL LAB explanatory visual

How Business Arena Evaluates an Agent

Evidence statusMeasured results reported

Build a realistic marketA cross-border wholesale market is built from real Alibaba.com sourcing data plus calibrated tariff and demand conditions.
Agent runs the shop long-term15 models use 60+ tools to handle sourcing, pricing, advertising, customer service, compliance, and finance over a long horizon.
Compare against expert strategiesDeterministic human-designed strategies using only agent-visible information estimate how much opportunity existed in the market.
Diagnose skills and actionsFinal net worth is decomposed into skill-level metrics and action-level attribution to reveal specific strengths, weaknesses, and causes of gains or losses.
Verify with mechanism ablationsIntended behavior is compared against neglect and shortcut policies to confirm strong scores reflect genuine business judgment.

An explanatory diagram made by METAL LAB, not a figure supplied by the paper's authors.

What they did

The researchers built Business Arena, a controlled environment where an agent buys from suppliers and sells to buyers in a cross-border shop over a long horizon, with products, supplier terms, prices, minimum order quantities, and lead times drawn from real Alibaba.com listings and demand cycles, tariffs, and other conditions calibrated from authoritative sources.
More than 60 tools cover the full business cycle -- market research, sourcing, inventory, pricing, advertising, customer service, compliance, and finance -- letting the agent decide and act on its own, while the market keeps changing through competitor repricing, shifting demand, and supplier disruptions regardless of what the agent does.
Fifteen frontier models were run under identical world conditions, including proprietary systems (GPT-5.6 Sol, GPT-5.5, Claude Fable 5, Opus 4.6/4.8, Gemini 3.1 Pro/3.5 Flash, Qwen 3.7 Max) and open-weight systems (GLM-5.2, Kimi K2.6/K3, DeepSeek V4 Pro, MiniMax M2.5/M3, Qwen-3.8-Max-Preview), each for ten runs.
Because a single net-worth score cannot explain why an agent succeeded or failed, the team compared agents against human-designed expert strategies to estimate the opportunity available in the market, used skill-level metrics to expose strengths and weaknesses, and traced realized gains and losses back to the specific actions that produced them.
To confirm that strong scores reflect genuine business judgment rather than exploiting simulator quirks or neglecting hard parts of the task, the team ran mechanism ablations comparing intended behavior against neglect and shortcut policies.

Figure 1: Model performance in Business Arena. Over the same long horizon, the strongest models more than double their capital, a middle group earns modest returns, and the weakest ones finish with less than they started.

Table 1: Mechanism ablations. For each mechanism, the stronger of the neglect and shortcut policies is set as the baseline. Values report the mean change in final net worth relative to this baseline. The customer-service comparison uses three matched worlds; the remaining financial ladders use ten seeds.
Mechanism	Intended behavior	Intended policy	Neglect	Shortcut or misuse
Portfolio	Use demand to choose products	Evidence-guided +$63.6k	Blind bulk buying -$14.6k	Buy only cheap SKUs 0
Market events	Check signals before investing	Evidence checked +$6.6k	Ignore events -$17.3k	Follow every rumor 0
Pricing	Cover costs while sustaining sales	Full-cost pricing +$50.3k	Price near cost -$58.1k	Extreme markup 0
Tariffs	Include tariffs when choosing markets	Tariff-aware routes +$25.9k	No active routing 0	Tariff-blind U.S. focus -$3.7k
Customer service	Use buyer and product evidence	Evidence-based replies +$5.6k	Ignore inquiries -$0.1k	Generic replies 0

Figure 2: Overview of arena design. The agent selects markets, purchases inventory, prices and lists products, learns from sales, and adapts its operation. It acts on partial observations while supplier disruptions, competitor repricing, and demand shifts create a changing market. Meanwhile, operational obligations persist and economic feedback remains delayed.

Table 2: Mapping from business capabilities to arena features and agent-facing tools. Capability headers form the first layer, arena features describe the corresponding business problems, and the final column lists the tools through which agents gather evidence and act.
Arena feature and purpose	Agent-facing tools
Decision-Making Under Uncertainty
Demand and events. Form market beliefs from structural demand, trends, calendars, public events, and policy shocks.	get_base_demand_intel(), get_trends(), get_calendar(), get_world(), get_tariff_events().
Market feedback and competition. Learn from realized outcomes and observe rival offers without accessing competitors’ private strategies.	get_orders(), get_competition().
Strategic Planning Under Constraints
Shop focus and initial portfolio. Select categories, compare opportunities, and decide how much capital to commit at opening.	get_catalog(), get_store_focus(), set_store_focus(), submit_setup(), skip_setup().
Sourcing and supplier diligence. Search and rank offers, inspect supplier risk, and purchase inventory under cost, MOQ, quality, and lead-time constraints.	get_supplier_catalog(), get_supplier_flags(), buy_supplier().
Capital, inventory, and recovery. Track deployed capital and obligations, finance expansion, and recover capital from weak positions.	get_state(), get_products(), get_payables(), get_loans(), borrow(), repay_loan(), get_factoring(), factor_ar(), liquidate_inventory().
Insight-to-Action Alignment
Pricing and listing. Translate market beliefs and cost calculations into concrete offers across products and segments.	get_listings(), list_product_on(), update_listing(), set_price_tiers().
Tariffs, shipping, and route economics. Calculate the route-specific cost stack and choose viable destinations and commercial terms.	get_platforms(), get_countries(), get_shipping_rules(), get_tariff_table(), set_default_incoterm().
Compliance. Identify market-entry requirements, apply early enough to clear approval lead times, and avoid unauthorized trading.	get_certifications(), get_compliance_status(), apply_certification().

Figure 3: Main leaderboard over 15 model families, averaged across ten runs under the same world condition. Dashed lines denote expert-designed strategies that use only agent-visible information.

Table 3: Mapping from business capabilities to arena features and agent-facing tools (continued).
Arena feature and purpose	Agent-facing tools
Insight-to-Action Alignment (continued)
Advertising. Allocate demand-generation spend and revise it using observed full-funnel performance.	get_ad_status(), set_ad_budget().
Cooperation & Competition
Customer service and buyer negotiation. Infer buyer needs, answer factual questions, and negotiate bulk transactions while protecting business value.	get_inquiries(), reply_inquiry(), get_rfqs(), respond_rfq().
Returns and disputes. Respond to post-sale problems while managing refund, replacement, and escalation risk.	get_return_requests(), respond_to_return(), get_returns(), dispute_return(), get_disputes(), resolve_dispute().
Supplier relationships and negotiation. Learn counterparty behavior, request better terms, and decide whether to accept supplier offers.	get_supplier_relations(), request_quote(), get_supplier_quotes(), respond_quote().
Competitive response. Compare rival offers and adjust prices or demand-generation decisions as competitors change.	get_competition(), update_listing(), set_price_tiers(), set_ad_budget().
Persistent Operation
Persistent state and daily feedback. Preserve observations and plans, inspect previous decisions, and advance the market after completing the current operating cycle.	write_note(), read_notes(), delete_note(), end_round().
Workspace and automation. Read and revise persistent files, construct reusable analyses, and execute model-authored workflows across business functions.	read(), write(), edit(), exec(), process().

Figure 4: Model diagnostic profiles. Models exhibit different strengths across operating fluency, capital deployment, selling, customer interaction, and compliance. Colors indicate cohort-relative performance from weaker to stronger.

Table 4: NPC seller archetypes grouped by pricing family. Top: competitor-aware strategies. Middle: cost-anchored strategies. Bottom: phase-switching and passive. The population is weighted toward high-liquidity archetypes (price_leader, follower, liquidator) that supply everyday buyer demand, with niche archetypes (event_sniper, dormant) appearing rarely. The distribution is fixed across all regions and seeds.
Archetype	Pricing Strategy	Behavior Summary
price_leader	undercut_median	Targets 95% of competitor median price; cuts further to 92% during peak festivals
follower	track_top_3	Tracks average of 3 cheapest competitors with +2% offset
liquidator	aggressive_low	Targets 78% of competitor median; clearance pricing at 70% during festival endings
opportunist	dynamic_demand	Raises price when trailing demand exceeds 1.2× baseline; heavy pre-festival stocking
premium	cost-plus	Prices at 3× cost basis; EU suppliers; holds firm during festivals
cross_border	cost-plus	Prices at 2.5× cost basis; EU suppliers; stable across phases
wholesale	cost-plus	Prices at 1.3× cost basis; large inventory (35-day target), volume-driven
event_sniper	cost-plus	Prices at 1.65× cost basis normally; spikes to 2.5× during peak (narrow 5-SKU catalog)
new_entrant	undercut_until_orders	Extreme discounts (82% of median) until 50 orders, then switches to 1.05× cost basis
dormant	static	Never reprices; decays 5%/day after 10-day no-sale grace period

Figure 5: Core operating trade-offs. Successful agents identify opportunities and deploy capital into inventory that will sell (left), then preserve margin without pricing themselves out of the market (right). Crosses denote model-family means and lighter points individual runs.

Table 5: Financial frictions and capital tools in Business Arena. The paper condition uses a mid-scale shop with $200 daily overhead. The mechanisms penalize idle capital, excess inventory, unfinished transactions, and poorly timed leverage, while allowing agents to pay explicit costs to recover or accelerate capital.
Financial factor	Current rule	Rationale
Operating drag	$200 fixed overhead per day, plus 0.5% of on-hand inventory value.	Penalizes passive operation and slow-moving stock; encourages sufficient throughput, disciplined purchasing, and inventory turnover.
Channel economics	Platform commission is generally 5%–12% of gross, with volume discounts; eligible export orders receive a 9% rebate.	Rewards pricing over the complete transaction-cost stack and selecting economically viable markets rather than maximizing gross revenue alone.
Short-term loans	Interest compounds at 0.15% per day and rises to 2.5× the normal rate when overdue.	Enables expansion when profitable opportunities exist, but penalizes borrowing without sufficiently fast and reliable capital recovery.
Supplier payables	Overdue balances accrue 0.10% per day, capped at 30% of the original invoice, and remain liabilities.	Rewards planning around payment deadlines; penalizes sourcing commitments that the agent cannot finance.
Invoice factoring	Eligible receivables can be converted to cash at a 5%–18% discount determined by maturity and buyer credit.	Lets agents accelerate cash recycling, while charging explicitly for liquidity obtained before customer payment.
Inventory liquidation	Inventory can be converted immediately to cash at 85% of cost basis.	Provides a controlled way to exit bad positions and redeploy capital, but preserves a meaningful loss so that poor sourcing is not costless.
Final settlement	Escrow and receivables recover at 97%, inventory at 85%, and liabilities remain at face value.	Penalizes unfinished operating cycles and rewards converting inventory and receivables into cash before the episode ends.

Table 6: Repeated-run variation across the 150-run cohort. Confidence intervals use two-sided t-intervals over ten runs. Capital preservation counts runs ending at or above the $80,000 starting capital.
Model	Mean	Within-model SD	95% CI	Capital preserved
Gemini 3.1 Pro	$188,488	$66,641	[$140,816, $236,160]	9/10
GPT-5.6 Sol	$168,867	$47,185	[$135,113, $202,620]	10/10
Fable 5	$164,204	$34,141	[$139,781, $188,627]	10/10
Gemini 3.5 Flash	$125,952	$28,321	[$105,692, $146,212]	10/10
GPT-5.5	$117,481	$18,816	[$104,021, $130,941]	10/10
Kimi K3	$112,278	$42,644	[$81,773, $142,784]	8/10
Opus 4.8	$93,946	$36,729	[$67,672, $120,220]	4/10
Opus 4.6	$93,066	$31,759	[$70,347, $115,785]	6/10
Qwen 3.8 Max	$89,423	$50,974	[$52,958, $125,887]	3/10
GLM 5.2	$55,742	$30,921	[$33,623, $77,862]	1/10
Kimi K2.6	$52,533	$27,568	[$32,812, $72,254]	1/10
Qwen 3.7 Max	$47,956	$31,145	[$25,676, $70,236]	0/10
MiniMax M3	$43,064	$35,218	[$17,871, $68,258]	1/10
DeepSeek V4 Pro	$40,804	$46,130	[$7,805, $73,803]	1/10
MiniMax M2.5	$20,856	$27,897	[$900, $40,813]	0/10

Figure 6: Compliance exposes deployment-critical reliability failures (left), while customer service reveals specialized strengths that do not follow the aggregate leaderboard (right).

Table 7: Coverage and coupling of strategy-reserve components. Each component declares the public evidence or upstream plan it consumes, the decision object it produces, and the realized feedback that can revise it.
Component	Strategy coverage	Connection to the decision system
Evidence and memory	Market signals, events, competition, supplier offers, route costs, firm state, and realized operating history	Normalizes current public evidence and retains bounded observations and prior plans for the next cycle.
Opportunity beliefs	Market-depth, trend- and event-aware, realized-velocity, unit-economics, inventory-risk, and customer-value signals	Converts heterogeneous evidence into demand, route, inventory, and customer beliefs consumed by the portfolio planner.
Capital and portfolio	Diversified probing, conviction-weighted deployment, broad velocity portfolios, compact capital-efficient books, adaptive focus, and recovery phases	Sets the cash reserve, risk budget, portfolio scope, and per-opportunity allocation that constrain all downstream spending.
Sourcing	Cost-balanced, fast-turn, quality-led, risk-adjusted, and relationship-aware supplier selection	Selects supplier, quantity, timing, and terms within the portfolio allocation; fulfillment and realized quality revise supplier eligibility.
Pricing and listing	Margin-preserving, competition-aware, volume-oriented, and premium offer policies with route-specific cost floors	Translates sourced inventory and route economics into viable offers; conversion, margin, and inventory age update future prices.
Compliance	Route checks, permit application, temporary listing pauses, and reopening after approval	Gates sourcing and listing plans before trade; permit status, violations, and fines feed operational reliability.
Ads and promotion	Bounded experimentation, test–scale–stop rules, event-timed promotion, and recovery shutdown	Operates only on viable, inventory-backed offers; full-funnel returns affect advertising and replenishment decisions.
Customer operations	Inquiry and RFQ handling, fulfillment-aware responses, conservative negotiation, and, where enabled, retention-oriented CRM	Converts incoming demand while protecting feasibility and contribution; service outcomes update customer-value and demand beliefs.
Finance and recovery	Cash reserves, evidence-gated borrowing and repayment, position limits, and stale-inventory liquidation	Expands deployment only when supported by visible economics and returns capital when continued ownership is no longer justified.
Feedback control	Sell-through learning, winner scaling, loser pauses, route adaptation, focus revision, and portfolio rebalancing	Routes realized orders, margins, stock, advertising, service, and failures back to both beliefs and capital allocation.

Findings

Across 15 models, mean final net worth ranged from $188,488 (Gemini 3.1 Pro) down to $20,856 (MiniMax M2.5), a 9.0x gap, with 51% of all runs ending in a loss relative to the $80,000 starting capital, and only four models preserving starting capital in every trial.
The strongest human-designed expert strategy reached $436,195 in the same world -- more than twice the best model's mean -- by coordinating market evidence with sourcing, pricing, compliance, service, and capital allocation.
The ten-run means were reliable (ICC=0.944), and disjoint five-run subsets preserved the model ranking (rho=0.898) and recovered the same leading group in 98.7% of cases.
Gemini 3.1 Pro, GPT-5.6 Sol, and Fable 5 reached cumulative capital utilization of 199%, 167%, and 150% respectively while keeping inventory turnover near 1.0, whereas Qwen 3.7 Max deployed only 22.3% of starting capital and MiniMax M3's 0.36 turnover left capital stranded in slow-moving inventory.
Expert strategies accepted 46-61% sell-through while preserving 58-73% order margins, while GPT-5.6 Sol and Opus 4.6 reached 94-95% sell-through but only 35-39% margins and still fell behind the top expert strategies in final net worth, and every expert strategy incurred zero compliance fines.

Figure 7: GPT-5.6 Sol behaves like a Volume Wholesaler. Its self-defined sourcing standards reject risky products, prioritize high-margin routes, and diversify inventory while retaining cash, supporting aggressive capital deployment and high sell-through.

Where it can be used

Developers building e-commerce or B2B sourcing automation could use this kind of diagnostic framework to check which stage (sourcing, pricing, customer service, compliance) an agent is weak at before deployment.
The methodology could inform the design of other long-horizon agent benchmarks dealing with delayed feedback and changing environments, such as inventory management or ad-budget allocation tasks.
The save-fork-load stateful evaluation technique could be reused for designing tests that compare alternative decisions from the same checkpoint state.

Figure 8: Gemini 3.1 Pro resembles a Premium House. Its route-aware repricer incorporates supplier cost, freight, tariffs, and competition while enforcing a 15% margin floor, producing higher margins at the cost of lower sell-through.

Limits and open work

The results reflect a specific simulated market calibrated to Alibaba.com data and U.S.-China tariff conditions; no evidence is given that findings generalize to live markets or other industries and regions.
Even the best model reached less than half the net worth of the top human-designed strategy, so current agents cannot be considered reliable operators of a real business yet.
The listed applications describe possible uses of the diagnostic approach, not performance verified in real deployment settings.
The trace-search test-time-compute experiment was run on only one model (Qwen 3.8 Max Preview), so it is unclear whether the same gains generalize to other models.

Figure 9: GPT-5.5 demonstrates adaptive recovery. After detecting zero cash and slow-moving inventory, it liquidates stock, reduces advertising, adopts FOB terms, and resets prices to release trapped capital and continue operating.

Why it matters

Testing agents in this kind of controlled marketplace first, before letting them run a real business, reveals where they are strong or weak without putting real capital, customers, or legal compliance at risk. Anyone building automated storefront, sourcing, or pricing tools now has a concrete diagnostic showing which specific capabilities -- capital deployment, margin discipline, compliance, customer service -- still need work.

Figure 10: Realized value attribution within one trajectory. Two decisions by Gemini 3.5 Flash produce opposite outcomes: route-specific landed-cost reasoning preserves margin for SH-04, while underestimated delivery cost makes a TB-03 order loss-making.

Terms in this paper

Business Arena · A controlled simulation where an AI agent independently runs a cross-border wholesale shop over a long horizon for evaluation
mechanism ablation · An experiment that compares intended agent behavior against neglect or shortcut policies to check whether good scores reflect real skill
skill-level metrics · Metrics that break a single final score into separate business capabilities such as inventory management, pricing, and customer service
action-level attribution · Analysis that traces a realized profit or loss back to the specific action (sourcing, pricing, etc.) that produced it
MCP calls · A standardized way for the model to invoke tools that lets it interleave reasoning with individual actions

Original abstract (English)

Running a business is a challenging form of intelligent work. Operators must infer opportunities from partial signals, commit capital under uncertainty, adapt to delayed outcomes in a changing market, and satisfy regulatory obligations before trading legally. Frontier LLM agents can increasingly complete complex workflows, yet business-related capabilities are rarely evaluated in existing agent benchmarks. We introduce Business Arena, a controlled environment where an AI agent runs a cross-border shop, buying from suppliers and selling to buyers over a long horizon. We ground the arena in real Alibaba.com sourcing data and market conditions calibrated from authoritative sources. Delayed and coupled consequences make individual business decisions difficult to judge, but their combined outcome is measurable through profit. Because profit alone cannot explain why an agent succeeds or fails, we compare agents with human-designed strategies to estimate available opportunity, use skill-level metrics to reveal underlying strengths and weaknesses, and trace realized gains and losses to the actions that produced them. We use mechanism ablations to establish that strong results reflect genuine business intelligence rather than neglect or simulator-specific shortcuts. We evaluate 15 frontier models and find a ninefold difference in mean final net worth. Even the best model falls behind human-designed strategies, indicating that business operation remains challenging for LLM agents. Skill-level analysis reveals operating styles, from margin-focused premium sellers to high-turnover wholesalers and customer-service specialists, while action-level attribution identifies the sourcing, pricing, and recovery decisions that create or destroy value. Together, Business Arena takes a first step toward a realistic and trustworthy testbed for evaluating end-to-end business agents.

Authors · Yijun Pan

Read on arXiv

Latest papers

All papers →

Latest from METAL LAB

Figures: Yijun Pan et al., arXiv:2608.08621, CC BY-SA 4.0