工作日早上 7 点读 AI，周日早上 8 点读周报订阅邮件

METAL LAB

Business Arena: Benchmarking LLM Agents in a Realistic Marketplace

arXiv:2608.086212026-08-08

让15个顶尖AI智能体各自经营一家跨境网店,结果最好的模型最终净资产是最差模型的9倍

Business Arena是一个让AI智能体独立经营跨境批发网店的测试环境,数据基于阿里巴巴国际站真实采购信息以及经过权威来源校准的关税和市场条件。研究团队在相同条件下评测了15个顶尖模型,平均最终净资产从约20,856美元到188,488美元不等,相差9倍,超过一半的运行最终亏损。即便表现最好的模型也明显落后于人类设计的专家策略,因此研究团队没有只看最终分数,而是用技能层面的指标和逐个动作的归因分析来解释每个模型成功或失败的原因。

METAL LAB 解读图

Business Arena的评测流程

证据状态已报告实测结果

  1. 构建真实市场基于阿里巴巴国际站真实采购数据,结合经过校准的关税和需求条件,搭建跨境批发市场。
  2. 智能体长期经营15个模型使用60多种工具,在长时间跨度内处理采购、定价、广告、客服、合规和财务。
  3. 与专家策略对比以只使用智能体可见信息的人类设计策略为基准,估算市场中实际存在的机会规模。
  4. 技能与动作层面诊断将最终净资产拆解为技能层面指标和逐动作归因,揭示具体的强项、弱项以及盈亏的产生原因。
  5. 机制消融验证将预期行为与忽视策略、走捷径策略对比,确认高分反映的是真实的商业判断力。
这是 METAL LAB 制作的解读图，并非论文作者提供的原图。

他们做了什么

  1. 研究团队搭建了Business Arena,让智能体在长时间跨度内向供应商采购、向买家销售,经营一家跨境网店,商品、供应商报价、价格、最小起订量和交货周期均来自阿里巴巴国际站真实商品信息,需求周期、关税等市场条件则依据权威来源校准。
  2. 环境提供了60多种工具,覆盖市场调研、采购、库存、定价、广告投放、客户服务、合规和财务管理等全流程,让智能体自主决策和行动,而市场本身会因竞争对手调价、需求变化和供应中断持续演变,不受智能体行动影响。
  3. 研究团队在完全相同的世界条件下评测了15个顶尖模型,包括GPT-5.6 Sol、GPT-5.5、Claude Fable 5、Opus 4.6/4.8、Gemini 3.1 Pro/3.5 Flash、Qwen 3.7 Max等商用模型,以及GLM-5.2、Kimi K2.6/K3、DeepSeek V4 Pro、MiniMax M2.5/M3、Qwen-3.8-Max-Preview等开源权重模型,每个模型运行十次。
  4. 由于单一的最终净资产分数无法解释模型为何成功或失败,团队将模型表现与人类设计的专家策略进行对比以估算市场中存在的机会规模,用技能层面的指标揭示模型的强项与弱项,并将实际发生的盈亏追溯到产生它们的具体行动。
  5. 为确认高分反映的是真实的商业判断力,而不是利用模拟器漏洞或忽视任务中困难的部分,团队进行了机制消融实验,将预期行为与忽视策略、走捷径策略进行比较。
Figure 1: Model performance in Business Arena. Over the same long horizon, the strongest models more than double their capital, a middle group earns modest returns, and the weakest ones finish with less than they started.
Figure 1: Model performance in Business Arena. Over the same long horizon, the strongest models more than double their capital, a middle group earns modest returns, and the weakest ones finish with less than they started.
Table 1: Mechanism ablations. For each mechanism, the stronger of the neglect and shortcut policies is set as the baseline. Values report the mean change in final net worth relative to this baseline. The customer-service comparison uses three matched worlds; the remaining financial ladders use ten seeds.
MechanismIntended behaviorIntended policyNeglectShortcut or misuse
PortfolioUse demand to choose productsEvidence-guided +$63.6kBlind bulk buying -$14.6kBuy only cheap SKUs 0
Market eventsCheck signals before investingEvidence checked +$6.6kIgnore events -$17.3kFollow every rumor 0
PricingCover costs while sustaining salesFull-cost pricing +$50.3kPrice near cost -$58.1kExtreme markup 0
TariffsInclude tariffs when choosing marketsTariff-aware routes +$25.9kNo active routing 0Tariff-blind U.S. focus -$3.7k
Customer serviceUse buyer and product evidenceEvidence-based replies +$5.6kIgnore inquiries -$0.1kGeneric replies 0
Figure 2: Overview of arena design. The agent selects markets, purchases inventory, prices and lists products, learns from sales, and adapts its operation. It acts on partial observations while supplier disruptions, competitor repricing, and demand shifts create a changing market. Meanwhile, operational obligations persist and economic feedback remains delayed.
Figure 2: Overview of arena design. The agent selects markets, purchases inventory, prices and lists products, learns from sales, and adapts its operation. It acts on partial observations while supplier disruptions, competitor repricing, and demand shifts create a changing market. Meanwhile, operational obligations persist and economic feedback remains delayed.
Table 2: Mapping from business capabilities to arena features and agent-facing tools. Capability headers form the first layer, arena features describe the corresponding business problems, and the final column lists the tools through which agents gather evidence and act.
Arena feature and purposeAgent-facing tools
Decision-Making Under Uncertainty
Demand and events. Form market beliefs from structural demand, trends, calendars, public events, and policy shocks.get_base_demand_intel(), get_trends(), get_calendar(), get_world(), get_tariff_events().
Market feedback and competition. Learn from realized outcomes and observe rival offers without accessing competitors’ private strategies.get_orders(), get_competition().
Strategic Planning Under Constraints
Shop focus and initial portfolio. Select categories, compare opportunities, and decide how much capital to commit at opening.get_catalog(), get_store_focus(), set_store_focus(), submit_setup(), skip_setup().
Sourcing and supplier diligence. Search and rank offers, inspect supplier risk, and purchase inventory under cost, MOQ, quality, and lead-time constraints.get_supplier_catalog(), get_supplier_flags(), buy_supplier().
Capital, inventory, and recovery. Track deployed capital and obligations, finance expansion, and recover capital from weak positions.get_state(), get_products(), get_payables(), get_loans(), borrow(), repay_loan(), get_factoring(), factor_ar(), liquidate_inventory().
Insight-to-Action Alignment
Pricing and listing. Translate market beliefs and cost calculations into concrete offers across products and segments.get_listings(), list_product_on(), update_listing(), set_price_tiers().
Tariffs, shipping, and route economics. Calculate the route-specific cost stack and choose viable destinations and commercial terms.get_platforms(), get_countries(), get_shipping_rules(), get_tariff_table(), set_default_incoterm().
Compliance. Identify market-entry requirements, apply early enough to clear approval lead times, and avoid unauthorized trading.get_certifications(), get_compliance_status(), apply_certification().
Figure 3: Main leaderboard over 15 model families, averaged across ten runs under the same world condition. Dashed lines denote expert-designed strategies that use only agent-visible information.
Figure 3: Main leaderboard over 15 model families, averaged across ten runs under the same world condition. Dashed lines denote expert-designed strategies that use only agent-visible information.
Table 3: Mapping from business capabilities to arena features and agent-facing tools (continued).
Arena feature and purposeAgent-facing tools
Insight-to-Action Alignment (continued)
Advertising. Allocate demand-generation spend and revise it using observed full-funnel performance.get_ad_status(), set_ad_budget().
Cooperation & Competition
Customer service and buyer negotiation. Infer buyer needs, answer factual questions, and negotiate bulk transactions while protecting business value.get_inquiries(), reply_inquiry(), get_rfqs(), respond_rfq().
Returns and disputes. Respond to post-sale problems while managing refund, replacement, and escalation risk.get_return_requests(), respond_to_return(), get_returns(), dispute_return(), get_disputes(), resolve_dispute().
Supplier relationships and negotiation. Learn counterparty behavior, request better terms, and decide whether to accept supplier offers.get_supplier_relations(), request_quote(), get_supplier_quotes(), respond_quote().
Competitive response. Compare rival offers and adjust prices or demand-generation decisions as competitors change.get_competition(), update_listing(), set_price_tiers(), set_ad_budget().
Persistent Operation
Persistent state and daily feedback. Preserve observations and plans, inspect previous decisions, and advance the market after completing the current operating cycle.write_note(), read_notes(), delete_note(), end_round().
Workspace and automation. Read and revise persistent files, construct reusable analyses, and execute model-authored workflows across business functions.read(), write(), edit(), exec(), process().
Figure 4: Model diagnostic profiles. Models exhibit different strengths across operating fluency, capital deployment, selling, customer interaction, and compliance. Colors indicate cohort-relative performance from weaker to stronger.
Figure 4: Model diagnostic profiles. Models exhibit different strengths across operating fluency, capital deployment, selling, customer interaction, and compliance. Colors indicate cohort-relative performance from weaker to stronger.
Table 4: NPC seller archetypes grouped by pricing family. Top: competitor-aware strategies. Middle: cost-anchored strategies. Bottom: phase-switching and passive. The population is weighted toward high-liquidity archetypes (price_leader, follower, liquidator) that supply everyday buyer demand, with niche archetypes (event_sniper, dormant) appearing rarely. The distribution is fixed across all regions and seeds.
ArchetypePricing StrategyBehavior Summary
price_leaderundercut_medianTargets 95% of competitor median price; cuts further to 92% during peak festivals
followertrack_top_3Tracks average of 3 cheapest competitors with +2% offset
liquidatoraggressive_lowTargets 78% of competitor median; clearance pricing at 70% during festival endings
opportunistdynamic_demandRaises price when trailing demand exceeds 1.2× baseline; heavy pre-festival stocking
premiumcost-plusPrices at 3× cost basis; EU suppliers; holds firm during festivals
cross_bordercost-plusPrices at 2.5× cost basis; EU suppliers; stable across phases
wholesalecost-plusPrices at 1.3× cost basis; large inventory (35-day target), volume-driven
event_snipercost-plusPrices at 1.65× cost basis normally; spikes to 2.5× during peak (narrow 5-SKU catalog)
new_entrantundercut_until_ordersExtreme discounts (82% of median) until 50 orders, then switches to 1.05× cost basis
dormantstaticNever reprices; decays 5%/day after 10-day no-sale grace period
Figure 5: Core operating trade-offs. Successful agents identify opportunities and deploy capital into inventory that will sell (left), then preserve margin without pricing themselves out of the market (right). Crosses denote model-family means and lighter points individual runs.
Figure 5: Core operating trade-offs. Successful agents identify opportunities and deploy capital into inventory that will sell (left), then preserve margin without pricing themselves out of the market (right). Crosses denote model-family means and lighter points individual runs.
Table 5: Financial frictions and capital tools in Business Arena. The paper condition uses a mid-scale shop with $200 daily overhead. The mechanisms penalize idle capital, excess inventory, unfinished transactions, and poorly timed leverage, while allowing agents to pay explicit costs to recover or accelerate capital.
Financial factorCurrent ruleRationale
Operating drag$200 fixed overhead per day, plus 0.5% of on-hand inventory value.Penalizes passive operation and slow-moving stock; encourages sufficient throughput, disciplined purchasing, and inventory turnover.
Channel economicsPlatform commission is generally 5%–12% of gross, with volume discounts; eligible export orders receive a 9% rebate.Rewards pricing over the complete transaction-cost stack and selecting economically viable markets rather than maximizing gross revenue alone.
Short-term loansInterest compounds at 0.15% per day and rises to 2.5× the normal rate when overdue.Enables expansion when profitable opportunities exist, but penalizes borrowing without sufficiently fast and reliable capital recovery.
Supplier payablesOverdue balances accrue 0.10% per day, capped at 30% of the original invoice, and remain liabilities.Rewards planning around payment deadlines; penalizes sourcing commitments that the agent cannot finance.
Invoice factoringEligible receivables can be converted to cash at a 5%–18% discount determined by maturity and buyer credit.Lets agents accelerate cash recycling, while charging explicitly for liquidity obtained before customer payment.
Inventory liquidationInventory can be converted immediately to cash at 85% of cost basis.Provides a controlled way to exit bad positions and redeploy capital, but preserves a meaningful loss so that poor sourcing is not costless.
Final settlementEscrow and receivables recover at 97%, inventory at 85%, and liabilities remain at face value.Penalizes unfinished operating cycles and rewards converting inventory and receivables into cash before the episode ends.
(b) Margin and sell-through.
(b) Margin and sell-through.
Table 6: Repeated-run variation across the 150-run cohort. Confidence intervals use two-sided t-intervals over ten runs. Capital preservation counts runs ending at or above the $80,000 starting capital.
ModelMeanWithin-model SD95% CICapital preserved
Gemini 3.1 Pro$188,488$66,641[$140,816, $236,160]9/10
GPT-5.6 Sol$168,867$47,185[$135,113, $202,620]10/10
Fable 5$164,204$34,141[$139,781, $188,627]10/10
Gemini 3.5 Flash$125,952$28,321[$105,692, $146,212]10/10
GPT-5.5$117,481$18,816[$104,021, $130,941]10/10
Kimi K3$112,278$42,644[$81,773, $142,784]8/10
Opus 4.8$93,946$36,729[$67,672, $120,220]4/10
Opus 4.6$93,066$31,759[$70,347, $115,785]6/10
Qwen 3.8 Max$89,423$50,974[$52,958, $125,887]3/10
GLM 5.2$55,742$30,921[$33,623, $77,862]1/10
Kimi K2.6$52,533$27,568[$32,812, $72,254]1/10
Qwen 3.7 Max$47,956$31,145[$25,676, $70,236]0/10
MiniMax M3$43,064$35,218[$17,871, $68,258]1/10
DeepSeek V4 Pro$40,804$46,130[$7,805, $73,803]1/10
MiniMax M2.5$20,856$27,897[$900, $40,813]0/10
Figure 6: Compliance exposes deployment-critical reliability failures (left), while customer service reveals specialized strengths that do not follow the aggregate leaderboard (right).
Figure 6: Compliance exposes deployment-critical reliability failures (left), while customer service reveals specialized strengths that do not follow the aggregate leaderboard (right).
Table 7: Coverage and coupling of strategy-reserve components. Each component declares the public evidence or upstream plan it consumes, the decision object it produces, and the realized feedback that can revise it.
ComponentStrategy coverageConnection to the decision system
Evidence and memoryMarket signals, events, competition, supplier offers, route costs, firm state, and realized operating historyNormalizes current public evidence and retains bounded observations and prior plans for the next cycle.
Opportunity beliefsMarket-depth, trend- and event-aware, realized-velocity, unit-economics, inventory-risk, and customer-value signalsConverts heterogeneous evidence into demand, route, inventory, and customer beliefs consumed by the portfolio planner.
Capital and portfolioDiversified probing, conviction-weighted deployment, broad velocity portfolios, compact capital-efficient books, adaptive focus, and recovery phasesSets the cash reserve, risk budget, portfolio scope, and per-opportunity allocation that constrain all downstream spending.
SourcingCost-balanced, fast-turn, quality-led, risk-adjusted, and relationship-aware supplier selectionSelects supplier, quantity, timing, and terms within the portfolio allocation; fulfillment and realized quality revise supplier eligibility.
Pricing and listingMargin-preserving, competition-aware, volume-oriented, and premium offer policies with route-specific cost floorsTranslates sourced inventory and route economics into viable offers; conversion, margin, and inventory age update future prices.
ComplianceRoute checks, permit application, temporary listing pauses, and reopening after approvalGates sourcing and listing plans before trade; permit status, violations, and fines feed operational reliability.
Ads and promotionBounded experimentation, test–scale–stop rules, event-timed promotion, and recovery shutdownOperates only on viable, inventory-backed offers; full-funnel returns affect advertising and replenishment decisions.
Customer operationsInquiry and RFQ handling, fulfillment-aware responses, conservative negotiation, and, where enabled, retention-oriented CRMConverts incoming demand while protecting feasibility and contribution; service outcomes update customer-value and demand beliefs.
Finance and recoveryCash reserves, evidence-gated borrowing and repayment, position limits, and stale-inventory liquidationExpands deployment only when supported by visible economics and returns capital when continued ownership is no longer justified.
Feedback controlSell-through learning, winner scaling, loser pauses, route adaptation, focus revision, and portfolio rebalancingRoutes realized orders, margins, stock, advertising, service, and failures back to both beliefs and capital allocation.
(b) Customer-service outcomes.
(b) Customer-service outcomes.

研究结果

  • 在15个模型中,平均最终净资产从Gemini 3.1 Pro的188,488美元到MiniMax M2.5的20,856美元,相差9.0倍,以80,000美元起始资金为基准,51%的运行最终亏损,只有4个模型在所有运行中都保住了起始资金。
  • 在相同的市场环境下,表现最强的人类设计专家策略达到436,195美元,超过表现最好模型平均值的两倍,其做法是将市场证据与采购、定价、合规、客服和资本调配统一协调。
  • 十次运行平均值的可靠性较高(ICC=0.944),将结果拆分为不重叠的五次运行子集后排名基本保持一致(ρ=0.898),98.7%的情况下能复现同样的领先梯队。
  • Gemini 3.1 Pro、GPT-5.6 Sol和Fable 5的累计资本利用率分别达到199%、167%和150%,库存周转率也接近1.0,而Qwen 3.7 Max只投入了起始资金的22.3%,MiniMax M3的周转率仅为0.36,资金被滞销库存占用。
  • 专家策略在46%至61%的销售完成率下仍能保持58%至73%的订单利润率,而GPT-5.6 Sol和Opus 4.6的销售完成率达到94%至95%,但利润率仅为35%至39%,最终净资产依然落后于表现最好的专家策略,且所有专家策略的合规罚款均为零。
Figure 7: GPT-5.6 Sol behaves like a Volume Wholesaler. Its self-defined sourcing standards reject risky products, prioritize high-margin routes, and diversify inventory while retaining cash, supporting aggressive capital deployment and high sell-through.
Figure 7: GPT-5.6 Sol behaves like a Volume Wholesaler. Its self-defined sourcing standards reject risky products, prioritize high-margin routes, and diversify inventory while retaining cash, supporting aggressive capital deployment and high sell-through.

可应用场景

  • 开发电商或跨境采购自动化工具的团队可以借鉴这套诊断框架,提前检查智能体在采购、定价、客服、合规等哪个环节存在短板。
  • 这套方法可以为其他涉及延迟反馈和不断变化环境的长周期智能体评测(如库存管理、广告预算分配)提供设计思路参考。
  • 保存-分支-加载的状态化评测技术可用于设计从同一检查点比较不同决策方案的测试。
Figure 8: Gemini 3.1 Pro resembles a Premium House. Its route-aware repricer incorporates supplier cost, freight, tariffs, and competition while enforcing a 15% margin floor, producing higher margins at the cost of lower sell-through.
Figure 8: Gemini 3.1 Pro resembles a Premium House. Its route-aware repricer incorporates supplier cost, freight, tariffs, and competition while enforcing a 15% margin floor, producing higher margins at the cost of lower sell-through.

局限与待验证事项

  • 这些结果基于依据阿里巴巴数据和中美关税变化校准的特定模拟市场,论文并未提供证据表明结论可推广到真实市场或其他行业与地区。
  • 即便是表现最好的模型,最终净资产也不到最强人类专家策略的一半,说明当前智能体尚不足以被视为可靠的真实企业经营者。
  • 文中列出的应用场景是这种诊断方法的可能用途,并非已在实际部署环境中验证过的性能。
  • 测试时计算量分配的轨迹搜索实验只在Qwen 3.8 Max Preview一个模型上进行,尚不清楚该效果是否能推广到其他模型。
Figure 9: GPT-5.5 demonstrates adaptive recovery. After detecting zero cash and slow-moving inventory, it liquidates stock, reduces advertising, adopts FOB terms, and resets prices to release trapped capital and continue operating.
Figure 9: GPT-5.5 demonstrates adaptive recovery. After detecting zero cash and slow-moving inventory, it liquidates stock, reduces advertising, adopts FOB terms, and resets prices to release trapped capital and continue operating.

为什么重要

在这样一个受控的模拟市场里先做测试,能在不让真实资金、客户和合规风险受损的前提下,提前看清智能体在哪些环节强、哪些环节弱。对正在开发自动化选品、采购或定价工具的人来说,这提供了一份具体的诊断参考,指出资本调配、利润控制、合规和客服等方面还有哪些不足。

Figure 10: Realized value attribution within one trajectory. Two decisions by Gemini 3.5 Flash produce opposite outcomes: route-specific landed-cost reasoning preserves margin for SH-04, while underestimated delivery cost makes a TB-03 order loss-making.
Figure 10: Realized value attribution within one trajectory. Two decisions by Gemini 3.5 Flash produce opposite outcomes: route-specific landed-cost reasoning preserves margin for SH-04, while underestimated delivery cost makes a TB-03 order loss-making.

本文术语

  • Business Arena · 让AI智能体在长时间跨度内独立经营一家跨境批发网店的受控模拟评测环境
  • 机制消融实验 · 将模型的预期行为与忽视策略或利用漏洞的策略进行对比,以验证高分是否反映真实能力的实验方法
  • 技能层面指标 · 将单一最终得分拆解为库存管理、定价、客服等各项具体商业能力的衡量指标
  • 逐动作归因分析 · 将已发生的盈利或亏损追溯到具体导致它的行动(如采购、定价决策等)的分析方法
  • MCP调用 · 一种让模型交替进行推理和具体行动来调用工具的标准化方式

论文原文摘要（英文）

Running a business is a challenging form of intelligent work. Operators must infer opportunities from partial signals, commit capital under uncertainty, adapt to delayed outcomes in a changing market, and satisfy regulatory obligations before trading legally. Frontier LLM agents can increasingly complete complex workflows, yet business-related capabilities are rarely evaluated in existing agent benchmarks. We introduce Business Arena, a controlled environment where an AI agent runs a cross-border shop, buying from suppliers and selling to buyers over a long horizon. We ground the arena in real Alibaba.com sourcing data and market conditions calibrated from authoritative sources. Delayed and coupled consequences make individual business decisions difficult to judge, but their combined outcome is measurable through profit. Because profit alone cannot explain why an agent succeeds or fails, we compare agents with human-designed strategies to estimate available opportunity, use skill-level metrics to reveal underlying strengths and weaknesses, and trace realized gains and losses to the actions that produced them. We use mechanism ablations to establish that strong results reflect genuine business intelligence rather than neglect or simulator-specific shortcuts. We evaluate 15 frontier models and find a ninefold difference in mean final net worth. Even the best model falls behind human-designed strategies, indicating that business operation remains challenging for LLM agents. Skill-level analysis reveals operating styles, from margin-focused premium sellers to high-turnover wholesalers and customer-service specialists, while action-level attribution identifies the sourcing, pricing, and recovery decisions that create or destroy value. Together, Business Arena takes a first step toward a realistic and trustworthy testbed for evaluating end-to-end business agents.

作者 · Yijun Pan

在 arXiv 阅读

最新论文

全部论文 →

METAL LAB 最新报道

图片来源： Yijun Pan et al., arXiv:2608.08621, CC BY-SA 4.0