MerchantBench: Benchmarking LLM Agents for Long-Term Coherence in E-Commerce Operations

What they did

1. Most existing agent benchmarks test short tasks with a clear pass/fail line; this paper instead targets 'Long-Term Coherence' - the ability to keep pursuing a goal sensibly over an extended period while updating decisions as evidence accumulates.
2. The environment is grounded in 98,843 real product records and 36,576 suppliers from 1688 (a large Chinese wholesale marketplace), simulated hourly across 8,760 steps (365 days), with 26 store-management tools (search, listing, pricing, finance checks, etc.) available to the agent.
3. A key design feature is mixed-latency feedback: placing a new order commits cash immediately, but bad outcomes like cancellations, refunds, or poor reviews surface only later and gradually erode the store's rating.
4. Eight models - GPT-5.6 Sol, Claude Opus 4.8, Qwen3.7-Max/Plus, GLM-5.2, DeepSeek-V4-Pro/Flash, and Kimi K2.6 - were each run three times under both ReAct (a minimal tool-calling controller) and Hermes (a fuller framework with code execution, planning, memory, and skill management), totaling 48 runs of 365 simulated days.
5. Results were compared against three human participants and a rule-based bot; the best LLM configuration still reached only 27.3% of the humans' mean final net assets, and traces showed patterns of declining activity over time (operational coherence loss) and drifting away from the profit goal (strategic coherence loss).

Findings

• The best LLM configuration (Qwen3.7-Max under Hermes) reached only 27.3% of the mean final net assets achieved by human participants.
• Averaged across eight models, Hermes produced 53.3% higher final net assets, 71.5% higher GMV, and 71.2% more orders than ReAct, with Hermes outperforming ReAct for seven of eight models (Kimi K2.6 was the exception, 4.1% lower under Hermes).
• Human participants sustained a Sustained Window Rate of 100%, while LLM configurations ranged from 10.6% to 99.4% under ReAct and 17.8% to 66.1% under Hermes, showing declining activity over the year.
• Some models entrenched flawed policies without revising them: one Hermes Claude Opus 4.8 run shrank its active listings from 47 to 3 based on a false belief that fewer listings would concentrate traffic, and one Hermes Kimi K2.6 run took no action in 355 of the remaining 523 decision windows after concluding on Day 104 that the store could not recover.
• Human participants widened their procurement price range as cash grew, moving from RMB 43.4-53.1 in the first three months to RMB 58.7-90.8 in the last three months, while GLM-5.2, DeepSeek-V4-Flash, and Kimi K2.6 kept comparatively flat price trajectories.

Where it can be used

• As an evaluation framework for autonomous agents meant to manage inventory, pricing, or cash flow over long, unsupervised periods, checking whether activity declines or goals drift over time.
• For comparing how different LLMs and agent frameworks (minimal tool-calling vs. ones with memory/skill management) affect long-horizon business performance.
• As a diagnostic method for testing whether an agent can trace delayed feedback (returns, refunds, reviews) back to earlier decisions and revise them.

Limits and open work

• The environment is a simulation grounded in 1688 marketplace data; real-world store operation involves additional regulatory, competitive, and customer-behavior factors not captured here.
• Results are based on three repeated runs per configuration, and some configurations (e.g., Qwen3.7-Max under Hermes) showed high variability (55.1% coefficient of variation), so results should be read with that instability in mind.
• The human baseline consists of only three participants without prior e-commerce experience, so the '27.3% of human performance' figure should not be generalized to expert human sellers.
• Model names referenced in the paper (GPT-5.6, Claude Opus 4.8, Qwen3.7 series, etc.) may not correspond to publicly known model versions, so the focus should be on the benchmark design rather than treating these as verified current model rankings.
• Why Hermes's added capabilities (code execution, skill creation) help some models but not others is discussed through case examples rather than a systematic causal analysis.

Why it matters

This gives concrete numbers to a real worry: LLM agents look competent on short tasks but can lose the thread over months-long autonomous operations like inventory or pricing management. Anyone designing long-running autonomous business agents can use this as an early signal of where failures accumulate - fading activity, premature give-up, and policies that never get revised despite new evidence.

Terms in this paper

• Long-Term Coherence · The capacity to keep pursuing a goal purposefully over a long period while adapting decisions to accumulated evidence
• POMDP (Partially Observable Markov Decision Process) · A decision-making framework where the agent only sees part of the true state and must act on incomplete information
• Sustained Window Rate (SWR) · The lowest share of scheduled decision windows, across any rolling 30-day period, in which the agent made at least one tool call
• Hermes / ReAct · ReAct is a minimal controller using only the 26 store tools; Hermes adds code execution, memory, and skill management on top
• Upstream Supplier Events / Downstream Order Outcomes · Fast-surfacing supplier-side events (price changes, delistings) versus slow-surfacing order-side outcomes (returns, refunds, bad reviews)

Latest papers

• SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?AI coding agents were tested on fixing real scientific software, and even the best one failed more than half the timearXiv:2608.197992026-08-21
• FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM ServingMaking sparse attention fast enough and accurate enough for real LLM serving, not just papersarXiv:2608.197582026-08-21
• PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM AgentsMaking customer-service AI agents follow the whole procedure, not just avoid one bad actionarXiv:2608.198612026-08-21
• EXIMO: VLM Guided Exploration of VLA PoliciesTeaching a robot new chores without human teleoperation, by letting a chatty AI supervise itarXiv:2608.198912026-08-21
• EnvHarness: Awakening Static Worlds for Agent LearningInstead of building new training worlds from scratch, this work adds a plug-in layer that reshapes existing ones around each agent's actual weaknessesarXiv:2608.198802026-08-21
• Remember, Verify, or Ask? Cross-Family Evaluation of Memory Commitment in LLM AgentsAI assistants would rather double-check facts than ask you a question, even when asking is the right callarXiv:2608.195642026-08-21
• SynFlow: A Multidimensional Diachronic Semantic Analysis ToolkitAn open-source tool that breaks down how a word's meaning changed, not just that it changedarXiv:2608.194722026-08-21
• Automated Summarization of Financial News Using Large Language Models and Retrieval-Augmented Generation: An Early Empirical Study (Fall 2023)Testing AI summaries of stock news, the simple approach beat the trendy retrieval-based onearXiv:2608.195262026-08-21

Latest from METAL LAB

• Sakana AI Signs Deal With Japan's Defense Ministry for Intelligence Analysis AI Trial
• Hermes Agent builds its own skills the more you use it
• Is training AI on copyrighted books legal? Courts are still fighting it out
• Chinese gray market sells Anthropic Claude tokens at 10% of list price
• Even the Best AI Runaway Response Plan Among Five Major Labs Scores Only 3