OpenAI beats Alibaba on Alibaba's own benchmark

Alibaba's Qwen team unveiled a new benchmark called E-Commerce Bench on September 3. It hands an AI agent ¥100,000 and asks it to run an online store on its own for 365 days. The agent has to make every call itself — sourcing, negotiation, pricing, promotions, inventory management, cash flow — inside a market built on real e-commerce data. But the model that actually posted the best score wasn't one of Alibaba's own; it was OpenAI's GPT-5.6 Sol. Alibaba's older Qwen3.5-Plus, meanwhile, went bankrupt in four of five runs.

In plain terms, the test puts an AI in the shoes of an actual store owner and hands it a full year of management decisions. It has to decide, day after day for 365 days, where to source goods, what price to sell them at, and how much inventory to hold — and its grade at the end is simply whatever's left in the bank.

Most existing AI benchmarks score a single question with a single answer. E-Commerce Bench is built differently. As days pass, yesterday's decisions shape today's inventory and cash position, and those effects compound into whatever assets remain a year later. That's why it's being described as a test of long-horizon autonomous operation.

The test environment has four layers

At the agent loop layer, the LLM agent calls tools, advances the environment's clock by one day, receives a daily news alert, and then repeats the cycle for the next turn. Context management and persistent memory also run at this layer. The tool layer supplies seven tools: store operations, market research, supplier negotiation, inventory management, financial management, memory manipulation, and time advancement.

The environment layer is the heart of the test. A dynamic economic engine models consumer and sales behavior, and random events — promotions, natural disasters, supplier bankruptcies — can hit at any time. Money is tracked across three separate accounts: bank, platform, and escrow. On top of that sits a deterministic negotiation kernel: if the agent asks for a bulk order of 1,000 bottles at a unit price of 1.5, the supplier might push back and say it can't go below 1.8, and the haggling continues over multiple rounds. The final data layer contains 12 store types, 60 product categories, 576 suppliers, and 6,886 products.

Results: OpenAI on top, Alibaba's models near the bottom

Each model ran five simulations, and scores reflect average year-end assets expressed as a multiple of the ¥100,000 starting capital.

Model	Developer	Year-end asset multiple	Bar	Bankruptcies (out of 5)
GPT-5.6 Sol	OpenAI	14.31x	100	0
Claude Opus 4.8	Anthropic	4.98x	35	0
Qwen3.8-Max-Preview	Alibaba	4.16x	29	0
Qwen3.7-Max	Alibaba	1.65x	12	0
Qwen3.6-Plus	Alibaba	0.47x	3	0
Qwen3.5-Plus	Alibaba	0.01x	1	4

OpenAI's model ended up with more than three times the assets of Alibaba's own latest model, while one of Alibaba's older models burned through nearly all of its starting capital. The table also shows GPT-5.5, Claude Opus 4.6, and Gemini 3.1 Pro each going bankrupt in two of five runs. Under long-horizon autonomous management, the leaderboard shuffled in ways that don't match the usual language-ability rankings.

The data is open for anyone to check

Alibaba's Qwen team posted an introduction to the benchmark on its blog and published the full methodology in an arXiv paper. Results can also be viewed on the benchmark's dedicated site, ecbench.github.io. The simulation environment and scoring code are up on the github.com/QwenLM/E-CommerceBench repository, so anyone can plug in a different model and run the same test themselves.

Editor's take

What stands out here is that Alibaba published a result showing its own model didn't win a test it built itself. Benchmarks released by model developers have typically come packaged with results that put the developer's own model near the top, so this is the opposite of that pattern. It reads as a sign that Alibaba built the benchmark around a deterministic negotiation kernel and a dynamic economic engine rather than simple question-and-answer scoring, making it hard to game the answers in advance. That's also a signal that the benchmark's creator designed it to resist manipulation of the results.

Map this kind of long-horizon autonomous test onto real work, and the same pattern keeps showing up. Being good at short question-and-answer exchanges is a different skill from staying solvent through hundreds of accumulated decisions. When models that top language benchmarks keep going bankrupt in long simulations like this one, it means you shouldn't hand an agent budget authority based on language-ability rankings alone.

If an e-commerce or logistics company is weighing whether to let an AI agent handle ordering and pricing decisions, the number to look at in this table isn't the top model's multiple — it's the bankruptcy rate. How many times out of five a model wiped out its capital matters more than average return once real money is on the line. At this stage, it's safer to keep a human approval step in place and let the agent make decisions only within a limited budget.

It's likely other developers will plug their own models into this repository over the coming weeks and publish reproduced results. How much Alibaba manages to climb the leaderboard with the next version of Qwen will be the next test of how much credibility this benchmark holds.