E-Commerce Bench

E-Commerce Bench is a test that hands an AI agent a virtual starting budget and has it run an online store by itself for a full simulated year, measuring how well it manages the business. The AI decides everything on its own — where to source products, what price to buy and sell at, and how much inventory to keep — and its score is based on how much money is left in the account after the year is over.

Most AI tests grade a single question with a single answer and stop there. This one is different. A decision made on "day one" affects the stock and cash available on "day two," and those effects pile up over 365 simulated days to produce the final result. The test environment includes a mechanism for haggling over prices with suppliers, surprise events like promotions or natural disasters, and a simulated marketplace stocked with thousands of products across different store types — creating pressure similar to actually running a real online store.

Alibaba built this benchmark, but the top score didn't go to an Alibaba model — it went to a rival company's AI. That twist, a company's own benchmark not being topped by its own model, is what made it notable.

News coverage framed it as "OpenAI beat Alibaba on Alibaba's own benchmark." OpenAI's GPT-5.6 Sol finished first, turning its starting capital into 14.31 times its value by year's end, while Alibaba's older Qwen3.5-Plus model went bankrupt in four out of five runs and ranked last. The easy-to-miss point: the company that designs a benchmark doesn't always win on it. Alibaba designed E-Commerce Bench, but an OpenAI model came out ahead.