Databricks unveils enterprise document reasoning benchmark 'OfficeQA Pro V2'

Databricks unveiled OfficeQA Pro V2 on its company blog. It's a benchmark designed to evaluate "grounded reasoning" in enterprise settings — the ability to reason by anchoring answers to given source material. The post is structured in the following order: performance results from running several agents against the benchmark, the process of building the benchmark from scratch, the method used to scale it up with synthetic data, sample questions, and benchmark specifications. The "V2" in the name signals that this is a successor to an earlier version.

Why "grounded reasoning" is a hard problem

A model doing well on exam-style questions and a model reading and answering from company documents are two different things. The latter fails in much messier ways. It might plausibly fabricate a number that doesn't appear in the source material (hallucination), draw a conclusion from only the first of three documents where the answer is actually scattered across all three, or correctly locate the right supporting sentence but botch the calculation built on top of it. When the incoming material mixes formats — PDF contracts, quarterly earnings tables, presentation slides — the difficulty jumps another notch.

The problem is that existing benchmarks don't catch these failures well. Knowledge-test-style evaluations like MMLU ask what the model already knows. What enterprises actually want to know is "does it perform well when given our documents?" As a result, it has become common for companies to build their own internal evaluation sets from proprietary data — but since those aren't public, there's no way to compare across them. That's why public benchmarks keep emerging.

There's another wall here. This kind of question needs to be hand-crafted by humans to achieve real quality, but that caps you at a few hundred items. The fact that Databricks devoted a separate section to "scaling with synthetic data" reads as an answer to how they got past this bottleneck. Generating questions with a model solves the scale problem, but raises new questions about whether difficulty stays consistent and whether the answers are actually verifiable against the source material.

Why Databricks is releasing a benchmark

Databricks is a data platform company founded by the research team behind Apache Spark. Its core business is layering analytics and AI on top of enterprise data, and in recent years its center of gravity has shifted toward agents running on that data. On August 10, the company relaunched its "Data + AI World Tour," spanning 20 cities, and has also been rolling out case studies of agent orchestration with accounting and advisory firm CLA. Agents that work grounded in internal company data are effectively the core scenario for this company's products. For a company like this to build its own evaluation standard is also, in effect, a claim about how the problem itself should be defined.

So what changes

For anyone evaluating internal AI adoption, there's now one more reference point. It means the question "can this model handle our documents?" can be asked with a public set of questions rather than a rough guess. In terms of broader trends, this lines up with a shift in the model-performance race — from "what does it know" to "what can it do with the material it's given." There's also a growing trend of benchmarking itself becoming a product — Artificial Analysis opened early access sign-ups on August 10 for a benchmarking product suite aimed at developers. That said, for specifics on which agents scored what in this release, you'll need to check the results table in the original post directly.