Summary
- Databricks released the "Big Book of AgentOps" on September 2, covering how to run AI agents in production
- DXC said it already has three agents in production and eight in pilot, and cut platform total cost of ownership by 30% after moving to Databricks
- Intercontinental Exchange's (ICE) text-to-SQL agent posted 77% syntax accuracy and 96% execution match across roughly 50 queries
Companies tend to stall at the same point when trying to move generative AI pilots into real production, according to Databricks. In a blog post, the company released an operational guide for AI agents called the "Big Book of AgentOps." It notes that getting to the design stage of an agent isn't the hard part — most teams stall on operational questions like what permissions to grant an agent, what happens when it fails, and who's on the hook for cost.
Why AgentOps became necessary
Databricks defines an AI agent not as a model that simply produces a response, but as a system that, during execution, chooses tools, pulls in enterprise data, calls APIs, and works through multiple steps on its own. Each of those capabilities is a point where something can go wrong. The examples cited include incorrect tool calls, overly broad permissions, and unexpected cost spikes.
To put it in context: the practice of putting predictive models into production used to be called MLOps. When language models like ChatGPT arrived, that evolved into LLMOps, which added prompt and model-version management. AgentOps is the next step after that — an operational discipline built for agents that choose their own tools and carry out tasks across multiple steps.
Where pilots get stuck
Databricks also identified common patterns among agent projects that never reach production. These include picking a use case that's too broad from the start, jumping into multi-agent orchestration before the complexity is justified, adding unnecessary reasoning loops, or pushing evaluation to the end. Even technically well-built projects often stall due to internal alignment problems — meaning executive sponsors, product owners, subject matter experts, and security, compliance, and finance teams all need to be pointed in the same direction.
A six-chapter, hands-on guide
The book is structured as six chapters that move from concept down to implementation. It starts by laying out agent types and anti-patterns, then covers four deployment architectures ranging from a single workspace to multi-account, multi-agent enterprise topologies. It also includes a seven-step roadmap running from team formation through data infrastructure, evaluation loops, and governance. Cost management gets particular emphasis here — a single request can balloon into multiple model calls through sub-agents, retries, and guardrail checks, which is why usage attribution and setting limits are described as essential.
The evaluation section applies principles of flow, feedback, and continuous learning — borrowed from "The DevOps Handbook" — to agent systems. The approach involves building golden evaluation datasets from actual execution logs and calibrating automated evaluators against subject matter expert judgment. The final chapter covers a RACI matrix for stakeholder alignment and communication cycles before and after launch.
Adoption cases, in numbers
The book also includes figures from companies that have actually deployed these systems.
| Company | Agent use case | Reported outcome |
|---|---|---|
| FactSet | Text-to-code knowledge agent | Moved from a single foundation model to a full agent system, improving accuracy by 44% |
| DXC Technology | Multi-agent portfolio | 3 in production, 8 in pilot/development, 30% reduction in platform TCO |
| ICE | Governed text-to-SQL | 77% syntax accuracy, 96% execution match across roughly 50 queries |
| Block | AI agent for seller operations | $10 million in productivity gains (per Databricks) |
Databricks said DXC is expanding its AI portfolio after moving to the platform, while Block uses Unity Catalog to manage data access across multiple business units, running both AI and operational workloads in the same environment.
What the Databricks platform provides
According to Databricks, as the number of agents grows, controls scattered across individual applications become harder to audit. The alternative it proposes is a platform-based approach that manages data access, model and tool usage, tracing, evaluation, and policy enforcement all in one place. On Databricks, MLflow handles evaluation and tracing, Unity Gateway manages model and tool traffic, and Unity Catalog handles discovery, permissions, lineage, and access control for data and AI assets.
Editor's take
It's no coincidence that Databricks is putting out an operations guide right now. Back in August, AWS released open-source agent skills for Bedrock and unveiled a local MCP bridge, and NVIDIA showed that swapping out just the harness — without touching the model — could push the same model's benchmark score from 30% to 100%. In other words, the race over raw model performance has largely played out, and the industry's attention has shifted to a different question: how do you run that model safely and predictably? It makes sense that a company like Databricks, which has always leaned more on the data and governance layer than on its own models, is stepping up its presence in this shift.
Case studies used to lean on a single accuracy number or benchmark score to make their point. The DXC, ICE, and Block examples in this book are all operational metrics instead. Putting TCO reduction, production-versus-pilot counts, and syntax accuracy alongside execution match rate side by side amounts to an admission that building an agent and being able to keep trusting it in daily use are two separate problems. In practice, when you put agent projects of this scale into real operation, the conclusion tends to be the same: locking down permission design and evaluation loops early saves far more time down the road than swapping models does.
For companies in Korea, the right sequence isn't jumping straight into designing multi-agent orchestration. It's picking one narrow use case, defining clear success metrics, and building an evaluation set from real execution logs first. Without establishing cost attribution and least-privilege access up front, it's hard to stop cost and risk from spiking together the moment sub-agents and retries start piling up. Over the next few months, expect other cloud and platform vendors beyond Databricks to roll out similar operational frameworks, shifting the center of gravity in the AI agent race further from model performance toward operational reliability.
Comments