Summary
- Databricks traced its internal AI agents' MCP tool calls and found seven small bugs behind roughly $1.2 million a year in losses — $499,000 in wasted tokens plus 12,000 hours of engineering time
- Combining Unity Gateway's tracing with natural-language queries in Genie One, the team found and fixed the bugs in just one hour
- The Jira tool failed 535 times a day and needed an average of 12 retries to recover, while the Google Drive tool failed on 49.6% of its calls
What Looked Like Just "Higher Usage"
Databricks widely uses AI agents internally for coding and other tasks, and as usage grew, so did costs. The problem was that there was no way to tell whether the rising costs reflected normal growth in usage or money leaking out from something gone wrong. When an agent's tool call failed, it didn't throw a visible error — it quietly retried, guessed, and eventually worked around the failure to finish the task anyway. From the outside, the task looked complete, and the aggregate dashboard simply showed token spend up 10%, which made it easy to mistake for ordinary growth in usage.
To put it simply, MCP (Model Context Protocol) is the standard AI agents use to call external tools like Jira or Google Drive. Databricks uncovered the hidden bugs by layering Genie One — which answers natural-language questions — on top of Unity Gateway, which automatically logs every MCP call.
Tracing Plus Natural-Language Queries: One Hour, Start to Finish
To confirm this suspicion, Databricks turned to Unity Gateway's tracing and Genie One. Unity Gateway automatically generates OpenTelemetry trace data for every MCP tool call, logging the tool name, argument values, whether an error occurred, token counts, latency, and session ID all in a single table. There was no need to add new instrumentation — because the gateway already sat on every call path, the data was ready to use immediately.
By connecting Genie One to the trace table and asking vague, natural-language questions like "it seems like the agent is spinning its wheels on Jira calls," the team got a ranked list of bugs back within minutes. Instead of writing SQL and digging through schemas, they got answers to plain questions. Most of the hour was spent reading those answers, not writing queries. The entire loop — finding the bugs, quantifying them, and fixing them with a coding agent — took about an hour.
What Went Wrong in Jira and Google Drive
The most frequent bug was in Jira's issues.search tool. The tool expected a comma-separated string for its fields parameter, like "key,summary,status," but the model — following JSON convention — naturally passed the value as an array instead. The server tried calling .split() on the array and just threw a Python traceback, giving the agent no useful information about what had gone wrong. The agent would then either repeat the same mistake or re-read the schema and flounder. Recovery took an average of 12 attempts, and 30% of sessions hit the same error more than once.
The Google Drive bug was bigger in scale. Calls to drive_file_get failed 49.6% of the time, because the model kept sending field names — like id, name, and mimeType — that looked plausible based on Google Drive's API documentation, but the tool's actual endpoint simply didn't accept them.
| Bug | Failure frequency | Avg. attempts to recover | Notes |
|---|---|---|---|
| Jira issues.search | 535/day | 12 | 30% of sessions recurred; $87,000 + 4,850 hours lost annually |
| Google Drive drive_file_get | 49.6% of all calls | Not specified in the source | Repeated failures from field-name mismatches |
Tools Need to Match How Models Actually Call Them
The surface-level takeaway is "write better error messages," and the data does support that. But the more interesting question is why the model was calling the tool "wrong" in the first place. Databricks argues that in most cases, the model wasn't actually wrong. MCP tool signatures are often deliberately loose — for the sake of generality, and because every parameter description is a token cost the model pays every single time. When a field's spec is ambiguous, the model fills the gap with a reasonable guess, and passing a field list as a JSON array was exactly that kind of reasonable guess. The problem was that the server accepted only one interpretation out of several plausible ones and simply choked on the rest.
The line "Knowing what to fix was always harder than fixing it" (Databricks blog) sums up the core of this case. The fixes themselves were simple: turning lists into strings, adding default values for missing parameters, and just accepting unexpected arguments instead of rejecting them.
What Other Teams Can Check Right Now
Databricks said it officially launched Unity AI Gateway on August 4. The tracing feature used in this case comes from the gateway's unified trace table, which is currently in beta. Any organization running agents connected to tools can run the same kind of investigation: first collect tool-call logs at a layer like the gateway, then attach a natural-language query interface on top of it. Databricks closed its post with the same advice: if you have agents wired up to your own tools, do the same thing — trace the calls, and ask Genie One what keeps going wrong.
Editor's Take
What makes this case matter isn't the dollar figure itself, but where the waste was hiding. When an agent swallows failures silently and papers over them with retries, the cost dashboard simply registers "usage went up." In a human-operated system, an error would show up on someone's screen and get noticed; an agent just works around it, so no alarm ever goes off. As agent spending keeps growing across organizations, this kind of silent failure is a structural problem likely to show up in more and more company budgets going forward.
Any team that has put agents into production will recognize this pattern. Token costs creep up a little more than expected at first, and only later — when someone finally opens the logs — does it become clear the same error has been repeating hundreds of times. What this case really demonstrates is that because the tracing infrastructure was already in place, the investigation itself shrank from a day-long project down to something you can just ask on the spot.
For teams running internal agents, the practical lesson is clear: it's fine to keep parameter specs loose when building tool servers, but the server has to absorb that looseness instead of returning a crash. And if agent costs are climbing, the first move shouldn't be staring at the aggregate dashboard — it should be breaking spend down by tool and by error type. In the months ahead, how agent tool servers handle errors could well become a real criterion in vendor selection and internal audits.
Comments