Microsoft Publishes a Production-Scale Study of Copilot Coding Agents

The central finding is that coding agents are a different kind of load from chatbots. According to the authors, after a user sends one message the agent autonomously runs an average of 6.6 LLM calls before handing control back, and 87% of all LLM calls are initiated by the agent rather than the user. The ratio of LLM calls to tool executions is nearly 1:1, at 40.6 versus 43.6 per session on average. "The agentic loop enforces a strict 1:1 LLM-tool coupling. Serving systems must treat LLM calls and their corresponding tool invocations as an inter-dependent pair, not independent requests," the paper states.

Session sizes are extremely skewed. The median session has 3 user turns, 15 LLM calls, and lasts 4.2 minutes, but the mean is 6.1 turns, 40.6 calls, and 62.6 minutes, and the top 10% of sessions exceed 15 user turns and 100 LLM calls and run for more than three hours. Token asymmetry is starker still: the median prompt for a single LLM call is 68,000 tokens while the median output is 247 tokens, an input-to-output ratio above 275:1. Conversation history makes up 48% of the prompt, tool-call results 28%, and the system prompt only 14%. Context grows not because of instructions but because of the agent's own prior reasoning and actions.

For an engineer, the most practical section is the lifecycle of the KV cache. Within a turn, the prefix-cache hit rate jumps from 45% on the first call to 86% on the second and settles at 92 to 94% from the third onward. But at a turn boundary, when the user sends the next message, it drops by an average of 26 percentage points to 55%, and once the idle gap between turns passes 10 minutes the median hit rate collapses to 0 to 5%. Switching models is worse: the hit rate immediately after a switch is 8%. Switches occur in about 6.4% of sessions, most of them reactions to errors or rate limiting, and when users hand manual selection over to auto mode, 52% of the resulting switches are downgrades to a cheaper model.

Context compaction is a hidden cost. When the context window fills, the agent summarizes its history and rewrites the prompt. This happens in only 7.8% of sessions, but those sessions account for 44.2% of all tokens and 37.1% of LLM calls. A single compaction drops a median 72.8% of prompt tokens, cuts the cache hit rate by a median 66.1 percentage points, and the compaction call itself consumes a median 22% of the turn's execution time. The authors classify compaction as a cache reset on par with a model switch. The most extreme session compacted 40 times.

Tool failures multiply cost. According to the paper, tools fail in 9% of turns, and when they do the agent runs autonomous retry loops with a growing context, amplifying compute by up to 4x. Deep-loop turns with failures make up 9.1% of all turns but involve 36 LLM calls, four times the median. Success rates for run_command, run_build, and edit_file fall to roughly 73%, and a failed terminal command takes 48 times longer at the 95th percentile than a successful one. A failed build injects 7 to 8 times more tokens of compiler diagnostics into the context than a successful one. The same table shows that a single tool, get_file, accounts for 35% of all tool invocations.

Users are not one kind of user either. The authors sort them into five archetypes: readers at 41.7%, coders at 30.4%, terminal users at 11.0%, deep-loop users at 9.2%, and chat-only users at 7.6%. Tokens per turn span a 50x range, from 23,000 for chat-only users to 1.1 million for deep-loop users. When the cache is evicted, a deep-loop user must re-prefill 1.1 million tokens while a chat-only user needs 23,000. Applying the same cache timeout to everyone, the authors calculate, imposes the largest latency tax on the heaviest users. Weekend sessions are fewer but longer and heavier in calls, which the authors read as developers delegating more ambitious tasks when uninterrupted.

The signal for reclaiming resources in this structure is the turn boundary. Within a turn, the median idle time for containers and KV caches is 5.8 seconds and 1.2 seconds respectively, too short to reclaim anything, but across turns it stretches to 243 and 172 seconds, tens to more than a hundred times longer, and the median wait until the user's next message is 25.2 minutes. The authors built a LightGBM model that predicts, as a survival curve at each turn boundary, how long a session will stay idle. The full ensemble is roughly 2MB, inference takes under 3 milliseconds, its ROC-AUC for predicting whether an idle gap exceeds 60 seconds is 0.73, and it captures 86 to 90% of total idle time. Esha Choukse, the Microsoft Azure Research researcher who submitted the paper to arXiv, and her co-authors write: "Even when the model cannot pinpoint exactly how long a session will remain idle, it reliably identifies that the session will remain idle long enough to be worth reclaiming."

In a TPM's language, this paper is a demand to redesign serving infrastructure built around requests into infrastructure built around sessions. If a chatbot is a customer who walks up to a counter, hands over one form, and leaves, a coding agent is a customer who sits at the counter exchanging forms fifteen times, steps away for 25 minutes, and comes back. So the paper proposes pinning a session to a single model, pre-warming the new model's cache when a switch is unavoidable, compacting incrementally in a way that preserves the prefix, and a tiered SLO model that gives different cache-retention priority to different user archetypes. It also suggests that on APIs with a fixed cache-retention window, a provider can send a cheap keep-alive request just before the deadline when the predicted idle time straddles that limit, avoiding a full recompute.

The boundaries of the data are stated plainly in the paper. The traces are client-side records from Visual Studio and VS Code, so there are no server-side metrics such as GPU utilization; prompt text and code were not collected, so task success cannot be judged; and the sample is entirely from US regions, spanning at most three time zones. The authors say the characteristics held steady from January through June, while noting that a workload whose tools and models change weekly needs longitudinal tracking, and they plan to release sanitized traces soon. METAL has reported that GitHub began counting agent usage separately in its Copilot metrics API; this paper is the first to show, in numbers, what the agents behind those counts are actually doing.

The cost of coding agents comes from structure, not from the model. One message from a user swells into six calls and six tool executions inside the agent, whether the cache survives in between decides latency and cost, and a single failed tool quadruples the compute. Infrastructure for agents is not chatbot infrastructure patched over; it has to be designed from scratch around the unit called a session.