Microsoft unveils Maia 300 AI inference chip

Microsoft on October 12 unveiled Maia 300, its third-generation in-house AI inference chip, and said it will scale the Maia family to multi-gigawatt deployments starting in 2027, making it the primary inference system for its products and services. The company expects Maia 300 to deliver 40% more tokens per megawatt than leading GPU-based systems. In a full 96-accelerator rack configuration, rack-scale token throughput reaches up to 7x that of the previous-generation Maia 200.

The announcement rests on the second generation that is already running. According to Microsoft, Maia 200 now handles Microsoft Copilot production workloads and supports the in-house MAI team as it develops next-generation models, with GitHub Copilot and Microsoft Foundry to follow by the end of the year. The chip runs OpenAI models, Microsoft's own MAI models and leading open-source model families. It is being deployed across the United States, Europe and Australia, and the company said it delivers up to 1.7x more tokens per watt than leading GPU infrastructure for AI capabilities built into Teams, Copilot and Cowork. "Maia 200 is the best performing AI infrastructure in our fleet, measured on real-world metrics for our AI services," Scott Guthrie, executive vice president of the Cloud + AI Group, said in the announcement video.

Maia 300 is designed for inference, and in particular for the long-context, memory-intensive decode stage. Each chip carries eight HBM4 memory stacks that provide 18.0 TB/s of peak memory bandwidth within a 950-watt thermal design power (TDP). It is Microsoft's first multi-chiplet design, separating compute, memory-controller and I/O functions, and the compute chiplets are built on TSMC's N2P process. On-die SRAM serves as a software-managed scratchpad to relieve pressure on HBM, and low-precision formats such as enhanced MXFP4 cut the memory needed for computation and KV-cache storage. Saurabh Dighe, corporate vice president of Azure Hardware Systems Planning and Architecture, summed up the changes as five advances: multi-chiplet architecture, HBM4, new precision data types, a next-generation scale-up network and a larger, more power-efficient rack.

The rack-level numbers underpin the 7x throughput claim. A single Maia 300 rack packs 96 accelerators into a 167-kilowatt power envelope and uses 27.6 TB of HBM and 1.7 PB/s of aggregate HBM bandwidth as one high-speed scale-up domain. Links between accelerators are tuned for tensor parallelism, and a custom Ethernet-based AI transport extends the same topology from a single rack to datacenter scale. According to Microsoft's published specification sheet, verified by METAL, each chip has 288 GB of HBM and 3.2 TB/s of bidirectional scale-up bandwidth, and a single scale-up domain links roughly 12,000 Maia accelerators.

There is a reason the company frames its numbers in watts and megawatts. "The economics of AI are increasingly determined not by peak compute performance, but how efficiently an entire system can generate tokens at scale," said Rani Borkar, president of Azure Hardware Systems and Infrastructure. Microsoft sees data movement, not raw compute, as the bottleneck in large inference systems, and chose to move less data through a software-defined dataflow architecture and AI-assisted kernel optimization. In an analyst commentary sheet distributed by the company, chip analyst Austin Lyons noted that inference is increasingly a data movement problem because agents must keep their context as they plan, call tools and run inference again.

Tighter ties with the model organization are another pillar of the announcement. "Chip-model co-design is critical for us to build frontier superintelligence," said Mustafa Suleyman, CEO of Microsoft AI. He added that lessons from running MAI models on Maia 200 over the past few years went straight into the design of Maia 300. Jim Kleewein, a technical fellow at Microsoft 365, described Maia 200 as the foundational infrastructure behind Copilot and Cowork for summarizing information, composing content and completing multi-step tasks. The company said the two generations are designed to cover GitHub's coding experiences, model serving in Foundry, inference and data generation for the MAI team, and Dragon Copilot, which reduces clinicians' administrative burden.

The push to cut dependence through in-house chips extends beyond the cloud. METAL has reported that Waymo is building its own chip for its robotaxis to reduce its reliance on Nvidia. In the same commentary sheet, Pat Moorhead of Moor Insights & Strategy said the progression from Maia 100 to 200 and now 300 shows a maturing custom silicon strategy, and that at that scale custom silicon can go beyond performance to become a factor that improves the economics of Microsoft's own AI services.

Production deployments of Maia 300 begin in 2027. Microsoft said it will share more about its strategy next month at Microsoft Ignite 2026 in San Francisco and online. The 40% per-megawatt figure is still a pre-deployment expectation, while Maia 200's 1.7x per watt was measured in live services. The multi-gigawatt rollout in 2027 will be where the distance between those two numbers is tested in real datacenters.