Summary
- NVIDIA's developer blog has published a tutorial showing how to combine KAI Scheduler and vCluster to set up fully isolated per-team Kubernetes tenant clusters on top of a single shared GPU infrastructure
- Each team gets its own API server, RBAC, CRDs, and cluster-admin privileges, while actually sharing the same underlying GPU nodes
- The tutorial is designed to be reproducible with a single NVIDIA L40S GPU and three teams, and the same approach is said to scale to environments with hundreds of nodes
NVIDIA's developer blog has introduced an architecture that lets multiple teams run fully isolated Kubernetes clusters on top of a single shared GPU infrastructure. The approach addresses a tradeoff: giving each team a dedicated cluster creates more isolation overhead than necessary, while having multiple teams share a single cluster raises problems like CRD version conflicts, overlapping RBAC, and contention over GPU capacity allocation.
Combining KAI Scheduler and vCluster
KAI Scheduler is a topology-aware scheduler built for AI workloads, supporting per-team quotas and dynamic GPU allocation, and it runs alongside the default kube-scheduler. Only pods explicitly assigned schedulerName: kai-scheduler are handled by it, while everything else continues to be scheduled as usual. It integrates with the NVIDIA GPU Operator and is reportedly capable of handling clusters with thousands of nodes.
vCluster provisions virtual clusters on top of physical nodes, each with its own API server, controllers, data store, and scheduler for every team. From a tenant's perspective, this isolation is indistinguishable from having a dedicated cluster, but in practice the underlying nodes and GPUs remain shared, eliminating the need to physically partition hardware.
A Reproducible Test Setup
The tutorial is built around a small-scale environment with a single NVIDIA L40S GPU split among three teams, making it accessible even to users with limited resources. The same approach is said to work identically in environments with hundreds of GPU nodes and dozens of teams, scaling by expanding node pools, queue hierarchies, and the number of tenant clusters. After completing the exercise, users can confirm that all three teams run actual GPU pods in their own isolated tenant clusters while remaining unable to see each other's workloads.
Comments