Course · Training · Workshop
AI Workloads on Kubernetes
Run AI and ML workloads in production on Kubernetes: GPU scheduling, model serving with KServe and vLLM, autoscaling with KEDA, orchestration with Kubeflow and Ray, and GitOps for models.
This hands-on course shows platform engineers, DevOps engineers, and SREs how to reliably operate AI and ML workloads on Kubernetes. The focus is not the model itself but the Kubernetes platform underneath – attaching and sharing GPU nodes, rolling out inference services declaratively, orchestrating training and batch jobs, and driving everything reproducibly through GitOps. Participants build a complete, production-ready AI platform stack on Kubernetes — from GPU scheduling with the NVIDIA GPU Operator, through model serving with KServe, Ray Serve, and vLLM, to autoscaling with KEDA and observability with DCGM and Prometheus. The course connects Kubernetes operations know-how with the specifics of AI workloads – expensive GPUs, large model artifacts, long-running batch jobs, and highly variable inference load.
Trainers
What participants say
Content
The course consists of the following topics and can be extended or adapted depending on the audience. The examples focus on widely used, cloud-native AI/ML tools and Kubernetes operations practices. For in-house courses there is a selection of technologies and deployment scenarios that can be chosen to better fit the audience.
– Fundamentals: AI workloads on Kubernetes:
- Why Kubernetes for AI/ML workloads
- Workload types: inference, training, batch, fine-tuning
- Specifics: GPUs, large artifacts, variable load
- Overview of the cloud-native AI ecosystem (CNCF, LF AI) – GPU nodes and scheduling:
- GPU nodes, node pools, taints, and tolerations
- NVIDIA GPU Operator: drivers, runtime, device plugin
- GPU sharing: time-slicing and Multi-Instance GPU (MIG)
- Resource requests, limits, and resource quotas for GPUs
- Scheduling strategies, affinity, and topology – Storage for models and data:
- Persistent volumes, claims, and CSI drivers
- Model caching and warm starts
- Model artifacts as OCI images and via registries
- Data pipelines and shared storage for training – Model serving on Kubernetes:
- KServe and Knative for serverless inference
- Running vLLM on Kubernetes
- Ray Serve for distributed serving
- NVIDIA Triton Inference Server
- Serving patterns: REST, gRPC, streaming, canary – Autoscaling for inference load:
- Horizontal and vertical pod autoscalers
- KEDA: event-driven autoscaling and scale-to-zero
- Cluster Autoscaler and Karpenter for GPU nodes
- Scaling on custom metrics (queue length, latency) – Orchestrating training and batch:
- Kubeflow Pipelines for ML workflows
- Ray on Kubernetes (KubeRay) for distributed training
- Argo Workflows for batch and training jobs
- Jobs, CronJobs, and gang scheduling – GitOps for models and platform:
- Declarative deployment with Argo CD and Flux
- Model rollouts, versioning, and canary
- Promotion across environments (dev, staging, prod)
- Reproducibility and rollbacks – Multi-tenancy and isolation:
- Namespaces, RBAC, and resource quotas
- Fair sharing of scarce GPU resources
- Network and security isolation between teams – Observability for AI workloads:
- GPU metrics with the DCGM exporter
- Prometheus and Grafana for cluster and inference metrics
- Monitoring latency, throughput, and tokens/s
- Logs and distributed tracing for inference pipelines
- Cost tracking and utilization analysis – Security and supply chain:
- Securing model images and artifacts
- Signatures, SBOMs, and admission control
- Secrets, network policies, and isolation
- Compliance aspects of operating AI workloads – Cost and efficiency:
- Optimizing GPU utilization
- Spot/preemptible nodes and bin-packing
- Scale-to-zero and on-demand provisioning
- Benchmarking and capacity planning – Best practices and future trends:
- Reference architectures for AI platforms on Kubernetes
- Team organization for AI platform teams
- Emerging standards and projects in the cloud-native AI space
The course combines theoretical foundations with intensive hands-on exercises. Participants build a production-ready AI platform stack on a real cluster and learn to operate AI workloads reliably and cost-effectively on Kubernetes.
The actual course content may differ from the above depending on the trainer, delivery, duration and the composition of participants.
Request this course in-house
Request a public date
No suitable public date? Register without obligation — once there is enough interest we schedule a new public date and let you know first.
More about AI workloads on Kubernetes
Kubernetes has become the standard platform for running containerized applications – and increasingly the platform for AI and ML workloads as well. Operating AI workloads, however, brings special requirements: GPUs are expensive and scarce, model artifacts are large, training and batch runs are long, and inference load varies strongly. With GPU scheduling, autoscaling, declarative deployment, and a rich ecosystem of operators, Kubernetes provides the building blocks to master these challenges.Further resources:
History
GPU support in Kubernetes began in 2017 with the device plugin framework, which made accelerators such as NVIDIA GPUs visible to the scheduler. The NVIDIA GPU Operator then automated rolling out drivers, runtime, and monitoring.
In parallel, projects emerged that standardized machine learning on Kubernetes: Kubeflow (2018) for ML pipelines, KServe (originally KFServing) for serverless model serving, and Ray for distributed training and serving. With the LLM boom from 2022, inference engines such as vLLM were added, which – run on Kubernetes and autoscaled with KEDA – today converge into a production-ready AI platform stack.