Course · Training · Workshop
Platform Engineering for AI Applications
Platform engineering for AI applications: Deployment, operation, and optimization of large language models and AI workloads with inference engines, LLM gateways, monitoring, and LLMOps.
This hands-on course provides platform engineers, DevOps engineers, and SREs with the necessary knowledge and skills to professionally operate AI applications and large language models. Participants learn how to build LLM infrastructure, configure inference engines, optimize performance, and operate AI workloads in production. The course covers the full spectrum from self-hosted model inference through optimized inference engines (vLLM, TensorRT-LLM) to LLM gateways, monitoring, and fine-tuning. Participants will work with real AI infrastructure projects including model deployment, GPU resource management, batching strategies, caching optimization, and observability. The course will address inference server configuration, performance tuning, cost optimization, monitoring setup, and LLMOps practices.
Trainers
What participants say
We had high expectations from the DevOps AI training from letsboot.ch and they were all met or overachieved. We had 3 great days with enough theory, the right amount of practical tasks as well as a lot of real-life examples and answers to our questions. For us the real life experience was gold. The lab environment is very well prepared and organized, you have everything there with a pretty modern interface and quite intuitive if you know your way a bit around a linux console. This was a great training and when we need trainings, letsboot.ch will be the first I ask.
Head of Digital Services Operations Technology and Infrastructure
These customers booked courses in the same topic cluster.More customers →
Content
The course will consist of the following topics and may be extended or adapted based on the audience. The examples in the course will focus on widely used AI/ML frameworks and platform engineering practices. For in-house courses there is a selection of technologies and deployment scenarios which can be chosen to better fit the audience.
– Basics and Theory of AI and Large Language Models:
- History of Artificial Intelligence and Neural Networks
- LLM architectures (Transformer, Attention Mechanism)
- Model types (Instruct, Chat, Code, Embedding)
- Wording: Tokens, Quantization, Weights, Layers, Context Window
- Open Source vs. Open Weight vs. Commercial Models
- Multimodality and other Neural Networks – AI/ML Frameworks and Standards:
- ONNX and model interchange formats
- Hugging Face Transformers and Model Hub
- Standards: MCP (Model Context Protocol), OpenAI API compatibility – Self-Hosted Model Inference and Deployment:
- Ollama for quick model deployment
- On-premises model weight storage
- Model download and management
- First inference tests and prompting – Inference Engines and Performance Optimization:
- Inference engines: vLLM, TensorRT-LLM, Triton Inference Server
- Batching strategies (Static vs. Dynamic Batching)
- Caching strategies (KV-Cache, Prompt Caching)
- Parallelization (Tensor Parallelism, Pipeline Parallelism)
- Quantization (INT8, INT4, GPTQ, AWQ, GGUF)
- Performance metrics (Latency, Throughput, TTFT, TPOT) – GPU Infrastructure and Resource Management:
- GPU selection and sizing for LLM workloads
- CUDA, ROCm and GPU driver management
- GPU sharing and multi-tenancy
- Kubernetes GPU scheduling and resource quotas
- Cost optimization for GPU workloads – LLM Gateways and API Management:
- LiteLLM: Installation, configuration and backends
- Model load balancing and fallback strategies
- Rate limiting and cost tracking
- API key management and authentication
- Multi-model routing and A/B testing – Guardrails and Content Filtering:
- Concepts and necessity of guardrails
- Content filter implementation
- Input/output validation
- Safety and compliance – Model Serving Patterns and Architectures:
- REST API and gRPC for model serving
- Synchronous vs. asynchronous inference
- Streaming responses and server-sent events
- Model versioning and canary deployments
- Blue-green deployments for models – Monitoring and Observability for AI Workloads:
- Metrics: Inter-token latency, throughput, token/s
- Prometheus integration for LLM metrics
- Grafana dashboards for AI monitoring
- OpenTelemetry for distributed tracing
- Log aggregation for inference logs
- Cost tracking and resource utilization – Retrieval Augmented Generation (RAG) Infrastructure:
- Vector databases (Pinecone, Weaviate, Qdrant, Milvus)
- Embedding models and deployment
- Chunking strategies and indexing
- Hybrid search and reranking
- RAG pipeline orchestration – Fine-Tuning and Model Adaptation:
- Fine-tuning concepts and use cases
- LoRA (Low-Rank Adaptation) and QLoRA
- Full fine-tuning vs. parameter-efficient fine-tuning
- Training infrastructure and GPU requirements
- Model evaluation and validation – LLMOps for LLMs:
- Model registry and versioning (MLflow, Weights & Biases)
- Experiment tracking and reproducibility
- Model lifecycle management
- CI/CD for model deployment
- Automated model testing and validation – Tool-Calling and Function-Calling:
- Tool-calling concepts and implementation
- Function-calling APIs
- Integration of external tools and services
- Agentic workflows (distinction from agentic coding) – Security and Compliance for AI Workloads:
- Model security and supply chain
- Data privacy and GDPR compliance
- Network security for model serving
- Audit logging and compliance reporting – Cloud Providers and Managed Services:
- AWS SageMaker, Bedrock and EC2 for LLMs
- Azure OpenAI Service and Azure ML
- Google Cloud Vertex AI and GKE
- Hybrid and multi-cloud strategies – Best Practices and Future Trends:
- Emerging technologies in AI infrastructure
- Cost optimization and sustainability
- Performance tuning and benchmarking
- Team organization for AI platform teams
The course combines theoretical foundations with intensive practical exercises. Participants work with real AI infrastructure projects and learn how to professionally operate LLM workloads.
The actual course content may differ from the above depending on the trainer, delivery, duration and the composition of participants.
Request this course in-house
Request a public date
No suitable public date? Register without obligation — once there is enough interest we schedule a new public date and let you know first.
More about AI Platform Engineering
Platform engineering for AI applications represents a new discipline in IT infrastructure. With the rapid rise of large language models and generative AI, new challenges emerge for platform engineers: GPU resource management, inference optimization, cost management, and scaling of AI workloads. Platform engineers learn how to bridge the gap between AI/ML teams and production infrastructure and operate AI applications reliably, performantly, and cost-effectively.Further resources:
History
MLOps emerged as a discipline to bridge the gap between data science experiments and production-ready machine learning operations. Tools like MLflow (2018) and Kubeflow (2018) standardized experiment tracking, model versioning, and deployment pipelines for classical ML models. With the breakthrough of Transformer models and LLMs from 2022 onward – triggered by GPT-3 and later ChatGPT – new challenges emerged: models with billions of parameters require GPU clusters, specialized inference engines, and new observability concepts.
LLMOps established itself as an extension of MLOps specifically for large language models. Tools like vLLM with its PagedAttention algorithm enable up to 24x higher throughput compared to traditional approaches. LLM gateways like LiteLLM unify access to various models and providers. Platform engineers today face the task of operating these complex stacks in a production-ready manner – a role that gave birth to the field of AI Platform Engineering.