Self-Hosted LLMs

Content

The course consists of the following topics and can be extended or adapted depending on the audience. The examples focus on open models and open-source tooling that you can run on your own hardware. For in-house courses we pick models, hardware scenarios and use cases that fit the audience.

– Why digital sovereignty matters for LLMs:

• Data protection: documented leaks and retention obligations at public providers
• Legal and geopolitical dependencies, provider jurisdiction
• The financial situation of the large providers and what it means for prices
• Lock-in, deprecated models and availability as an operational risk
• When an external provider is the right choice anyway – Three operating models compared:
• Public: using ChatGPT, Claude or Mistral as a service
• Hosted: AWS Bedrock, Azure AI Foundry, Google Vertex AI – and European or Swiss providers when jurisdiction matters
• Local: your own hardware, full control, full responsibility
• Trade-offs in control, effort, cost and compliance
• Open models from Europe and Switzerland, such as Apertus by ETH, EPFL and CSCS – Memory as the real bottleneck:
• Why memory sets the limit and not FLOPS
• The memory budget: parameters, KV cache and workspace
• Context length and what it does to the memory footprint
• Why memory bandwidth decides how fast a model answers
• Measuring throughput, latency and tokens per second yourself – Quantization:
• What quantization does to the weights
• Methods and levels: Q8, Q4, K-quants, AWQ, GPTQ
• The trade-off between memory saved and quality lost
• Downloading quantized models or quantizing them yourself – Model architecture and model choice:
• Large generalists versus small specialists
• Dense versus mixture of experts: the active parameters set the speed, all parameters together set the memory footprint
• Reading model names on Hugging Face: name, version, size, active parameters, quantization
• Licenses of open models and what they mean for you
• Size classes by available memory: up to 16 GB, 24 to 96 GB, and what only runs in a cluster – Hardware and sizing:
• Consumer GPUs, professional GPUs and Apple Silicon with unified memory
• Rules of thumb: which model needs how much memory
• Sizing for several concurrent users
• Buy or rent, amortization against API cost
• Getting more out of the GPUs you already have, for example with batch jobs overnight – Running llama.cpp and Ollama:
• Loading GGUF models and setting parameters
• Ollama day to day: modelfiles, offloading, GPU layers
• The OpenAI-compatible API as the de-facto standard
• Where it runs out of steam with several users – Production operation with vLLM:
• PagedAttention and efficient KV cache management
• Continuous batching and throughput optimization
• Serving vLLM from the CLI and using it from Python
• Configuring GPU memory utilization, context length and API keys
• How it compares to alternatives such as TGI and SGLang
• Benchmarking and capacity planning – Building applications against your own LLM:
• Pointing OpenAI-compatible clients at your own base URL
• Structured output, tool calling and streaming with open models
• Embeddings and RAG with local models
• Fallback and routing strategies between local and external – Fine-tuning small specialist models:
• When to fine-tune, when to prompt, when to use RAG – and when not to use an LLM at all
• Fine-tuning is not training from scratch: what is realistically feasible
• LoRA and QLoRA as the affordable route
• Fine-tuning with Unsloth, preparing the dataset, and how much GPU it takes
• Shipping adapters and serving them with vLLM – Evaluation and quality assurance:
• Why public benchmarks say little about your use case
• Building your own test harness with representative prompts
• Nondeterminism: repeated runs, scoring and regression tests
• Comparing models, versions and quantization levels objectively
• When your own evaluation is worth the effort and when it is not – Outlook on operations in the team and the company:
• vLLM on Kubernetes, GPU sharing and time-slicing
• Several models and teams on shared hardware
• Monitoring GPU utilization, latency and cost
• Access, auditability and the step to your own LLM platform – Best practices and outlook:
• “Dare to go small” – the smallest solution that covers the use case
• Reference architectures for running LLMs yourself
• Judging where open models and hardware are heading

Participants run their own models on real hardware, compare quantization levels and architectures with their own measurements, and leave with a solid decision on which AI workloads to self-host from now on.

The actual course content may differ from the above depending on the trainer, delivery, duration and the composition of participants.