LLM-d: A Deep Dive

The LLM-d: A Deep Dive course is designed to help learners understand how large language models are served, routed, scaled, and optimized in production environments. Starting with the fundamentals of model serving and hardware, the course progressively explores what happens inside an individual request, how LLM workloads scale across GPU fleets, and how llm-d enables intelligent, distributed inference on Kubernetes.

Through conceptual lessons, guided demonstrations, hands-on experiments, live cluster walkthroughs, and practical benchmarks, you'll develop a deep understanding of LLM serving architecture and the decisions that affect performance, scalability, and resource utilization. You'll explore request routing, KV-cache management, expert parallelism, prefill and decode separation, autoscaling, rollouts, observability, and production operations.

The course goes beyond the concepts to show how these components work together in a running llm-d stack. You'll work with real workloads to observe request routing, cache reuse, scaling behavior, live traffic, and performance metrics, building the practical understanding needed to operate and optimize LLM inference systems.

Course Modules & Learning Outcomes

From ChatGPT to a Model Server

Build a foundation in how LLMs move from a model running in a script to a production-ready model-serving system. Understand what a model is, how CPU, GPU, and VRAM affect inference, and how a model server exposes models as services. You'll also explore the key components involved in serving an LLM.

Inside a Single Request

Understand what happens when an LLM receives a request and generates a response. Explore token generation, streaming, latency, throughput, and the role of the model's scratchpad during inference. You'll develop an intuitive understanding of the factors that determine how quickly an LLM can process requests.

From One Server to a Fleet

Explore the challenges of serving LLMs for multiple users and scaling beyond a single GPU or machine. Learn how larger models are distributed across hardware, how multiple servers work together, and why traditional load-balancing approaches can be ineffective for LLM workloads.

Meet llm-d

Learn how llm-d addresses the challenges of distributed LLM serving. Explore its architecture, Kubernetes-based components, routing decisions, and scheduling mechanisms. You'll understand how llm-d uses workload and model information to make intelligent routing decisions and manage inference resources.

Tour the Running Stack

Get hands-on with a complete llm-d deployment and explore the cluster behind the course demonstrations. Send requests through the stack, trace their path through the system, and observe how individual requests are handled across the infrastructure.

Smart Routing, Live

Understand how intelligent routing improves LLM serving efficiency. Explore the scoring mechanisms behind routing decisions, prefix-aware routing, request distribution, and how llm-d handles cold requests. You'll see how routing strategies affect cache reuse and overall system performance.

Getting More from the KV Cache

Learn how KV-cache management can improve inference performance and reduce redundant computation. Explore precise prefix-cache routing, tiered caching across GPU, CPU, and disk, and cache reuse across multi-turn conversations.

Serving Bigger Models with llm-d

Understand how llm-d supports serving models that require distributed GPU resources. Explore expert parallelism, prefill/decode (P/D) separation, and the practical limits of GPU-based inference. You'll connect these concepts to a live cluster and understand the trade-offs involved in serving large models.

Running It Day to Day

Learn how to operate an llm-d deployment under real-world traffic. Explore flow control, request queuing, autoscaling based on meaningful workload signals, and zero-downtime rollouts. You'll also observe how a rolling update behaves while the system is handling live requests.

Seeing What Is Happening

Learn how to monitor and evaluate an llm-d deployment using metrics, Grafana dashboards, and benchmarks. Understand the key performance indicators reported by llm-d, interpret operational dashboards, and run benchmarks to evaluate serving performance.

Wrap Up: Agents and Beyond

Explore how LLM serving evolves beyond traditional request-response workloads. Learn about agentic and multimodal serving and examine how llm-d can support multi-turn, agent-style workloads. You'll bring together the concepts covered throughout the course to understand emerging patterns in LLM infrastructure.

Course Features

• Hands-on exploration of a running llm-d stack and Kubernetes cluster.
• Practical demonstrations of intelligent routing, KV-cache reuse, scaling, and distributed inference.
• Deep coverage of LLM serving fundamentals, request processing, GPU utilization, and performance bottlenecks.
• Real-world operational scenarios covering autoscaling, flow control, zero-downtime rollouts, monitoring, and benchmarking.
• Advanced concepts including expert parallelism, prefill/decode separation, tiered caching, agentic workloads, and multimodal serving.

Who Should Enroll?

• DevOps and platform engineers working with AI/ML infrastructure.
• Kubernetes engineers responsible for deploying and operating AI workloads.
• ML engineers and AI infrastructure engineers working with LLM inference.
• SREs focused on reliability, scalability, and performance of AI workloads.
• Cloud engineers building production-grade LLM serving platforms.
• Engineers who want to understand llm-d and modern distributed LLM serving architectures.

Build a deep understanding of how LLM inference systems work, and learn how llm-d can be used to intelligently route, scale, optimize, monitor, and operate LLM workloads in Kubernetes-based environments.