VLLM Architecture Deep Dive: From V0 to V1 and Beyond
VLLM has evolved from a research project into the industry standard for LLM inference. This article explores its core architectural principles, the revolutionary changes in V1, and the roadmap toward V2. Understanding these internals is crucial for anyone deploying large language models at scale.
Part 1: Core Architectural Principles
The Foundation: PagedAttention
At the heart of VLLM lies PagedAttention, a memory management technique inspired by operating system virtual memory. Traditional LLM serving stores the Key-Value (KV) cache in contiguous memory blocks, leading to significant fragmentation and waste.
How PagedAttention Works
- KV cache is divided into fixed-size blocks (like OS pages)
- Blocks are stored non-contiguously in GPU memory
- Dynamic allocation: blocks allocated on-demand as sequences grow
- Memory sharing: copy-on-write for parallel sampling and beam search
Traditional KV Cache (Wasteful): ┌─────────────────────────────────────┐ │ Seq A: ████████████░░░░░░░░░░░░░░░ │ <- 40% waste │ Seq B: ████████░░░░░░░░░░░░░░░░░░░ │ <- 60% waste │ Seq C: ████████████████░░░░░░░░░░░ │ <- 50% waste └─────────────────────────────────────┘ PagedAttention (Efficient): Block Table: Seq A: [Block 0] -> [Block 3] -> [Block 7] Seq B: [Block 1] -> [Block 4] Seq C: [Block 2] -> [Block 5] -> [Block 6] -> [Block 8] Physical GPU Memory: ┌─────────────────────────────────────┐ │ [B0][B1][B2][B3][B4][B5][B6][B7][B8]│ <- Near 0% waste └─────────────────────────────────────┘
Continuous Batching
VLLM employs continuous (or in-flight) batching to maximize GPU utilization. Unlike static batching where all requests must complete before new ones start, VLLM dynamically adds and removes requests from batches.
Static Batching
Wait for all requests to complete. Fast requests wait for slow ones. GPU idle time is significant.
Continuous Batching
New requests join immediately when others finish. GPU stays busy. Higher throughput.
Part 2: VLLM V0 Architecture
The original VLLM architecture (V0) introduced the foundational concepts but had several limitations that V1 addressed. Understanding V0 helps appreciate the improvements in V1.
V0 Workflow
VLLM V0 Architecture:
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ API Server │────▶│ Engine │────▶│ Worker │
│ (AsyncIO) │ │ (Central) │ │ (GPU) │
└─────────────┘ └──────┬──────┘ └─────────────┘
│
┌──────┴──────┐
│ Scheduler │
│ (FCFS) │
└──────┬──────┘
│
┌──────┴──────┐
│ Block │
│ Manager │
└─────────────┘
Limitations:
1. Synchronous scheduling - CPU waits for GPU
2. Complex tensor-parallel communication
3. Monolithic design
4. Limited CPU/GPU overlap
Part 3: VLLM V1 - The Major Rewrite
Released in January 2025, VLLM V1 marked a comprehensive re-architecture. The core components - scheduler, KV cache manager, worker, sampler, and API server - were rebuilt from scratch while retaining stable elements like model implementations.
Key V1 Improvements
1. Multiprocessing Architecture
V1 integrates multiprocessing deeply into AsyncLLM, creating isolated EngineCore execution loops. This allows CPU-intensive tasks (tokenization, multimodal processing) to overlap with GPU execution.
2. Unified Scheduler
V1 treats prompt and output tokens uniformly. Uses dynamic token budgets per request, enabling chunked prefills, prefix caching, and speculative decoding without strict phase separation.
3. Clean Tensor-Parallel Inference
V1 addresses V0's asymmetric tensor-parallel limitations by caching request states on workers and transmitting only incremental updates (diffs) at each step.
4. Async-First Design
CPU prepares inputs for step N+1 while GPU executes step N. This overlap maximizes GPU utilization and throughput.
V1 Architecture Diagram
VLLM V1 Architecture:
┌─────────────────────────────────────────────────────────┐
│ API Server Process │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────────┐ │
│ │ HTTP │ │ Tokenizer │ │ Input Processor │ │
│ │ Handler │ │ │ │ (Multimodal) │ │
│ └──────┬──────┘ └──────┬──────┘ └────────┬────────┘ │
└─────────┼────────────────┼──────────────────┼──────────┘
│ │ │
└────────────────┴──────────────────┘
│
┌──────┴──────┐
│ ZMQ Queue │ <- Async communication
└──────┬──────┘
│
┌──────────────────────────┼──────────────────────────────┐
│ Engine Core Process │
│ ┌───────────────────────┼─────────────────────────┐ │
│ │ ▼ │ │
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────┐ │ │
│ │ │ Scheduler │ │ Block │ │ Output │ │ │
│ │ │ (Unified) │ │ Manager │ │ Proc │ │ │
│ │ └──────┬──────┘ └─────────────┘ └────┬────┘ │ │
│ │ │ │ │ │
│ └─────────┼────────────────────────────────┼──────┘ │
└────────────┼────────────────────────────────┼──────────┘
│ │
┌────────┴────────┐ ┌───────┴───────┐
│ GPU Worker 0 │ │ GPU Worker N │
│ ┌───────────┐ │ │ ┌───────────┐│
│ │ Model │ │ │ │ Model ││
│ │ Forward │ │ │ │ Forward ││
│ └───────────┘ │ │ └───────────┘│
└─────────────────┘ └───────────────┘
Key Features:
- CPU and GPU work in parallel
- Tensor-parallel with minimal communication
- Dynamic batching with chunked prefills
- Prefix caching support
V1 Workflow Deep Dive
Request Arrival & Preprocessing
API server receives HTTP request, tokenizes input, processes multimodal data (images, audio). All CPU-intensive work happens here.
Async Queue
Preprocessed requests enter ZMQ queue for async handoff to EngineCore. API server immediately ready for next request.
Unified Scheduling
Scheduler treats prefills and decodes uniformly. Allocates dynamic token budgets. Supports FCFS, priority-based, and custom policies.
GPU Execution
Workers execute model forward pass with PagedAttention. Tensor-parallel workers communicate via efficient all-reduce operations.
Streaming Response
Generated tokens stream back to client. Completed sequences removed from batch, new sequences added dynamically.
Part 4: Toward V2 and Model Runner V2 (MRV2)
While V2 as a full release is still evolving, Model Runner V2 (MRV2) announced in March 2026 represents the next architectural evolution. It builds on V1's foundations with even more GPU-native execution.
MRV2 Key Innovations
Persistent Batch V2
Moves bookkeeping from CPU to GPU. Input preparation becomes GPU-native, reducing CPU-GPU synchronization overhead.
Zero Synchronization
Async-first design aims for zero sync between CPU and GPU across all models and features. Enables seamless speculative decoding.
Hybrid Allocator
Minimizes fragmentation, improves memory reuse. Different allocation/eviction algorithms for various KV cache architectures.
Enhanced Speculative Decoding
Better CUDA Graph support, torch.compile integration, and suffix decoding for improved token generation speed.
Performance Comparison
| Feature | V0 | V1 | MRV2 |
|---|---|---|---|
| CPU-GPU Overlap | Limited | High | Maximum |
| Scheduling | Phase-separated | Unified | GPU-native |
| Tensor Parallel | Complex | Clean | Optimized |
| Speculative Decode | Basic | Good | Seamless |
Conclusion
VLLM's evolution from V0 to V1 and toward V2 demonstrates the continuous innovation in LLM inference optimization. The shift from synchronous to async-first design, from CPU-centric to GPU-native execution, represents a fundamental rethinking of how to serve large language models efficiently.
For practitioners, understanding these architectural details helps in optimizing deployments, debugging performance issues, and making informed decisions about model serving infrastructure. As models grow larger and more complex, the innovations in VLLM will only become more critical.