Distributed LLM Serving Architecture: vLLM PagedAttention vs TensorRT-LLM Execution Graphs

Deploying large language models (LLMs) in high-throughput enterprise production shifts the primary engineering bottleneck from compute operations (FLOPs) to memory bandwidth. At batch size 1, autoregressive token generation is memory-bound: every generated token requires loading gigabytes of model weights from High Bandwidth Memory (HBM) to GPU compute cores. As concurrency scales, managing Key-Value (KV) cache memory fragmentation determines whether an inference cluster sustains target latency SLAs or experiences out-of-memory (OOM) failures.

1. The Memory Bottleneck in Autoregressive Decoding

Every token generated in a decoder-only transformer requires caching the Key and Value projection vectors of preceding tokens across every attention layer. The formula for KV cache memory consumption per request is defined by:

Memory = 2 x Sequence Length x Hidden Layers x Hidden Size x Bytes per Element

In naive serving runtimes, contiguous VRAM is pre-allocated based on the model's maximum context window (e.g., 8k or 32k tokens). Because user queries vary wildly in length, this traditional design wastes between 60% and 80% of available GPU memory due to internal and external fragmentation.

2. vLLM: Virtual Paging and Dynamic KV Allocation

vLLM mitigates fragmentation by borrowing virtual memory concepts from operating systems through its core engine, PagedAttention. Rather than forcing continuous memory blocks, vLLM divides the KV cache into fixed-size logical memory pages (typically 16 or 32 tokens per block).

• Block Tables: A dynamic block table maps virtual token blocks to physical GPU memory addresses on the fly, eliminating external memory fragmentation.
• Dynamic Prefill-Decode Splitting: By maintaining memory fragmentation below 4%, vLLM dramatically increases the maximum continuous batch size, driving Streaming Multiprocessor (SM) saturation.
• Copy-on-Write Branching: Complex sampling architectures (such as parallel sampling or tree-based speculative decoding) share KV cache prefixes across requests without physical memory duplication.

3. TensorRT-LLM: Hardware-Fused CUDA Kernel Graphs

Where vLLM prioritizes runtime memory elasticity and rapid framework deployment, NVIDIA's TensorRT-LLM approaches the problem through aggressive hardware compilation and execution graph optimization:

• Ahead-of-Time Kernel Fusion: TensorRT-LLM fuses normalization layers, matrix multiplications, and activation functions into monolithic CUDA kernels. This prevents intermediate activation data from being read from and written back to global HBM.
• In-Flight Batching (Continuous Batching): TensorRT-LLM pipelines iteration-level scheduling natively at the driver level, reducing scheduling overhead between the prefill and decoding phases.
• CUDA Graph Capture: The entire execution loop is captured as static CUDA graphs, eliminating CPU-to-GPU launch overheads and stabilizing Time-To-First-Token (TTFT) metrics under load.

4. Operational Trade-Offs for Enterprise Infrastructure

Selecting the serving engine dictates the long-term MLOps burden and GPU infrastructure budget:

• Model Loading Velocity: vLLM reads directly from PyTorch or SafeTensors weights in seconds, enabling dynamic model hot-swapping. TensorRT-LLM requires architecture-specific engine compilation steps, making frequent weights updates slower in rapid CI/CD environments.
• Throughput vs. Latency: In benchmark deployments running Llama architectures at FP8 on H100 clusters, TensorRT-LLM achieves 15% to 30% higher peak token throughput due to custom kernel assembly, directly lowering cost-per-token at scale.
• Hardware Portability: vLLM maintains portability across heterogeneous hardware fleets (including AMD ROCm and diverse NVIDIA generations), whereas TensorRT-LLM provides maximum performance strictly within dedicated NVIDIA ecosystems.