vLLM

vLLM is the de facto open-source serving engine for large language models, originally born out of UC Berkeley's Sky Computing Lab and now governed by an open community of contributors across Anyscale, Meta, NVIDIA, Databricks, AMD, and many others. Its core innovation is PagedAttention, a virtual-memory-style allocator for KV cache that dramatically reduces fragmentation and lets a single GPU host serve far more concurrent requests than a naive transformer stack. On top of PagedAttention the project layers continuous batching, speculative decoding, prefix caching, tensor and pipeline parallelism, quantization (AWQ, GPTQ, FP8, INT4), and an OpenAI-compatible HTTP server. vLLM supports nearly every popular architecture — Llama, Qwen, DeepSeek, Mistral, Phi, Gemma, multimodal models like Llava and Qwen-VL, and embedding/reranker models — across NVIDIA, AMD, Intel, AWS Inferentia, and Apple Silicon hardware. Because it is open source under Apache 2.0 there is no subscription cost; teams pay for the GPUs they run it on. vLLM ships as a Python package, a Docker image, a Kubernetes operator, and is the default backend behind many managed inference clouds (Together, Fireworks, Lepton, RunPod, parts of AWS Bedrock). Production engineering teams use vLLM when they need self-hosted control of latency, cost, privacy, and routing for their LLM workloads.