SGLang

SGLang is an open-source LLM serving framework developed by the LMSYS team (the group behind Chatbot Arena) and a broad community of contributors. Its differentiator is RadixAttention — a prefix-tree KV cache that aggressively reuses shared prefixes across requests — combined with a constrained-decoding engine that makes structured outputs (JSON, regex grammar, function calls) close to free in latency terms. On many real-world workloads SGLang reports throughput improvements over earlier vLLM versions, particularly for prompts with shared system prefixes (very common in agent loops) and for structured output use cases. The framework supports tensor and pipeline parallelism, FP8/AWQ/GPTQ quantization, speculative decoding, prefix caching, and a wide model catalog: Llama, Qwen, DeepSeek (including DeepSeek-V3 and -R1 variants), Mistral, multimodal Llava-class models, embedding models, and reward models. Like vLLM, SGLang exposes an OpenAI-compatible HTTP server, ships Docker images, and runs on NVIDIA, AMD ROCm, and increasingly other accelerators. The project is Apache 2.0, so there is no license fee — costs are the hardware you run it on. Teams that hit a ceiling with vLLM on structured/agent workloads, or who need maximal throughput on DeepSeek-class MoE models, often evaluate SGLang as either a replacement or a complementary backend.