KTransformers

KTransformers is a research-grade LLM inference framework maintained by the KVCache.ai team (Tsinghua University) that lets you run 200B+ parameter Mixture-of-Experts models such as DeepSeek-V3, DeepSeek-R1, and Kimi K2 on a single consumer or prosumer workstation. The trick is aggressive expert-offloading: hot expert weights stay on the GPU, cold experts live in system RAM or NVMe, and a KV-cache-aware scheduler streams what each token actually needs. In practice this means a 24 GB GPU plus 256 GB of DDR5 can serve a 236B DeepSeek-V3 checkpoint at usable interactive speeds — a workload that would otherwise require a multi-GPU H100 rig.

Under the hood it is a drop-in replacement for HuggingFace Transformers with a compatible generation API, plus custom CUDA and CPU kernels for grouped-GEMM, INT4/INT8 quantization, and speculative decoding. It exposes an OpenAI-compatible HTTP server, so anything that speaks the ChatGPT API (Continue.dev, Cline, Aider, LibreChat, custom agents) can be pointed at it with zero code changes. There is no vendor UI, no hosted plan, and no telemetry: you run the Docker container or install the Python package, load your quantized weights, and hit /v1/chat/completions. It is the go-to project for solo researchers, homelab enthusiasts, and startups that want to serve frontier open-weights models without paying cloud GPU rates.