AirLLM

AirLLM is a lightweight Python inference library with a very specific superpower: it lets you run extremely large transformer models on very small GPUs by loading exactly one transformer layer into VRAM at a time, running it, and immediately paging it back out to system RAM or disk. That means a 70B Llama-3 or Qwen model runs on a laptop RTX 4060 (8 GB) and a 405B Llama-3.1 model runs on a single 4090 (24 GB). The catch, of course, is throughput: single-token latency is dominated by weight-loading, so you get batch-style outputs measured in tokens per minute rather than tokens per second. That is a perfectly good trade-off for offline batch inference, evaluation runs, dataset generation, and "I just need to run this model once on my own hardware" experiments.

The library is a thin wrapper around HuggingFace Transformers with automatic 4-bit and 8-bit quantization via bitsandbytes, chunked prefetching, and a compression option that shrinks the on-disk footprint. There is no server, no UI, and no hosted product — you pip install airllm, point it at a HuggingFace model ID, and call .generate() from Python. It has become the standard "can I run this on my Mac / gaming PC?" answer for the open-weights community, and pairs well with KTransformers for interactive workloads or with vLLM for full production serving.