Guidance

Guidance is a free, open-source Python library and constrained generation framework from Microsoft Research, categorized among AI agent builder and structured output tools, that gives developers deterministic control over large language model output by enforcing JSON schemas, regex patterns, and context-free grammars at the token level during generation — completely free with no paid tiers, usage fees, or API costs.

With over 19,000 GitHub stars and more than 100 contributors, Guidance has become one of the most widely adopted structured output libraries in the LLM ecosystem. The project averages over 50,000 monthly downloads on PyPI and has accumulated more than 3,500 forks on GitHub, reflecting strong community interest across research and production use cases. Originally released in 2023, Guidance has been under continuous development with frequent releases and an active issue tracker.

Unlike traditional prompting approaches where developers send text to a model and hope it responds in the correct format, Guidance interleaves fixed template text with constrained generation steps. This means the model can only produce tokens that satisfy the specified constraints — whether those are regex patterns, JSON schemas with full support for oneOf, allOf, anyOf, and recursive definitions, or arbitrary context-free grammars. Invalid output is structurally impossible, not merely unlikely.

At the core of Guidance is llguidance, a high-performance grammar engine written in Rust that efficiently processes constraints during token generation. This engine validates and masks tokens in real time, ensuring that every generated token conforms to the specified grammar without significant performance overhead. The Rust implementation provides substantial speed improvements over pure-Python alternatives, handling complex nested schemas and large grammars with minimal latency impact.

Guidance supports multiple model backends through a unified programming interface. Developers can use OpenAI GPT models, Azure OpenAI deployments, local models via Hugging Face Transformers or llama.cpp, and experimental Anthropic model support. The strongest structural guarantees are available with local model backends where Guidance can directly control token sampling and masking. Cloud API backends leverage provider-specific structured output features but may have some limitations compared to local execution.

Key programming primitives include gen() for constrained text generation, select() for forcing output to match one of predefined options, and the @guidance decorator for packaging reusable generation patterns as composable Python functions. Token healing automatically corrects tokenization boundary artifacts where template text meets generated content. Conditional logic with if/else blocks and loop constructs enables dynamic multi-step generation pipelines.

Guidance integrates naturally with Jupyter notebooks, providing token-level highlighting and visualization for debugging and rapid prototyping. The Mock model allows developers to validate grammar constraints and test generation pipelines without making any real LLM API calls, reducing development costs and iteration time. These features make Guidance particularly valuable for research teams and developers who need to experiment with complex generation strategies before deploying to production.

The library is MIT licensed with zero telemetry, no usage tracking, and complete source code available for audit. When used with local model backends, all data processing occurs entirely on-premises with no external network calls, making it suitable for sensitive or regulated environments.