Loading tool details…
Open-source CLI and library for testing, evaluating, and red-teaming LLM prompts, models, and RAG pipelines — runs locally on your machine or in CI.
Open-source CLI and library for testing, evaluating, and red-teaming LLM prompts, models, and RAG pipelines — runs locally on your machine or in CI.
Promptfoo is an open-source tool that has become the most popular CLI for evaluating LLM prompts and applications. You write a YAML config that lists prompts, providers, test cases, and assertions and Promptfoo runs the matrix locally, caches results, and shows a web UI diff between configurations.
Promptfoo evaluates prompts, models, and RAG pipelines so teams can compare behavior across changes. This is useful for regression testing, factuality checks, hallucination reduction, and validating whether a model or retrieval change improves real application outputs.
The Red Teaming product is designed to proactively identify and fix vulnerabilities in AI applications. Teams can use it to test jailbreak resistance, adversarial prompts, unsafe completions, and other security risks before users encounter them.
Promptfoo’s Guardrails are positioned as real-time protection against jailbreaks and adversarial attacks. This makes the platform relevant not only for offline evaluation but also for teams considering runtime safety controls around LLM applications.
The MCP Proxy is described as a secure proxy for Model Context Protocol communications. This is important for agentic systems that use MCP connections and need a security boundary around model-to-tool or model-to-context interactions.
Promptfoo’s Code Scanning product finds LLM vulnerabilities in IDE and CI/CD workflows. That lets engineering teams catch AI-specific security issues earlier in the software development process instead of relying only on manual review or production monitoring.
Free (MIT)
Custom / usage-based
Custom
Ready to get started with Promptfoo?
View Pricing Options →We believe in transparent reviews. Here's what Promptfoo doesn't handle well:
The scraped website content states “Promptfoo is now part of OpenAI” and shows © 2026 Promptfoo, Inc. The provided content does not include a dated release note or detailed 2025-2026 changelog beyond that update.
AI Evaluation
Braintrust review 2026: unified evals, prompt workbench, observability, and AI proxy used by OpenAI, Stripe, Notion — features, pricing, pros, cons.
LLM Observability
LangSmith helps developers trace, evaluate, test, and monitor applications built with language models and agent workflows.
Developer Tools
Humanloop is a Developer Tools tool for teams that need practical AI-assisted workflows, with pricing, governance, strengths, limitations, and best-fit use cases to verify before adoption.
Testing & Quality
DeepEval is the open-source, Pytest-style LLM evaluation framework from Confident AI. Ship it in your test suite the same way you ship pytest — it runs 50+ research-backed metrics (hallucination, tool-use correctness, contextual precision/recall, conversational quality) offline in dev and inline in CI/CD, then optionally streams the results to Confident AI's paid cloud for team dashboards and prompt versioning.
Pricing and features change often. If anything on this Promptfoo page is wrong or missing, tell us and include a link to the source. We review every correction.