Loading tool details…
Open-source evaluation framework for retrieval-augmented generation — metrics for faithfulness, context relevance, and answer quality.
Open-source evaluation framework for retrieval-augmented generation — metrics for faithfulness, context relevance, and answer quality.
Ragas is the go-to open-source evaluation framework for retrieval-augmented generation systems. Rather than eyeballing whether a RAG pipeline is "good enough," Ragas provides a standard set of LLM-as-judge and reference-based metrics — faithfulness (does the answer cite only the retrieved context?), context precision and recall (did retrieval find the right chunks?), answer relevance and correctness, and noise sensitivity — that let engineering teams put a number on regression as they change embeddings, prompts, chunkers, or LLMs. It integrates natively with LangChain, LlamaIndex, and Haystack pipelines, generates synthetic test sets from your own documents (so you don't need to hand-write hundreds of Q/A pairs), and produces per-example breakdowns you can drill into. The library is Apache 2.0 licensed, has 8k+ stars on GitHub, and is maintained by Exploding Gradients. In 2026 Ragas shipped an agent-specific evaluation module (tool-use trajectories, plan quality, cost/latency budgets), improved judge-model prompts to reduce bias, and added first-class support for evaluating MCP-tool-based agents (so trace-level assertions can be written across a whole tool-calling flow). Free and open source; the parallel Ragas Cloud offers a hosted UI + collaborative datasets on a paid tier for teams that don't want to build their own eval dashboards.
RAGAS includes RAG-specific metrics such as Context Precision, Context Recall, Context Entities Recall, Noise Sensitivity, Response Relevancy, and Faithfulness. These help teams separate retrieval failures from generation failures instead of treating the entire RAG pipeline as a black box.
The documentation includes agent and tool-use metrics such as Topic Adherence, Tool Call Accuracy, Tool Call F1, and Agent Goal Accuracy. This makes RAGAS useful for workflows where the AI system must call tools, follow a goal, or stay on topic across a task.
RAGAS supports testset generation for RAG, agents, and tool-use cases, along with knowledge graph building and scenario generation. The docs also reference persona generation, non-English testset generation, custom single-hop queries, custom multi-hop queries, and pre-chunked data workflows.
The documentation lists framework integrations with AG-UI, Griptape, Haystack, LangChain, LangGraph, LlamaIndex, LlamaIndex Agents, LlamaStack, R2R, and Swarm. It also includes provider guidance for Amazon Bedrock, Google Gemini, OCI Gen AI, and Vertex AI models.
RAGAS includes customization guides for models, run configuration, caching, cancelling tasks, LLM adapters, metric prompts, language adaptation, and training or aligning metrics. It also includes prompt optimization and cost analysis guidance, which is useful when evaluation needs to be integrated into an iterative development workflow.
Free
Paid tier
Ready to get started with Ragas?
View Pricing Options →We believe in transparent reviews. Here's what Ragas doesn't handle well:
AI Evaluation
Open-source CLI and library for testing, evaluating, and red-teaming LLM prompts, models, and RAG pipelines — runs locally on your machine or in CI.
AI Evaluation
Braintrust review 2026: unified evals, prompt workbench, observability, and AI proxy used by OpenAI, Stripe, Notion — features, pricing, pros, cons.
LLM Observability
LangSmith helps developers trace, evaluate, test, and monitor applications built with language models and agent workflows.
Testing & Quality
DeepEval is the open-source, Pytest-style LLM evaluation framework from Confident AI. Ship it in your test suite the same way you ship pytest — it runs 50+ research-backed metrics (hallucination, tool-use correctness, contextual precision/recall, conversational quality) offline in dev and inline in CI/CD, then optionally streams the results to Confident AI's paid cloud for team dashboards and prompt versioning.
Pricing and features change often. If anything on this Ragas page is wrong or missing, tell us and include a link to the source. We review every correction.