Opik

Opik is an open-source platform built by Comet that covers the full lifecycle of LLM application development — from debugging and evaluation to production monitoring. It provides comprehensive tracing for LLM calls, RAG pipelines, and multi-agent systems, recording every step an application takes to generate a response. Developers can define and compute evaluation metrics, run experiments with different prompts against test sets, and use built-in LLM judges for hallucination detection, factuality checking, and content moderation. Opik includes automated prompt optimization with four distinct optimizers (Few-shot Bayesian, MIPRO, evolutionary, and MetaPrompt) that iterate toward high-performing system prompts and freeze them as reusable production assets. Built-in guardrails screen user inputs and LLM outputs to detect and redact PII, competitor mentions, off-topic content, and other unwanted material. The platform supports LLM unit testing within CI/CD pipelines via PyTest integration, letting teams establish performance baselines and run comprehensive test suites on every deploy. In production, Opik logs all traces to identify issues, tracks model performance on unseen data, and generates datasets for new development iterations. The full feature set is available in the open-source code on GitHub for self-hosting, with a free cloud-hosted option and an enterprise tier for teams needing scalability, SSO, and dedicated support.