ARBR

ARBR is an open-source, self-hosted control plane for teams that already send meaningful AI traffic to production. It focuses on a specific operational problem: identifying requests that may be served by a cheaper model, proving the replacement works on representative traffic, approving the switch, and measuring whether savings held after rollout. This is more focused than a generic API gateway. ARBR links cost, latency, selected model, and outcomes to applications, teams, workflows, task types, and users, then turns the observed workload into model-switching recommendations.

Governance and routing workflow

The project documents a six-stage lifecycle: observe real workloads, discover optimization opportunities, build an evaluation dataset, compare candidates, approve a guarded rollout, and verify actual results. Evaluations can measure quality, cost, latency, format adherence, and critical failures. Production changes remain human-governed: explicitly pinned models stay pinned, while only requests using model: auto follow enabled routing rules. Teams can use shadow evaluation, canary rollout, rollback, and budget actions that alert, downgrade, or block spending at a configured cap. Recording both requested and served models makes promised savings auditable.

ARBR provides native and OpenAI-compatible gateway endpoints. JavaScript/TypeScript and Python clients include retries, timeouts, typed errors, and a LangChain adapter. The separate arbr-audit npm CLI can inspect a request log or wrap a coding-agent session without running the server, database, or signup flow. For a full local demo, the README specifies Node 18 or newer and Docker Compose, which starts MongoDB, seeded sample data, and a dashboard on port 4100. Provider credentials can come from environment configuration or the dashboard, where they are encrypted at rest; production teams should supply their own encryption key and secrets-management process.

Price and operating cost

The repository carries an MIT license, so the software has no license fee in the reviewed source. The conventional GitHub /pricing route returned 404, and no first-party hosted-service price was found, so this record remains marked for manual verification. Free software is not zero-cost operations: budget for MongoDB, compute, backups, gateway availability, telemetry storage, upgrades, incident response, and model-provider charges. Support and SLA terms are also not established by the public repository.

Honest assessment

ARBR is useful when a team can supply representative production traces and needs evidence before changing models. Its advisory defaults, reversible rules, and realized-savings measurement are stronger governance than opaque least-cost routing. It is excessive for a prototype with little traffic, and self-hosting shifts reliability and security ownership to the user. Compare LiteLLM, OpenRouter, LangSmith, and the guide to monitoring AI agents in production.

Pilot with one high-volume task class. Replay at least 100 sanitized requests through the incumbent and two candidate models, blind-review output quality, and record critical failure rate, p50/p95 latency, cost per accepted result, and projected versus realized savings. Test a canary, forced rollback, budget cap, pinned-model bypass, credential rotation, and database restore before putting ARBR in the request path.