Kestra

Kestra is an open-source orchestration platform used to run scheduled jobs, event-triggered pipelines, and AI/agent workflows at scale. Flows are declared in YAML — so they live in Git and are reviewable in pull requests — but the platform ships a rich web UI where non-engineers can run, monitor, and edit flows visually. Where legacy orchestrators like Airflow feel weighed down by Python operators and DAG boilerplate, Kestra separates orchestration from execution: any language, container, script, dbt project, Spark job, or LLM call can run inside a task, and the engine handles retries, backfills, dependencies, and parallel fan-out declaratively. Its 600+ plugins cover cloud storage, databases, warehouses (Snowflake, BigQuery, Redshift), messaging (Kafka, Pub/Sub), and — increasingly — LLM providers (OpenAI, Anthropic, Bedrock, Ollama) plus AI-specific plugins for embeddings, RAG, and prompt chaining. Kestra also acts as an MCP server so agent hosts like Claude Desktop and Cursor can inspect flows, trigger executions, and query state. Deployable via Docker or Kubernetes with a Postgres backend, Kestra has an active OSS community and a commercial Enterprise Edition offering SSO, RBAC, secrets managers, worker groups, and audit logs. Popular use cases include AI content pipelines, ETL with LLM enrichment, event-driven agent orchestration, and CI/CD for ML models.