LiveKit Agents Framework

LiveKit Agents Framework is an open-source Python framework for building real-time voice and multimodal AI agents. It provides the complete pipeline for voice-based agent interactions: speech-to-text transcription, LLM processing, text-to-speech synthesis, and real-time audio/video streaming — all integrated into a coherent framework with low-latency performance.

The framework is built on LiveKit's real-time communication infrastructure, which handles the complex networking, codec management, and streaming protocols required for low-latency audio/video. This means developers focus on agent logic rather than WebRTC, audio processing, and network engineering.

The VoicePipelineAgent is the framework's flagship component. It orchestrates the STT→LLM→TTS pipeline with built-in turn detection, interruption handling, and conversation flow management. The agent can detect when a user stops speaking, process their input, generate a response, and speak it — all with sub-second latency when using optimized providers.

The framework supports multiple STT providers (Deepgram, AssemblyAI, Azure, Google, OpenAI Whisper), LLM providers (OpenAI, Anthropic, Google, local models), and TTS providers (ElevenLabs, OpenAI, Azure, Google, Cartesia). This mix-and-match approach lets developers optimize for quality, latency, and cost across each pipeline stage.

Multimodal support extends beyond voice. Agents can process video input, share their screen, send images, and use vision models — enabling use cases like visual assistants, remote diagnostics, and interactive tutoring. The framework handles the complexity of synchronizing multiple media streams.

LiveKit Agents Framework includes built-in function calling, allowing voice agents to execute tools during conversations. An agent can search databases, call APIs, process files, and control external systems while maintaining a natural voice conversation — a critical capability for practical voice agent applications.

The framework deploys on LiveKit's infrastructure or self-hosted LiveKit servers. It supports horizontal scaling, session management, and load balancing for production deployments serving many concurrent voice sessions.

For teams building voice-first AI agents — customer support, virtual receptionists, voice assistants, interactive tutoring, or accessibility tools — LiveKit Agents Framework provides the most complete open-source solution for real-time, low-latency voice AI.