senn-techAI Phone Agents: Voice Bots with Pipecat & Asterisk
AI chatbots are familiar. But what happens when the customer doesn’t type, but calls? The real-time voice pipeline — speech-to-text, LLM, text-to-speech — is technically demanding, but achievable with Pipecat and Asterisk.
The Pipeline: STT → LLM → TTS
A caller speaks. Asterisk (or another SIP-PBX) answers the call and routes the audio stream to Pipecat. Pipecat orchestrates:
- STT: Speech is transcribed in real time — Whisper, Deepgram, or a local model.
- LLM: The transcript is sent to an LLM that responds — context-aware, with prompts defining the agent.
- TTS: The text response is synthesized — ElevenLabs-class or local models like XTTS.
All of this happens in under two seconds. Latency is the critical path.
Pipecat: The Framework for the Pipeline
Pipecat is a Python framework that handles transport and pipeline logic. It communicates with Asterisk via SIP or WebRTC, controls audio input/output, and switches between pipeline stages.
from pipecat.pipeline.pipeline import Pipeline
from pipecat.services.deepgram import DeepgramSTTService
from pipecat.services.openai import OpenAILLMService
This code defines the pipeline. Pipecat manages the rest — audio chunks flow, the LLM receives context, the response goes back to the phone.
SIP Integration via Asterisk
Asterisk is the bridge to the telephone network. It receives the SIP trunk, routes the audio stream, and treats the AI agent like an extension — only without a human behind it.
The advantage: Asterisk can route both human agents and AI agents in parallel. Complex inquiries go to humans, standard questions (opening hours, status checks) to the bot.
What Still Doesn’t Work
Natural pauses, interruptions, emotional nuances — these remain research topics. Real-time voice is impressive, but latency and the inability to understand “uh” and hesitation remind after three sentences that no human is on the other end.
Conclusion
AI phone agents aren’t replacements for support staff, but they relieve the burden of standard questions. Pipecat and Asterisk make the technology accessible — the rest is prompt design and honest expectation management.
How does the real-time voice pipeline of an AI phone agent work?+
Asterisk answers the call and routes the audio stream to Pipecat. Pipecat orchestrates three stages: speech-to-text transcribes in real time, an LLM responds context-aware, and text-to-speech synthesizes the reply. All of this happens in under two seconds – latency is the critical path. STT, LLM, and TTS are each selectable, such as Whisper, a local LLM, or XTTS.
Can the AI agent hand off to a human employee?+
Yes. Asterisk is the bridge to the telephone network and treats the AI agent like an extension. It can route human agents and AI agents in parallel: complex inquiries go to a human, standard questions like opening hours or status checks to the bot. The agent therefore does not replace support staff but relieves them of recurring routine inquiries.
Where do today's limits of voice AI on the phone lie?+
Natural pauses, interruptions, and emotional nuance remain research topics. Real-time voice is impressive, but latency and the inability to understand a hesitant uh remind you after three sentences that no human is on the other end. AI phone agents are therefore not a replacement for support staff, but a relief for clearly scoped standard questions.