Self-Hosted Voice AI: A Local-First Hybrid Pipeline

At a Glance

The Product

Voice assistants often send captured audio to someone else's cloud. For regulated industries, internal tools, and privacy-conscious deployments, reducing that exposure can be a core architectural requirement.

This project demonstrates a local-first alternative. WebRTC delivers real-time audio, and the VAD, STT, and LLM stages run locally on the host. The default Edge-TTS provider sends generated response text to Microsoft Edge's online text-to-speech service and returns synthesized audio, so the deployed system is hybrid rather than local-only.

The engagement centered on four objectives:

• Achieve natural, low-latency voice conversation over real-time WebRTC transport.
• Keep voice detection, transcription, and reasoning local with Silero VAD, faster-whisper, and Ollama.
• Make the TTS provider swappable, with Edge-TTS as the online default and configuration available for other providers.
• Make consumer hardware sufficient, with Apple Silicon GPU acceleration doing the heavy lifting.

The Challenge

The core tension of this project was latency versus data placement. Cloud voice APIs are fast because of data-center GPUs; running the compute-heavy VAD, STT, and LLM stages on local hardware means each one has to earn its milliseconds while the online TTS boundary remains visible.

1. Four models, one deadline. A voice turn chains voice activity detection, speech-to-text, LLM inference, and text-to-speech. Their latencies add up, and past roughly two seconds, conversation stops feeling like conversation.

2. Real-time audio is unforgiving. HTTP request-response cannot carry natural dialogue. The system needed WebRTC transport, utterance boundary detection, and interruption handling, the plumbing that separates a voice assistant from a voice memo.

3. Local hardware, real constraints. The VAD, STT, and LLM stages had to run on a MacBook, which meant choosing models that fit in memory, exploiting Metal GPU acceleration, and resisting the temptation to solve every problem with a bigger model.

4. Flexibility without forking. Some deployments will want Groq, OpenAI, or Cartesia for a stage. The pipeline had to swap providers through configuration, keeping one codebase across local-first and more cloud-dependent setups.

My Approach

I built the system on LiveKit Agents, with a clean separation between media transport, the agent worker, and the AI pipeline.

WebRTC transport via LiveKit. A Dockerized LiveKit server handles media routing. The browser client, built with Next.js 15 and React 19, requests a JWT from a FastAPI token service and connects over WebRTC. LiveKit dispatches each session to an agent worker that runs the conversation.

A four-stage local-first pipeline. VAD, STT, and LLM run locally: Silero detects speech segments, faster-whisper (large-v3-turbo) transcribes with Metal acceleration, and Ollama running llama3.2:3b generates responses. The default Edge-TTS provider sends response text to Microsoft Edge's online TTS service and streams the synthesized audio back into the pipeline.

Resilience wrappers on every stage. SafeSTT, SafeLLM, and SafeTTS wrappers give each stage graceful error handling, so a single model hiccup degrades the turn instead of killing the session.

Configuration-driven providers. Every stage reads its provider from environment configuration. Swap the LLM to Groq, the STT to OpenAI, or the TTS to Cartesia by editing .env; the system auto-detects provider types at startup. The TTS provider is swappable, so the online boundary is explicit rather than hidden.

Engineering Highlights

• LiveKit Agents orchestration. SDK 1.5 voice pipeline with proper utterance detection and turn handling, not a walkie-talkie loop.
• Local-first hybrid stack. Silero VAD, faster-whisper, and Ollama run locally; default Edge-TTS uses Microsoft Edge's online speech service.
• Observed sub-2s conversational latency. Seen in local development on Apple Silicon with streaming handoffs and Metal acceleration; not a controlled cross-hardware benchmark or guarantee.
• Safe wrappers per stage. Graceful degradation on STT, LLM, and TTS failures keeps sessions alive through transient errors.
• Provider swapping via .env. Any stage moves to Groq, OpenAI, Cartesia, or any OpenAI-compatible API without code changes.
• JWT-secured sessions. FastAPI service issuing tokens and health checks, keeping room access controlled.
• One-command media layer. Dockerized LiveKit server for reproducible WebRTC infrastructure.
• Modern web client. Next.js 15, React 19, and Tailwind CSS 4 frontend with live conversation UI.

Results

Tech Stack and Deliverables

• Platform: LiveKit Agents SDK 1.5, FastAPI, Python 3.12, Docker, WebRTC
• AI pipeline: Silero VAD, faster-whisper (large-v3-turbo), Ollama (llama3.2:3b), Edge-TTS
• Frontend: Next.js 15, React 19, Tailwind CSS 4
• Deliverables: Complete voice assistant stack, local-first hybrid pipeline, provider-swapping configuration system, resilience wrappers, JWT token service, Dockerized media infrastructure, web client