// the find
huggingface/speech-to-speech
Build local voice agents with open-source models
A modular VAD→STT→LLM→TTS voice agent pipeline from Hugging Face that exposes an OpenAI Realtime-compatible WebSocket API. You can mix-and-match backends for each stage, run fully local or point at hosted APIs, and drop it behind any Realtime-compatible client. It's actually running in production on Reachy Mini robots, which gives it credibility beyond a demo.
- OpenAI Realtime protocol compatibility means you can swap it in for the hosted API without rewriting client code — that's a genuinely useful abstraction boundary.
- Backend coverage is wide and practical: Parakeet, Whisper variants, Faster Whisper, Paraformer for STT; Qwen3-TTS, Kokoro, Pocket TTS for speech output; llama.cpp, vLLM, mlx-lm, or any OpenAI-compatible endpoint for the LLM. Platform markers in pyproject.toml handle macOS vs. Linux differences automatically.
- Test suite is real — there are unit tests for VAD iterator, speculative turns, transcription events, tool call parsing, WebSocket session lifecycle, and the realtime service. Not just smoke tests.
- The VAD tuning parameters (min_speech_ms, min_speech_continuation_ms, unanswered_reopen_ms) show someone has actually thought about turn-taking edge cases rather than shipping a naive threshold.
- Dependency management is painful: the CUDA wheel situation for Qwen3-TTS is user-hostile (manually pip-installing from a personal HF dataset before installing the package), and the numpy<2 vs numpy>=2 conflict between DeepFilterNet and Pocket TTS will silently bite people who install both.
- Hard-pinned versions in the macOS dependencies (mlx==0.31.1, mlx-lm==0.31.1, transformers==5.6.2, openai==2.28.0) will cause resolver conflicts the moment any other package in the environment has different requirements.
- The TCP socket mode is explicitly described as missing interruption handling, live transcripts, and tool-call events — so it's not a real alternative to the Realtime mode, but it's still surfaced as a first-class option in the docs without a strong deprecation warning.
- No latency benchmarks or profiling guidance are included. For a project where latency is the main selling point, there's no data on what to expect end-to-end on typical hardware, making it hard to evaluate before committing to the stack.