Why Go Local for Your Voice Robot?
Cloud-based voice assistants are convenient, but they come with privacy concerns, recurring API costs, and limited control. Running everything locally on your own hardware changes the game:
- Privacy: Audio never leaves your network.
- Zero API fees: No per-minute or per-token charges.
- Full flexibility: Swap any component (VAD, STT, LLM, TTS) whenever a better model appears.
This guide walks you through setting up a complete local speech-to-speech pipeline for the Reachy Mini robot, using the open-source speech-to-speech library from Hugging Face.
Step-by-Step Setup
1. Serve the LLM Locally
We'll use llama.cpp to run a powerful model like Gemma 4 E4B. Install it via brew install llama.cpp (macOS) or winget install llama.cpp (Windows). Then start the server:
llama-server -hf ggml-org/gemma-4-E4B-it-GGUF -np 2 -c 65536 -fa on --swa-full
Flag breakdown:
-hf ...– pulls the model from Hugging Face Hub (first run downloads it).-np 2– two parallel slots for handling interruptions.-c 65536– 64k context window for long conversations.-fa on– enables flash attention for speed and lower memory.--swa-full– keeps full sliding-window attention cache (faster prompt processing).
2. Install and Run Speech-to-Speech
In a new terminal, install the library:
uv pip install speech-to-speech
Then run the pipeline in local mode (for testing via terminal):
speech-to-speech --responses_api_base_url "http://127.0.0.1:8080" --responses_api_api_key "" --mode local
This will download Parakeet-TDT 0.6B v3 (STT) and Qwen3-TTS (TTS) on first run.
3. Connect Reachy Mini
Once the local mode works, restart the speech-to-speech server without --mode local to serve the robot. In the Reachy Mini desktop app, open the conversation app and select "edit connection" to choose the local HF backend. You're now ready to talk to your robot—fully offline!
Customization and Advanced Options
Choosing Your LLM Backend
You can swap the LLM to fit your needs. Here are some options:
| Backend | Command | Best For |
|---|---|---|
| llama.cpp | speech-to-speech --llm_backend responses-api --model_name "ggml-org/gemma-4-E4B-it-GGUF" | Fast local inference |
| vLLM | vllm serve Qwen/Qwen3-4B-Instruct-2507 --enable-auto-tool-choice --tool-call-parser qwen3_coder | High throughput |
| MLX (Apple Silicon) | speech-to-speech --llm_backend mlx-lm --model_name "mlx-community/Qwen3-4B-Instruct-2507-bf16" | Mac users |
| Transformers | speech-to-speech --llm_backend transformers --model_name "Qwen/Qwen3-4B-Instruct-2507" | CUDA/CPU |
| OpenAI-compatible | speech-to-speech --responses_api_base_url "https://api.openai.com/v1" | Cloud fallback |
Important Considerations
- Latency: The LLM is the bottleneck. Disable thinking mode (
--default-chat-template-kwargs '{"enable_thinking":false}') for natural conversations. - Hardware: For best results, use a GPU with at least 8GB VRAM. MTP speculative decoding (vLLM) can reduce latency.
- Network: To run the engine on a laptop and robot on the same LAN, bind to
0.0.0.0and use the laptop's IP (e.g.,192.168.x.x).
Next Steps
- Experiment with different STT/TTS models (e.g., Whisper, Coqui) to fine-tune quality vs. speed.
- Explore tool-calling capabilities for more complex agentic tasks.
- Check out Beyond Runtime Guardrails: The Missing Trust Layer for AI Agents for securing your voice agent.
Conclusion
You now have a fully local voice loop: Silero VAD detects speech, Parakeet-TDT transcribes it, your chosen LLM generates a response, and Qwen3-TTS speaks it back. This setup offers complete privacy, zero API costs, and endless customization. Star the repos and share your custom pipeline in the discussions!
Source: Hugging Face Blog
Related Reading: CSS border-shape: The Game-Changer for Shaped Elements with Borders