I built a sub-500ms latency voice agent from scratch
Get picks like this daily. Today's launch numbers, the top launches, AI and tech news, and a daily opportunity pick, straight to your inbox.
Want to reach these readers? Sponsor an issue.
I built a voice agent from scratch that averages ~400ms end-to-end latency (phone stop → first syllable). That’s with full STT → LLM → TTS in the loop, clean barge-ins, and no precomputed responses.What moved the needle:Voice is a turn-taking problem, not a transcription problem. VAD alone fails; you need semantic end-of-turn detection.The system reduces to one loop: speaking vs listening. The two transitions - cancel instantly on barge-in, respond instantly on end-of-turn - define the experience.STT → LLM → TTS must stream. Sequential pipelines are dead on arrival for natural conversation.TTFT dominates everything. In voice, the first token is the critical path. Groq’s ~80ms TTFT was the single biggest win.Geography matters more than prompts. Colocate everything or you lose before you start.GitHub Repo: https://github.com/NickTikhonov/shuoFollow whatever I next tinker with: https://x.com/nick_tikhonov
HN · launched Mar 2, 2026 · 570 upvotes (top 1%) · 0 views
Niche: ai voice agents and platforms · 197 launches, 20 in the last 7 days ▲ +25% · Explore this niche →
Category and tags
- Horizontal
- Agent / copilot
- Developer
- AI-native
- Hobby / open-source project
Visit website ↗ Source listing ↗
More like this: all 197 launches in ai voice agents and platforms →
Could you build this?
Partial Wiring together STT, LLM, and TTS APIs is straightforward, but achieving sub-500ms voice turn-taking requires custom low-latency streaming pipeline orchestration, optimized VAD, and server co-location.
What it would actually take: The system requires a persistent WebRTC or bidirectional WebSocket audio pipeline, often in Go, Rust, or optimized Node.js, combined with a fast client-side/server-side Voice Activity Detection (VAD) model (like Silero VAD) to detect intent and handle instant barge-in cancellations. The critical bottleneck is end-to-end latency: chunking streaming STT (e.g., Deepgram), speculative LLM streaming with token-level phrase chunking, and streaming TTS (e.g., Cartesia/ElevenLabs), with minimal network jitter by co-locating servers in the same cloud region. Building this reliably at scale requires deep real-time systems programming and network audio streaming expertise.
Discussion
20 comments analyzed.
Competitors mentioned: Pipecat, Dograh, Sesame, Moshi, Twilio + ElevenLabs stack
Concerns raised: Barge-in cancellation timing issues across providers, Uncanny valley risk with mistimed filler words, Cultural differences in conversation turn-taking expectations, Language-specific challenges (verb position in Japanese/German)
Feature requests: Long-term memory/context between calls, Echo handling, Tool calls for external services, Semantic VAD improvements, Hardware wake-word models for low-power devices
Competitors
Other products that read as similar to this one : 317 launches clear the similarity bar, closest 8 shown.
Attention rank: #2 of 318 (itself plus its competitors, highest first, normalized so YC and Product Hunt are compared fairly).
Launched 115 days after the earliest competitor.
- RealtimeVAD · hn · 2026-11-12 · 32 upvotes · similarity 0.63
- Realtime TTS-2 · ph · 2026-05-06 · 152 upvotes · similarity 0.48
- Mistral Voxtral 3 · ph · 2026-10-22 · 194 upvotes · similarity 0.48
- genpark-realtime-voice-agent-latency-telemetry-skill · github · 2026-09-26 · 7 upvotes · similarity 0.48
- Sparrow-2 · hn · 2026-09-08 · 11 upvotes · similarity 0.48
- genpark-voice-turn-taking-endpoint-detector-skill · github · 2026-09-26 · 7 upvotes · similarity 0.47
- Nari Qwen3-TTS and Qwen3-ASR · hn · 2026-09-14 · 90 upvotes · similarity 0.46
- Sparrow-1 · hn · 2026-01-14 · 123 upvotes · similarity 0.45
The niche, labels, vibe code rating and competitor comparison on this page are analysis by Nicheloom. If you use them, please credit nicheloom.com. How you may use our data.