Real-time AI (audio/video in, voice out) on an M3 Pro with Gemma E2B
Get picks like this daily. Today's launch numbers, the top launches, AI and tech news, and a daily opportunity pick, straight to your inbox.
Want to reach these readers? Sponsor an issue.
Related: https://news.ycombinator.com/item?id=47653752
HN · launched Apr 5, 2026 · 298 upvotes (top 2%) · 0 views
Niche: multimodal ai platforms and tools · 215 launches, 16 in the last 7 days ▼ -47% · Explore this niche →
Category and tags
- Horizontal
- Model & infra
- Developer
- AI-native
- Hobby / open-source project
Visit website ↗ Source listing ↗
More like this: all 215 launches in multimodal ai platforms and tools →
Could you build this?
No Real-time multimodal processing (audio and video in, streaming voice out) on local Apple Silicon requires low-latency Metal/MPS kernel optimizations, real-time audio pipeline synchronization, and local multimodal model inference.
What it would actually take: The architecture requires an optimized C++/Metal/MLX pipeline to run quantized multimodal models (like Gemma) alongside low-latency audio capture/streaming (AVFoundation, WebRTC), speech-to-text, and zero-latency neural TTS. The hardest part is streaming concurrent audio/video frame ingestion into a single autoregressive context window while keeping round-trip conversational latency under 500ms on consumer hardware. This requires deep systems programming, real-time DSP, and GPU-level inference optimization expertise.
Discussion
20 comments analyzed.
Competitors mentioned: Google Home voice assistant, Apple Siri, Gemma E2B, Google Gemma4
Concerns raised: Response time too slow for real-time use (2.5s+ latency), Voice recognition performance inadequate on consumer hardware (M1 Max, RTX 5060 Ti), Requires internet connection to function, Image processing adds ~0.5s overhead, blocking video real-time capability, Video understanding limited to frame snapshots, not temporal context
Feature requests: Reduce response time to sub-100ms for true real-time, Disable image input option to improve speed, Add filler speech sounds ("uh", "umm") during processing, Real-time video understanding with temporal context awareness
Competitors
Other products that read as similar to this one : 244 launches clear the similarity bar, closest 8 shown.
Attention rank: #5 of 245 (itself plus its competitors, highest first, normalized so YC and Product Hunt are compared fairly).
Launched 153 days after the earliest competitor.
- Emma, a real-time AI video persona from Nuance Labs · hn · 2026-11-09 · 9 upvotes · similarity 0.46
- MJ-X · ph · 2026-11-14 · 2 upvotes · similarity 0.43
- VIDEO AI ME · ph · 2026-04-27 · 206 upvotes · similarity 0.42
- Lucy Labs · ph · 2026-09-22 · 2 upvotes · similarity 0.42
- mib2q-carplay · github · 2026-10-01 · 15 upvotes · similarity 0.42
- SpeechifyAI Simba Voice Agents · ph · 2026-07-13 · 161 upvotes · similarity 0.41
- Flying Car AI · ph · 2026-10-20 · 1 upvotes · similarity 0.41
- Multimodal perception system for real-time conversation · hn · 2026-02-10 · 54 upvotes · similarity 0.40
The niche, labels, vibe code rating and competitor comparison on this page are analysis by Nicheloom. If you use them, please credit nicheloom.com. How you may use our data.