Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s
Get picks like this daily. Today's launch numbers, the top launches, AI and tech news, and a daily opportunity pick, straight to your inbox.
Want to reach these readers? Sponsor an issue.
I built slotstream, a way to run Qwen3.8-Flash-Next 4-bit on a low-memory mac starting from 16GB, a 125B parameter model that would need 100GB+ memory/RAM, thanks to expert-offloading/ssd-streaming. Easy to install/update, and mac-native using MLX and Swift.It ships with auto-mode, which makes a good tradeoff between memory usage and speed. I'll be implementing and porting the MTP module for speculative decoding next
HN · launched Sep 1, 2026 · 240 upvotes (top 2%) · 1 view
Niche: llm inference engines · 133 launches, 4 in the last 7 days ▼ -33% · Explore this niche →
Category and tags
- Horizontal
- Model & infra
- Developer
- AI-native
- Hobby / open-source project
Visit website ↗ Source listing ↗
More like this: all 133 launches in llm inference engines →
Could you build this?
No Slotstream implements custom low-level SSD streaming, expert offloading, and memory paging for 125B MoE models directly against Apple Silicon hardware via Swift and MLX.
What it would actually take: Requires low-level Apple Silicon kernel and memory optimization using Swift, Metal Performance Shaders (MPS), and the MLX C++ API. Developing dynamic weight streaming requires direct I/O (O_DIRECT, posix_fadvise) asynchronously pulling expert matrices from NVMe storage straight into unified memory ahead of execution layers. This requires deep systems programming, systems architecture, and GPU compute pipeline optimization skills.
Discussion
20 comments analyzed.
Competitors mentioned: ollama, oMLX, MTP (Medusa Token Prediction), vLLM
Concerns raised: Memory bandwidth constraints limit performance gains, Context length degradation (model behavior degrades above 70-80k tokens), High per-process memory usage (8.1GB per slotserve process), Disk I/O not fully optimized for non-sequential expert weight access, Installation time and complexity (1 hour setup)
Feature requests: Native desktop app wrapper, Improved local hardware optimization tooling (like MTPLX interface), MacOS Swift app integration, Reduce per-process memory footprint, Performance metrics and hardware comparison benchmarks
Competitors
Other products that read as similar to this one : 339 launches clear the similarity bar, closest 8 shown.
Attention rank: #12 of 340 (itself plus its competitors, highest first, normalized so YC and Product Hunt are compared fairly).
Launched 298 days after the earliest competitor.
- qwen38-flash-next-w4a16-cmp170hx · github · 2026-09-16 · 10 upvotes · similarity 0.75
- lean-qwenflash · github · 2026-10-24 · 48 upvotes · similarity 0.70
- collabosm · github · 2026-09-25 · 90 upvotes · similarity 0.68
- qwen38-flash-next-b70-offload · github · 2026-10-06 · 21 upvotes · similarity 0.63
- glyd-condor · github · 2026-10-14 · 29 upvotes · similarity 0.60
- Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone · hn · 2026-08-03 · 312 upvotes · similarity 0.59
- Speedy · github · 2026-11-01 · 35 upvotes · similarity 0.58
- Shoehorn, a library to quantize an LLM to fit your Mac's VRAM · hn · 2026-08-14 · 6 upvotes · similarity 0.57
The niche, labels, vibe code rating and competitor comparison on this page are analysis by Nicheloom. If you use them, please credit nicheloom.com. How you may use our data.