Run 500B+ Parameter LLMs Locally on a Mac Mini
Get picks like this daily. Today's launch numbers, the top launches, AI and tech news, and a daily opportunity pick, straight to your inbox.
Want to reach these readers? Sponsor an issue.
Hi HN, I built OpenGraviton, an open-source AI inference engine that pushes the limits of running extremely large LLMs on consumer hardware. By combining 1.58-bit ternary quantization, dynamic sparsity with Top-K pruning and MoE routing, and mmap-based layer streaming, OpenGraviton can run models far larger than your system RAM—even on a Mac Mini. Early benchmarks: TinyLlama-1.1B drops from ~2GB (FP16) to ~0.24GB with ternary quantization. At 140B scale, models that normally require ~280GB fit within ~35GB packed. Optimized for Apple Silicon with Metal + C++ tensor unpacking, plus speculative decoding for faster generation. Check benchmarks, architecture, and details here: https://opengraviton.github.io GitHub: https://github.com/opengraviton This project isn’t just about squeezing massive models onto tiny hardware—it’s about democratizing access to giant LLMs without cloud costs. Feedback, forks, and ideas are very welcome!
HN · launched Mar 9, 2026 · 17 upvotes (top 29%) · 1 view
Niche: llm inference engines · 133 launches, 4 in the last 7 days ▼ -33% · Explore this niche →
Category and tags
- Horizontal
- Model & infra
- Developer
- AI-native
- Hobby / open-source project
Visit website ↗ Source listing ↗
More like this: all 133 launches in llm inference engines →
Could you build this?
No Creating an inference engine capable of running 500B+ parameter models on consumer hardware demands expert low-level systems programming, custom Metal/GPU kernels, ternary quantization math, and layer streaming.
What it would actually take: This requires writing custom C++/Metal kernels for 1.58-bit ternary matrix-vector multiplications, building dynamic Top-K routing routines for Mixture-of-Experts, and managing OS virtual memory via customized mmap layer-streaming architectures to overcome physical RAM constraints. Developers need deep expertise in Apple Silicon hardware architecture, memory bandwidth saturation, and low-bit quantized model compilation.
Discussion
10 comments analyzed.
Competitors mentioned: Gemini API, Cloud-based LLM APIs
Concerns raised: Quantization to 1.58-bit creates only 'mini' versions of models, not true full models, Benchmarks lack real-world results for larger-than-memory models beyond TinyLlama, Memory pressure and thermal throttling on sustained inference with Mac Mini 16GB, Latency degradation when layers stream from disk during inference at 140B scale, Sloppy or incomplete benchmark documentation
Feature requests: Benchmarks for 140B models on base Mac Mini 16GB configuration, Performance data for M1 Max and M3 Ultra systems, Latency metrics for layer streaming from disk mid-inference, Real-world token/sec results for larger models beyond TinyLlama
Competitors
Other products that read as similar to this one : 468 launches clear the similarity bar, closest 8 shown.
Attention rank: #165 of 469 (itself plus its competitors, highest first, normalized so YC and Product Hunt are compared fairly).
Launched 126 days after the earliest competitor.
- OpenGraviton · hn · 2026-03-07 · 13 upvotes · similarity 0.89
- Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac · hn · 2026-07-29 · 919 upvotes · similarity 0.53
- Llm.sql · hn · 2026-04-24 · 8 upvotes · similarity 0.52
- Tiny-vLLM · hn · 2026-05-29 · 205 upvotes · similarity 0.50
- strixite · github · 2026-10-04 · 11 upvotes · similarity 0.50
- IronMule · ph · 2026-09-17 · 1 upvotes · similarity 0.50
- Shoehorn, a library to quantize an LLM to fit your Mac's VRAM · hn · 2026-08-14 · 6 upvotes · similarity 0.49
- You can now train your own LLMs with less memory, on your consumer GPUs · hn · 2026-11-10 · 5 upvotes · similarity 0.49
The niche, labels, vibe code rating and competitor comparison on this page are analysis by Nicheloom. If you use them, please credit nicheloom.com. How you may use our data.