OpenGraviton
Run 500B+ parameter models on a consumer Mac Mini
Get picks like this daily. Today's launch numbers, the top launches, AI and tech news, and a daily opportunity pick, straight to your inbox.
Want to reach these readers? Sponsor an issue.
Hi HN,I built OpenGraviton, an open-source AI inference engine designed to push the limits of running extremely large models on consumer hardware.The system combines several techniques to drastically reduce memory and compute requirements:• 1.58-bit ternary quantization ({-1, 0, +1}) for ~10x compression • dynamic sparsity with Top-K pruning and MoE routing • mmap-based layer streaming to load weights directly from NVMe SSDs • speculative decoding to improve generation throughputThese allow models far larger than system RAM to run locally.In early benchmarks, OpenGraviton reduced TinyLlama-1.1B from ~2.05GB (FP16) to ~0.24GB using ternary quantization. Synthetic stress tests at the 140B scale show that models which would normally require ~280GB FP16 can fit within ~35GB when packed with the ternary format.The project is optimized for Apple Silicon and currently uses custom Metal + C++ tensor unpacking.Benchmarks, architecture, and details: https://opengraviton.github.ioGitHub: https://github.com/opengraviton
HN · launched Mar 7, 2026 · 13 upvotes (top 33%) · 2 views
Niche: llm inference engines · 133 launches, 4 in the last 7 days ▼ -33% · Explore this niche →
Category and tags
- Horizontal
- Model & infra
- Developer
- AI-native
- Hobby / open-source project
Visit website ↗ Source listing ↗
More like this: all 133 launches in llm inference engines →
Could you build this?
No Implementing a custom local inference engine supporting 1.58-bit ternary quantization, layer-by-layer SSD streaming, and custom GPU/Metal compute kernels requires deep low-level systems and ML compilation expertise.
What it would actually take: Requires high-performance C++/Rust or Metal/CUDA engineering to write custom matrix multiplication kernels optimized for ternary or low-bit weights. The system needs asynchronous disk I/O pipelines (io_uring on Linux, direct I/O on macOS) to stream multi-gigabyte layer weights continuously into GPU unified memory without stalling compute. Building speculative decoding and dynamic sparsity into an engine requires expert-level understanding of transformer internals and hardware cache architectures.
Discussion
5 comments analyzed.
Concerns raised: Hardware detection issues, engine.generate() not implemented, Missing implementation yields empty string
Feature requests: Implement engine.generate() method, Improve hardware detection
Competitors
Other products that read as similar to this one : 668 launches clear the similarity bar, closest 8 shown.
Attention rank: #255 of 669 (itself plus its competitors, highest first, normalized so YC and Product Hunt are compared fairly).
Launched 123 days after the earliest competitor.
- Run 500B+ Parameter LLMs Locally on a Mac Mini · hn · 2026-03-09 · 17 upvotes · similarity 0.89
- Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac · hn · 2026-07-29 · 919 upvotes · similarity 0.56
- strixite · github · 2026-10-04 · 11 upvotes · similarity 0.53
- kindi · github · 2026-10-20 · 40 upvotes · similarity 0.51
- open_rknpu2_lm_infer · github · 2026-10-14 · 13 upvotes · similarity 0.50
- Serve 100 Large AI models on a single GPU with low impact to TTFT · hn · 2025-11-08 · 7 upvotes · similarity 0.49
- whirl-llm · github · 2026-10-03 · 15 upvotes · similarity 0.49
- IronMule · ph · 2026-09-17 · 1 upvotes · similarity 0.49
The niche, labels, vibe code rating and competitor comparison on this page are analysis by Nicheloom. If you use them, please credit nicheloom.com. How you may use our data.