I run 30B 22tok/s, 109tok/s not novel,6GB/16GB RAM overcoming llama.cpp
Get picks like this daily. Today's launch numbers, the top launches, AI and tech news, and a daily opportunity pick, straight to your inbox.
Want to reach these readers? Sponsor an issue.
Democratisation of local AI is key. I've been working on pushing the limits of commercial hardware, squeezing any extra bit possible. My Scientific Agentic AI hareness helped me to reallocate every single bit of it. I rewrote the Kernel, I went down the CUDA rabbit hole until I have been able to explain any bit and any ms of computational power involved in the process pushing the Qwen 30B-A3B from 8 tok7s to 19 tok/s with llama.cpp up to 22.2 tok/s with my project and 109 tok/s on not novel content and speeding up the prefill by 5-9X
HN · launched Jul 29, 2026 · 5 upvotes (top 89%) · 0 views
Niche: llm inference and deployment tools · 188 launches, 6 in the last 7 days · Explore this niche →
Category and tags
- Horizontal
- Model & infra
- Developer
- AI-native
- Hobby / open-source project
Visit website ↗ Source listing ↗
More like this: all 188 launches in llm inference and deployment tools →
Could you build this?
No Rewriting GPU inference kernels, custom memory paging, and optimizing low-level CUDA execution to outperform libraries like llama.cpp requires elite systems programming and GPU architecture expertise.
What it would actually take: This demands writing custom CUDA/C++ kernels, highly optimized quantized matrix multiplication (e.g., Marlin, AWQ, FP4/FP8 routines), tensor parallelism, and manual unified memory/VRAM-host streaming management. It requires specialized engineers deeply versed in NVIDIA hardware architecture (SMs, tensor cores, memory hierarchy) and modern LLM runtime internals.
Discussion
No comments on this launch.
Competitors
Other products that read as similar to this one : 802 launches clear the similarity bar, closest 8 shown.
Attention rank: #720 of 803 (itself plus its competitors, highest first, normalized so YC and Product Hunt are compared fairly).
Launched 273 days after the earliest competitor.
- OS Megakernel that match M5 Max Tok/w at 2x the Throughput on RTX 3090 · hn · 2026-04-08 · 6 upvotes · similarity 0.52
- GhostBox · hn · 2026-05-01 · 126 upvotes · similarity 0.52
- Shoehorn, a library to quantize an LLM to fit your Mac's VRAM · hn · 2026-08-14 · 6 upvotes · similarity 0.52
- llama-cpp-pascal · github · 2026-11-09 · 15 upvotes · similarity 0.50
- Llama 3.1 70B on a single RTX 3090 via NVMe-to-GPU bypassing the CPU · hn · 2026-02-21 · 395 upvotes · similarity 0.49
- I'm tired of my LLM bullshitting. So I fixed it · hn · 2026-01-22 · 5 upvotes · similarity 0.48
- Anos · hn · 2026-04-04 · 115 upvotes · similarity 0.48
- Serve 100 Large AI models on a single GPU with low impact to TTFT · hn · 2025-11-08 · 7 upvotes · similarity 0.48
The niche, labels, vibe code rating and competitor comparison on this page are analysis by Nicheloom. If you use them, please credit nicheloom.com. How you may use our data.