Makes local LLMs faster and more reliable by optimizing for your device
Get picks like this daily. Today's launch numbers, the top launches, AI and tech news, and a daily opportunity pick, straight to your inbox.
Want to reach these readers? Sponsor an issue.
Time to first token is 39% faster Agent wall times decrease by 46% No swapsTracks your resource usage in real-time and adjusts how the model runs so that it works perfectly on your device.Implements KV cache sizing, prefix caching, live RAM pressure management, context trimming, KV quantization, and more.Built a ton of features
HN · launched Jun 30, 2026 · 6 upvotes (top 69%) · 0 views
Niche: llm optimization tools · 129 launches, 5 in the last 7 days ▼ -29% · Explore this niche →
Category and tags
- Horizontal
- Model & infra
- Developer
- AI-native
- Commercial product
Visit website ↗ Source listing ↗
More like this: all 129 launches in llm optimization tools →
Could you build this?
Partial While wrapping Ollama as a reverse proxy is straightforward, managing dynamic OS memory pressure, dynamic Metal buffer resizing, and quantization transitions without crashing local inference requires non-trivial systems programming.
What it would actually take: Built as an HTTP proxy in Python or Go intercepting Ollama's API, calculating prompt token lengths, and inspecting host memory stats via OS syscalls (e.g., mach APIs on macOS or /proc/meminfo on Linux). The difficult challenge is reliably predicting KV cache memory overhead, tuning Metal/CUDA buffer buckets to prevent memory fragmentation, and cleanly orchestrating model reloads or context trims without corrupting ongoing token generation streams.
Discussion
No comments on this launch.
Competitors
Other products that read as similar to this one : 140 launches clear the similarity bar, closest 8 shown.
Attention rank: #104 of 141 (itself plus its competitors, highest first, normalized so YC and Product Hunt are compared fairly).
Launched 230 days after the earliest competitor.
- KV-psi, using Linux PSI to to trim an LLM KV cache · hn · 2026-06-27 · 8 upvotes · similarity 0.52
- Llmspeed · hn · 2026-10-17 · 329 upvotes · similarity 0.51
- Long term Memory and 50M token window for LLM · hn · 2026-10-09 · 6 upvotes · similarity 0.47
- local-llm-performance-profile-switcher · github · 2026-10-19 · 20 upvotes · similarity 0.43
- Rapid-MLX · hn · 2026-04-18 · 9 upvotes · similarity 0.43
- local-llm-tuning-guide · github · 2026-10-16 · 21 upvotes · similarity 0.42
- Cuts Long Horizon Inference Costs by 50% via external KV Cache Offload · hn · 2026-07-26 · 22 upvotes · similarity 0.42
- Optimizing LiteLLM with Rust · hn · 2025-11-18 · 27 upvotes · similarity 0.42
The niche, labels, vibe code rating and competitor comparison on this page are analysis by Nicheloom. If you use them, please credit nicheloom.com. How you may use our data.