KV-psi, using Linux PSI to to trim an LLM KV cache
Get picks like this daily. Today's launch numbers, the top launches, AI and tech news, and a daily opportunity pick, straight to your inbox.
Want to reach these readers? Sponsor an issue.
I thought it'd be interesting to use Linux PSI (Pressure Stall Information) for an LLM runtime to trim the KV cache. This is mainly useful imo for edge devices like the Jetson Orin super nano kit which have unified memory. I haven't benched much, but plan to do so more over time and see if I can make a real use of it as I run local LLMs. Let me know if it makes sense :P (I of course vibed this idea)
HN · launched Jun 27, 2026 · 8 upvotes (top 50%) · 0 views
Niche: llm inference and deployment tools · 188 launches, 6 in the last 7 days · Explore this niche →
Category and tags
- Horizontal
- Model & infra
- Developer
- AI-native
- Hobby / open-source project
Visit website ↗ Source listing ↗
More like this: all 188 launches in llm inference and deployment tools →
Could you build this?
No Integrating Linux kernel Pressure Stall Information (PSI) triggers with low-level LLM inference engines (such as llama.cpp or vLLM) to dynamically evict KV cache tokens requires deep kernel and GPU memory architecture expertise.
What it would actually take: This requires modifying low-level C++/CUDA inference runtimes (like vLLM or llama.cpp) to interface with Linux `/proc/pressure/` epoll triggers. The difficult challenges involve designing non-disruptive, latency-critical KV cache quantization/pruning algorithms that preserve context coherence without causing GPU pipeline stalls on unified memory architectures like Jetson. This demands deep systems programming and ML systems hardware optimization skills.
Discussion
No comments on this launch.
Competitors
Other products that read as similar to this one : 376 launches clear the similarity bar, closest 8 shown.
Attention rank: #185 of 377 (itself plus its competitors, highest first, normalized so YC and Product Hunt are compared fairly).
Launched 239 days after the earliest competitor.
- Shoehorn, a library to quantize an LLM to fit your Mac's VRAM · hn · 2026-08-14 · 6 upvotes · similarity 0.56
- Makes local LLMs faster and more reliable by optimizing for your device · hn · 2026-06-30 · 6 upvotes · similarity 0.52
- nano-lmcache · github · 2026-10-19 · 23 upvotes · similarity 0.51
- Cuts Long Horizon Inference Costs by 50% via external KV Cache Offload · hn · 2026-07-26 · 22 upvotes · similarity 0.51
- ExANS · hn · 2026-08-05 · 15 upvotes · similarity 0.48
- GhostBox · hn · 2026-05-01 · 126 upvotes · similarity 0.46
- Lowfat · hn · 2026-06-05 · 156 upvotes · similarity 0.45
- Tokenflood · hn · 2025-11-12 · 21 upvotes · similarity 0.45
The niche, labels, vibe code rating and competitor comparison on this page are analysis by Nicheloom. If you use them, please credit nicheloom.com. How you may use our data.