Llama 3.1 70B on a single RTX 3090 via NVMe-to-GPU bypassing the CPU
Get picks like this daily. Today's launch numbers, the top launches, AI and tech news, and a daily opportunity pick, straight to your inbox.
Want to reach these readers? Sponsor an issue.
Hi everyone, I'm kinda involved in some retrogaming and with some experiments I ran into the following question: "It would be possible to run transformer models bypassing the cpu/ram, connecting the gpu to the nvme?"This is the result of that question itself and some weekend vibecoding (it has the linked library repository in the readme as well), it seems to work, even on consumer gpus, it should work better on professional ones tho
HN · launched Feb 21, 2026 · 395 upvotes (top 1%) · 0 views
Niche: llm inference engines · 133 launches, 4 in the last 7 days ▼ -33% · Explore this niche →
Category and tags
- Horizontal
- Model & infra
- Developer
- AI-native
- Hobby / open-source project
Visit website ↗ Source listing ↗
More like this: all 133 launches in llm inference engines →
Could you build this?
No Bypassing the CPU/system RAM to stream LLM weights directly from NVMe to GPU requires low-level kernel driver manipulation and GPU Direct Storage (GDS) / SPDK engineering.
What it would actually take: This architecture requires low-level C++/CUDA systems engineering utilizing NVIDIA GPUDirect Storage (cuFile API) or custom user-space NVMe drivers (SPDK) over PCIe to transfer model layers directly into VRAM on demand. It involves designing asynchronous prefetching pipelines that overlap layer compute with NVMe DMA reads, precisely synchronized to memory bandwidth and GPU compute streams without CPU/OS page cache intervention.
Discussion
20 comments analyzed.
Competitors mentioned: MemeRadar, llama.cpp, vLLM, SGLang
Concerns raised: Cloud latency breaks real-time signal processing, Memory bandwidth bottleneck limits performance more than raw TFLOPS, Quantization loses precision needed for subtle pattern detection, Local hardware TCO beats cloud rentals for 24/7 production, MoE expert routing requires keeping all parameters in memory or waiting for slower storage loads
Feature requests: Predictive MoE expert loading to VRAM via PCIe before inference, CPU optimization for 50% speed improvement potential, Research on MoE scheduling and optimization, CXL memory support for improved bandwidth across system, Selective layer fine-tuning instead of full LoRA on all layers
Competitors
Other products that read as similar to this one : 558 launches clear the similarity bar, closest 8 shown.
Attention rank: #8 of 559 (itself plus its competitors, highest first, normalized so YC and Product Hunt are compared fairly).
Launched 115 days after the earliest competitor.
- LLMKube · hn · 2025-11-18 · 5 upvotes · similarity 0.50
- I run 30B 22tok/s, 109tok/s not novel,6GB/16GB RAM overcoming llama.cpp · hn · 2026-07-29 · 5 upvotes · similarity 0.49
- VRAMGlass · ph · 2026-09-19 · 1 upvotes · similarity 0.49
- fast-rtxvsr · github · 2026-09-09 · 15 upvotes · similarity 0.49
- CUDA/graphics in QEMU-KVM VMs without passing the Nvidia card to them · hn · 2026-09-08 · 5 upvotes · similarity 0.48
- You can now train your own LLMs with less memory, on your consumer GPUs · hn · 2026-11-10 · 5 upvotes · similarity 0.48
- geforce-sriov · github · 2026-11-03 · 82 upvotes · similarity 0.47
- HelixSR · github · 2026-10-05 · 161 upvotes · similarity 0.47
The niche, labels, vibe code rating and competitor comparison on this page are analysis by Nicheloom. If you use them, please credit nicheloom.com. How you may use our data.