Compressing a Histogram from 16GB to 2KB
Get picks like this daily. Today's launch numbers, the top launches, AI and tech news, and a daily opportunity pick, straight to your inbox.
Want to reach these readers? Sponsor an issue.
Thought I'd dive into this cool streaming/approximation algorithms problem I encountered a few months ago.TLDR: we can create our own custom floating point representation for encoding and decoding integers and use this to index into a tiny 2D histogram, upper-bounding approximation error based on the number of bits per integer we specify to keep.---Broadly useful for aggregating statistics from massive data streams of user data. Also turns out this is incredibly similar to data structures used in production, like HDRHistogram.
HN · launched Jan 15, 2026 · 5 upvotes (top 90%) · 0 views
Niche: file and data compression tools · 100 launches, 14 in the last 7 days ▲ +75% · Explore this niche →
Category and tags
- Horizontal
- Data infrastructure
- Developer
- Not AI
- Hobby / open-source project
Visit website ↗ Source listing ↗
More like this: all 100 launches in file and data compression tools →
Could you build this?
No This is low-level streaming algorithm research designing a custom floating-point encoding to compress high-dimensional histograms with bounded error.
What it would actually take: The solution requires designing a custom compression algorithm involving non-standard floating-point representations, bit-level packing, and error-bounded streaming histograms (similar to t-digest or DDSketch). Implementing this necessitates low-level systems programming (C/C++ or Rust) and deep theoretical knowledge of approximation algorithms, streaming analytics, and numeric precision.
Discussion
No comments on this launch.
Competitors
Other products that read as similar to this one : 160 launches clear the similarity bar, closest 8 shown.
Attention rank: #151 of 161 (itself plus its competitors, highest first, normalized so YC and Product Hunt are compared fairly).
Launched 75 days after the earliest competitor.
- Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB · hn · 2026-06-23 · 112 upvotes · similarity 0.49
- Coderive · hn · 2025-12-22 · 8 upvotes · similarity 0.43
- FastLanes based integer compression in Zig · hn · 2025-12-01 · 12 upvotes · similarity 0.42
- genpark-run-length-encoding-rle-delta-skill · github · 2026-09-28 · 7 upvotes · similarity 0.42
- ExANS · hn · 2026-08-05 · 15 upvotes · similarity 0.41
- ordered-floating-point-addition · github · 2026-11-07 · 12 upvotes · similarity 0.41
- ToolKB · ph · 2026-09-20 · 1 upvotes · similarity 0.41
- I made a new compression algorithm · hn · 2026-01-29 · 9 upvotes · similarity 0.40
The niche, labels, vibe code rating and competitor comparison on this page are analysis by Nicheloom. If you use them, please credit nicheloom.com. How you may use our data.