Compressing a Histogram from 16GB to 2KB
Details
- External ID
- 46634366
- Source
- HN
- Company
- —
- Product
- Compressing a Histogram from 16GB to 2KB
- Website domain
- alexkranias.com
- Launched
- Jan. 15, 2026
- Cohort
- —
- Upvotes
- 5
- Upvotes percentile
- 0.09617918313570488
- Tags
- —
- Fetched at
- Sept. 7, 2026, 9:25 p.m.
- Updated at
- Sept. 7, 2026, 9:25 p.m.
Description
Thought I'd dive into this cool streaming/approximation algorithms problem I encountered a few months ago.TLDR: we can create our own custom floating point representation for encoding and decoding integers and use this to index into a tiny 2D histogram, upper-bounding approximation error based on the number of bits per integer we specify to keep.---Broadly useful for aggregating statistics from massive data streams of user data. Also turns out this is incredibly similar to data structures used in production, like HDRHistogram.
Enrichment
- Theme
- 3D graphics and physics simulation tools
- Vertical
- Horizontal
- Function
- Data infrastructure
- Audience
- Developer
- AI stance
- Not AI
- Project type
- Hobby / open-source project
- Normalized one-liner
- histogram compression algorithm
- Manually corrected
- False
Could you build this?
No This is low-level streaming algorithm research designing a custom floating-point encoding to compress high-dimensional histograms with bounded error.
What it would actually take: The solution requires designing a custom compression algorithm involving non-standard floating-point representations, bit-level packing, and error-bounded streaming histograms (similar to t-digest or DDSketch). Implementing this necessitates low-level systems programming (C/C++ or Rust) and deep theoretical knowledge of approximation algorithms, streaming analytics, and numeric precision.
Discussion
No comments on this launch.
Competitors
Other products that read as similar to this one — 76 launches clear the similarity bar, closest 8 shown.
Attention rank: #71 of 77 (itself plus its competitors, highest first — normalized so YC and Product Hunt are compared fairly).
Launched 75 days after the earliest competitor.
- Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB · hn · 2026-06-23 · 112 upvotes · similarity 0.48
- Coderive · hn · 2025-12-22 · 8 upvotes · similarity 0.42
- ToolKB · ph · 2026-09-20 · 1 upvotes · similarity 0.41
- FastLanes based integer compression in Zig · hn · 2025-12-01 · 12 upvotes · similarity 0.40
- ExANS · hn · 2026-08-05 · 15 upvotes · similarity 0.40
- genpark-run-length-encoding-rle-delta-skill · github · 2026-09-28 · 7 upvotes · similarity 0.40
- Image In Kb · ph · 2026-09-07 · 1 upvotes · similarity 0.39
- SwiftShrink · hn · 2026-07-02 · 6 upvotes · similarity 0.39
Other launches for this product
- No other launches for this product.
Same idea, different domain
Nobody's really built a data infrastructure tool for Media & entertainment yet.