Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB
Get picks like this daily. Today's launch numbers, the top launches, AI and tech news, and a daily opportunity pick, straight to your inbox.
Want to reach these readers? Sponsor an issue.
I built an experiment that uses an overfitted transformer and arithmetic coding to compress individual files.Instead of training the model to generalize, I train a 900KB transformer to memorize a single file and predict the next byte. Those predictions are fed into an arithmetic coder to produce the compressed output.On a 100MB NYC taxi CSV, it compresses to about 7MB (~0.5 bits/byte). On a 100MB slice of enwik9, it compresses to about 21MB (~1.68 bits/byte).It's pretty slow right now (roughly 20–30 minutes of training and 45 minutes each for compression and decompression on my AMD 7800XT).Checkout the repo - https://github.com/samyak112/pym-particles
HN · launched Jun 23, 2026 · 112 upvotes (top 8%) · 0 views
Niche: file and data compression tools · 100 launches, 15 in the last 7 days ▲ +88% · Explore this niche →
Category and tags
- Horizontal
- Data infrastructure
- Developer
- AI-native
- Hobby / open-source project
More like this: all 100 launches in file and data compression tools →
Could you build this?
No This is a specialized machine learning and information theory research experiment involving custom miniature transformer architectures, next-byte entropy modeling, and integration with an arithmetic coding pipeline.
What it would actually take: The architecture involves PyTorch/JAX training an extremely compact auto-regressive transformer model to minimize cross-entropy loss on a single dataset, paired with an arithmetic coding implementation (often in C++ or Rust for performance). The hard parts are tuning model capacity vs. parameter footprint to avoid under/over-saturating the arithmetic coder, and handling finite-precision arithmetic coder stability with neural network logits. It requires deep research expertise in data compression, entropy coding, and neural network optimization.
Discussion
20 comments analyzed.
Competitors mentioned: ZPAQ compression algorithm, TabPFN v2, Conventional compression algorithms (Matt Mahoney's benchmarks), TensorDyne GPU chip for neural operations
Concerns raised: Model only works for specific file; retraining required for each new file, How to securely share the model beforehand for decryption without RSA, Very slow decompression (45 minutes for 100MB), Accuracy concerns with overfitted transformer adding random bytes, Heavy computational load running transformer in user's browser
Feature requests: Support for multiple files without retraining, Faster decompression speed, Better documentation of prior work and related approaches
Competitors
Other products that read as similar to this one : 317 launches clear the similarity bar, closest 8 shown.
Attention rank: #12 of 318 (itself plus its competitors, highest first, normalized so YC and Product Hunt are compared fairly).
Launched 234 days after the earliest competitor.
- ExANS · hn · 2026-08-05 · 15 upvotes · similarity 0.53
- Smaller than WinRaR but 4x faster · ph · 2026-09-18 · 1 upvotes · similarity 0.50
- Compressing a Histogram from 16GB to 2KB · hn · 2026-01-15 · 5 upvotes · similarity 0.49
- Compress to 100KB · ph · 2026-09-09 · 1 upvotes · similarity 0.49
- EasyPDF Exact Size Compressor · ph · 2026-11-14 · 1 upvotes · similarity 0.48
- halfsize · ph · 2026-11-10 · 1 upvotes · similarity 0.48
- Unboundcove Video Tools · ph · 2026-11-11 · 1 upvotes · similarity 0.47
- I compressed videos by upto 78x while preserving faces,text and details · hn · 2026-08-28 · 5 upvotes · similarity 0.47
The niche, labels, vibe code rating and competitor comparison on this page are analysis by Nicheloom. If you use them, please credit nicheloom.com. How you may use our data.