Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m
Details
- External ID
- 47378781
- Source
- HN
- Company
- —
- Product
- Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m
- Website domain
- huggingface.co
- Launched
- March 14, 2026
- Cohort
- —
- Upvotes
- 408
- Upvotes percentile
- 0.990159901599016
- Tags
- —
- Fetched at
- Sept. 7, 2026, 9:26 p.m.
- Updated at
- Sept. 7, 2026, 9:26 p.m.
Enrichment
- Theme
- Hacker News clients, datasets, and tools
- Vertical
- Horizontal
- Function
- Data infrastructure
- Audience
- Developer
- AI stance
- Not AI
- Project type
- Hobby / open-source project
- Normalized one-liner
- hacker news dataset in parquet format
- Manually corrected
- False
Could you build this?
Yes This is a simple scheduled script (cron/GitHub Actions) polling the public Hacker News Firebase API and writing batches to Parquet files uploaded to Hugging Face.
Discussion
20 comments analyzed.
Competitors mentioned: DuckDB with SQLite extension, HuggingFace hub, Microsoft.AI.Extensions (.NET), Alternative HN frontends
Concerns raised: AI slop flooding makes pre-2020 content preservation harder, Legal permissions unclear for using HN data, Ingestion time too slow (week-long process), Rate limiting and API access constraints
Feature requests: DuckDB to SQLite conversion (reduce ingestion to 5 minutes), Batch abstraction for chat/embedding interfaces, Near real-time data refresh capability, C# / .NET equivalent implementation
Competitors
Other products that read as similar to this one — 146 launches clear the similarity bar, closest 8 shown.
Attention rank: #5 of 147 (itself plus its competitors, highest first — normalized so YC and Product Hunt are compared fairly).
Launched 127 days after the earliest competitor.
- Hackerbrief · hn · 2026-03-16 · 82 upvotes · similarity 0.59
- Hacker News RSS Feed Directory · hn · 2026-04-04 · 8 upvotes · similarity 0.55
- Hacker News, but every headline is hysterical clickbait · hn · 2025-12-19 · 188 upvotes · similarity 0.50
- Hacker News with reduced priority for AI driven content · hn · 2026-08-06 · 6 upvotes · similarity 0.50
- Hacker News with reduced priority for AI driven content · hn · 2026-08-05 · 24 upvotes · similarity 0.50
- The Front Page · hn · 2026-05-24 · 29 upvotes · similarity 0.49
- Hacker News, summarized and printed to your local printer · hn · 2026-04-02 · 5 upvotes · similarity 0.49
- The Hacker News Puzzle · hn · 2026-08-18 · 5 upvotes · similarity 0.48
Other launches for this product
- No other launches for this product.
Same idea, different domain
Nobody's really built a data infrastructure tool for Media & entertainment yet.