Nicheloom visits: 58,257 all-time · 7,637 in the last 7 days

The opportunity tracker for new startups.

Today: 1,008 launches · Niche of the day: specialized calculation tools · See what launched today →

Infini-News

1.36B news articles from Common Crawl, queryable in ms

Get picks like this daily. Today's launch numbers, the top launches, AI and tech news, and a daily opportunity pick, straight to your inbox.

Want to reach these readers? Sponsor an issue.

Infini-News is ten years of CC-NEWS (the news subset of Common Crawl), cleaned, enriched and turned into a full-text index so you can count any keyword or phrase across 1.36B articles in sub-second time (ok, now maybe a few seconds, but circumstantial), without downloading anything. It's free and open on Hugging Face. I did it because I was sick of having to manually scrape news websites and the like for research purposes and because it felt interesting personally to tackle a project of this scale. On top of data cleaning, we have run language, country (via TLDs and some other heuristics) and topic tagging over all the articles and I have indexed all of them using a recent new n-gram indexing technology that I consider akin to magic. I would encourage you to read the blogpost and play with the interactive viz I made for it. Also, of course, happy to answer questions. Blog: https://cs2.uni-graz.at/blog/infini-news/ Dataset: https://huggingface.co/datasets/ruggsea/infini-news-corpus Index: https://huggingface.co/datasets/ruggsea/infini-news-index Preprint: https://arxiv.org/abs/2605.18337

HN · launched Jul 1, 2026 · 7 upvotes (top 60%) · 0 views

Niche: vertical search engines and directories · 98 launches, 7 in the last 7 days ▼ -46% · Explore this niche →

Category and tags
  • Media & entertainment
  • Search & retrieval
  • Developer
  • Not AI
  • Commercial product

Visit website ↗ Source listing ↗

More like this: all 98 launches in vertical search engines and directories →

Could you build this?

No Indexing 1.36 billion articles from Common Crawl for millisecond-latency full-text querying involves petabyte-scale big data engineering, custom distributed inverted indexing, and substantial computing infrastructure that vibe coding cannot deliver.

What it would actually take: The architecture requires a distributed data processing pipeline (Apache Spark/Flink) over Common Crawl WARC archives, coupled with a distributed search engine index (such as heavily sharded Tantivy, Lucene, or ClickHouse with custom tokenization). The hard parts are massive scale ETL (clearing duplicates, boilerplate extraction, compression) and architecting memory-mapped indexes capable of sub-second aggregations across billions of records without astronomical cloud hosting bills. This requires veteran data infrastructure and search engineering expertise.

Discussion

1 comment analyzed.

Competitors

Other products that read as similar to this one : 85 launches clear the similarity bar, closest 8 shown.

Attention rank: #43 of 86 (itself plus its competitors, highest first, normalized so YC and Product Hunt are compared fairly).

Launched 242 days after the earliest competitor.

Related data

The niche, labels, vibe code rating and competitor comparison on this page are analysis by Nicheloom. If you use them, please credit nicheloom.com. How you may use our data.