Real-time system that tracks how news spreads across 200k websites
Get picks like this daily. Today's launch numbers, the top launches, AI and tech news, and a daily opportunity pick, straight to your inbox.
Want to reach these readers? Sponsor an issue.
I built a system that monitors ~200,000 news RSS feeds in near real-time and clusters related articles to show how stories spread across the web.It uses Snowflake’s Arctic model for embeddings and HNSW for fast similarity search. Each “story cluster” shows who published first, how fast it propagated, and how the narrative evolved as more outlets picked it up.Would love feedback on the architecture, scaling approach, and any ways to make the clusters more accurate or useful.Live demo: https://yandori.io/news-flow/
HN · launched Nov 26, 2025 · 256 upvotes (top 3%) · 1 view
Niche: news aggregation platforms · 244 launches, 12 in the last 7 days ▼ -54% · Explore this niche →
Category and tags
- Media & entertainment
- Analytics & BI
- B2B
- Not AI
- Commercial product
Visit website ↗ Source listing ↗
More like this: all 244 launches in news aggregation platforms →
Could you build this?
No Ingesting, parsing, embedding, and clustering 200,000 live RSS feeds in near real-time requires substantial distributed data streaming infrastructure and high-throughput vector pipeline management.
What it would actually take: A production system requires a distributed crawler (e.g., Go or Rust with worker pools) capable of polling hundreds of thousands of feeds while respecting rate limits, coupled with Kafka or Redpanda for message streaming. The embedding and clustering pipeline needs batched GPU inference (Arctic embeddings), incremental HNSW graph indexing in Milvus or Qdrant, and real-time graph traversal to calculate story propagation velocity. Operating this requires senior data engineering and distributed systems reliability expertise.
Discussion
20 comments analyzed.
Competitors mentioned: Associated Press news aggregation, Private versions at large organizations, websitelaunches (creator's own project)
Concerns raised: AI-generated content detection issues, Timezone handling complexity with dynamic data, Inaccurate published dates from sources, High cost of Twitter/social media API access, Spam and non-news content (e.g., Black Friday deals) getting through
Feature requests: Network graphs or timeline visualizations of news propagation, Establish equivalency between duplicate articles instead of filtering, Hot-linked titles and copy-paste functionality, Social media data integration, Multi-language support
Competitors
Other products that read as similar to this one : 154 launches clear the similarity bar, closest 8 shown.
Attention rank: #7 of 155 (itself plus its competitors, highest first, normalized so YC and Product Hunt are compared fairly).
Launched 25 days after the earliest competitor.
- News Pulse · hn · 2026-02-28 · 9 upvotes · similarity 0.48
- Uncited · hn · 2025-11-19 · 12 upvotes · similarity 0.45
- European tech news in 6 languages · hn · 2025-11-14 · 46 upvotes · similarity 0.45
- Feed.news · hn · 2026-03-26 · 5 upvotes · similarity 0.44
- AI Heat Map Sourced from over 140 News Feeds · hn · 2026-10-25 · 5 upvotes · similarity 0.44
- 20 years of Hacker News discussions, clustered and visualized · hn · 2026-03-22 · 7 upvotes · similarity 0.43
- newsnest · github · 2026-10-31 · 19 upvotes · similarity 0.43
- WeSearch · hn · 2026-05-01 · 16 upvotes · similarity 0.43
The niche, labels, vibe code rating and competitor comparison on this page are analysis by Nicheloom. If you use them, please credit nicheloom.com. How you may use our data.