Nicheloom visits: 58,280 all-time · 7,637 in the last 7 days

The opportunity tracker for new startups.

Today: 1,008 launches · Niche of the day: specialized calculation tools · See what launched today →

Open database of link metadata for large-scale analysis

Get picks like this daily. Today's launch numbers, the top launches, AI and tech news, and a daily opportunity pick, straight to your inbox.

Want to reach these readers? Sponsor an issue.

I would like to share an open database focused on link-level metadata extraction and aggregation, which may be of interest to researchers.The project maintains a structured dataset of links enriched with metadata such as:- page title- description / summary- publication date (when available)- thumbnail / preview image- etc.The goal is to provide a reusable, inspectable set of link metadata that can be used for experiments in areas such as:- RSS and feed analysis- news analysis- link rot analysis?The database is publicly available here:https://github.com/rumca-js/RSS-Link-Database-2025There are also databases for previous years

HN · launched Jan 3, 2026 · 15 upvotes (top 39%) · 0 views

Niche: data enrichment and lookup apis · 83 launches, 5 in the last 7 days ▼ -17% · Explore this niche →

Category and tags
  • Horizontal
  • Data infrastructure
  • Developer
  • Not AI
  • Hobby / open-source project

Visit website ↗ Source listing ↗

More like this: all 83 launches in data enrichment and lookup apis →

Could you build this?

Partial Extracting metadata and exposing a database or API is straightforward, but maintaining an open, large-scale, continuously updated crawl dataset requires heavy data pipeline infrastructure and anti-scraping management.

What it would actually take: The architecture needs a distributed web crawler (e.g., Scrapy, Playwright clusters, or Go-based workers) ingesting millions of URLs, extracting OpenGraph/schema metadata, and storing petabyte-scale outputs in ClickHouse or Parquet on object storage. The difficult parts are rate-limiting, IP proxy rotation, dynamic JavaScript rendering at scale, and high-throughput data deduplication.

Discussion

1 comment analyzed.

Concerns raised: Feed structure changes (fields added/removed, summaries truncated), Data normalization strategy unclear, Longitudinal dataset consistency

Feature requests: Handle RSS source schema evolution, Store raw payload alongside normalized data

Competitors

Other products that read as similar to this one : 119 launches clear the similarity bar, closest 8 shown.

Attention rank: #49 of 120 (itself plus its competitors, highest first, normalized so YC and Product Hunt are compared fairly).

Launched 55 days after the earliest competitor.

  • PhantomCollect · hn · 2025-11-11 · 7 upvotes · similarity 0.44
  • HelixDB · hn · 2026-06-10 · 159 upvotes · similarity 0.44
  • SEO Meta Extractor · ph · 2026-09-29 · 2 upvotes · similarity 0.42
  • Free URL SEO Metadata Checker · ph · 2026-10-26 · 1 upvotes · similarity 0.42
  • database · github · 2026-09-17 · 24 upvotes · similarity 0.40
  • OpenFable · hn · 2026-04-08 · 5 upvotes · similarity 0.40
  • Image URL Dataset · ph · 2026-09-11 · 1 upvotes · similarity 0.39
  • YaraDB · hn · 2025-11-12 · 10 upvotes · similarity 0.39

Related data

The niche, labels, vibe code rating and competitor comparison on this page are analysis by Nicheloom. If you use them, please credit nicheloom.com. How you may use our data.