Nicheloom visits: 62,674 all-time · 8,415 in the last 7 days

The opportunity tracker for new startups.

Today: 1,441 launches · Niche of the day: digital marketing agency services · See what launched today →

Litelink

local-first, embedded stream capture into Iceberg tables

Get picks like this daily. Today's launch numbers, the top launches, AI and tech news, and a daily opportunity pick, straight to your inbox.

Want to reach these readers? Sponsor an issue.

Hi HN! I just wanted to share litelink a local-first, embedded capture library I built in python (code is heavily AI generated but designed and reviewed by yours truly). I've been using this for point-and-shoot WebSocket capture but I imagine it could also be useful for observability/metrics ingestion as well. Litelink supports a single writer per stream.I've been doing a lot of development and deployments on tiny VMs (2 vCPU, 8GB, 50-100GB disk) and didn't want the complexity or cost of managing central brokers (Kafka), databases (Postgres), and CDC/connectors just to get queryable WebSocket stream capture running.With litelink, you configure a log in code, and end-to-end setup takes <5 minutes (see the example scripts in the repo). The log is itself an Iceberg table (actually two: a local and archive table), so there's no second copy of your data to keep in sync or connector to manage.I'm sure there are still bugs, but I recently migrated all the capture feeds for a personal research project to litelink, and the experience has been night and day. Before that, I'd hand-rolled a capture system and was dealing with all the issues you'd expect (e.g. small file problem). I'll post some before/after stats in a comment below.I tried to channel the same ethos as LanceDB/Iceberg/SQLite. Everything runs local first without a network connection required. I've tried to abstract the complexity of stream/data lifecycle maintenance away behind a few public library methods. Hopefully someone else finds this useful! Let me know what you think.repo: https://github.com/nhobin219/litelinkspec: https://github.com/nhobin219/litelink/blob/main/docs/SPEC.mdpypi: `pip install litelink`

HN · launched Sep 3, 2026 · 9 upvotes (top 46%) · 0 views

Niche: event streaming platforms · 39 launches, 1 in the last 7 days · Explore this niche →

Category and tags

  • Horizontal
  • Data infrastructure
  • Developer
  • Not AI
  • Commercial product

Visit website ↗ Source listing ↗

Could you build this?

Partial A Python capture script is simple, but reliably streaming high-throughput events into Apache Iceberg tables locally without heavy JVM overhead requires low-level Parquet writing and metadata catalog management.

What it would actually take: Uses PyIceberg or PyArrow with DuckDB to serialize streaming WebSocket JSON frames directly into Apache Parquet format while handling Iceberg schema evolution, snapshot generation, and manifest list commits. The hard part is managing micro-batching and transactional Iceberg metadata ACID guarantees without running external heavy distributed services or memory leaks under continuous ingestion. Requires data engineering expertise in storage formats and tabular streaming specifications.

Discussion

2 comments analyzed.

Concerns raised: Litelink overhead dominates in small datasets due to auto-generated offset column, Storage efficiency worse than legacy system for certain data types, File count explosion compared to legacy approach

Feature requests: Optimize or eliminate litelink_offset column overhead for small rows, Compression improvements for low-compression datasets

Competitors

Other products that read as similar to this one : 110 launches clear the similarity bar, closest 8 shown.

Attention rank: #69 of 111 (itself plus its competitors, highest first, normalized so YC and Product Hunt are compared fairly).

Launched 306 days after the earliest competitor.

The niche, labels, vibe code rating and competitor comparison on this page are analysis by Nicheloom. If you use them, please credit nicheloom.com. How you may use our data.