Nicheloom visits: 58,031 all-time · 7,694 in the last 7 days

The opportunity tracker for new startups.

Today: 1,953 launches · Niche of the day: specialized calculation tools · See what launched today →

DuckDB for Kafka Stream Processing

Get picks like this daily. Today's launch numbers, the top launches, AI and tech news, and a daily opportunity pick, straight to your inbox.

Want to reach these readers? Sponsor an issue.

Hello Everyone! We built SQLFlow as a lightweight stream processing engine.We leverage DuckDB as the stream processing engine, which gives SQLFlow the ability to process 10's of thousands of messages a second using ~250MiB of memory!DuckDB also supports a rich ecosystem of sinks and connectors!https://sql-flow.com/docs/category/tutorials/https://github.com/turbolytics/sql-flowWe were tired of running JVM's for simple stream processing, and also of bespoke one off stream processorsI would love your feedback, criticisms and/or experiences!Thank you

HN · launched Dec 8, 2025 · 77 upvotes (top 14%) · 0 views

Niche: event streaming platforms · 38 launches, 2 in the last 7 days · Explore this niche →

Category and tags
  • Horizontal
  • Data infrastructure
  • Developer
  • Not AI
  • Hobby / open-source project

Visit website ↗ Source listing ↗

More like this: all 38 launches in event streaming platforms →

Could you build this?

No Building a stream processing engine integrating Kafka with embedded DuckDB at 10,000+ msgs/sec in a tiny memory footprint requires deep low-level systems and streaming engine internals expertise.

What it would actually take: The engine is likely built in Rust or Go, embedding DuckDB via C/FFI bindings, with custom streaming buffer management, Kafka consumer group rebalancing, tumbling/sliding window aggregation logic, and exactly-once/at-least-once sink delivery. The difficult parts are zero-copy buffer handoffs, backpressure handling, stateful stream crash-recovery, and high-throughput low-latency memory management.

Discussion

13 comments analyzed.

Competitors mentioned: Tributary, Flink, Snowflake, BigQuery, SQLite

Concerns raised: Unclear production-readiness and operational deployment story, Batch processing vs. true streaming semantics unclear, Lacks DevOps, testing, configuration-as-code tooling, Stream-to-stream joins not supported, Feels hacky for continuous streaming use cases

Feature requests: Stream-to-stream joins support, Apache Pulsar support (in addition to Kafka), Avro and Protobuf schema support, Kafka transactions for writes, Materialized views for streaming aggregates

Competitors

Other products that read as similar to this one : 199 launches clear the similarity bar, closest 8 shown.

Attention rank: #30 of 200 (itself plus its competitors, highest first, normalized so YC and Product Hunt are compared fairly).

Launched 40 days after the earliest competitor.

  • Akema · ph · 2026-11-03 · 66 upvotes · similarity 0.53
  • Stratum · hn · 2026-03-12 · 12 upvotes · similarity 0.52
  • KDBS · ph · 2026-10-16 · 1 upvotes · similarity 0.49
  • StreamHouse · hn · 2026-02-25 · 10 upvotes · similarity 0.48
  • Dbxlite · hn · 2025-12-12 · 5 upvotes · similarity 0.48
  • kafuck · github · 2026-11-10 · 66 upvotes · similarity 0.47
  • duckdb-rust-workers · github · 2026-09-29 · 17 upvotes · similarity 0.47
  • BlazeMQ · hn · 2026-02-10 · 5 upvotes · similarity 0.46

Related data

The niche, labels, vibe code rating and competitor comparison on this page are analysis by Nicheloom. If you use them, please credit nicheloom.com. How you may use our data.