DuckDB for Kafka Stream Processing
Details
- External ID
- 46195007
- Source
- HN
- Company
- —
- Product
- DuckDB for Kafka Stream Processing
- Website domain
- sql-flow.com
- Launched
- Dec. 8, 2025
- Cohort
- —
- Upvotes
- 77
- Upvotes percentile
- 0.8625954198473282
- Tags
- —
- Fetched at
- Sept. 7, 2026, 9:25 p.m.
- Updated at
- Sept. 7, 2026, 9:25 p.m.
Description
Hello Everyone! We built SQLFlow as a lightweight stream processing engine.We leverage DuckDB as the stream processing engine, which gives SQLFlow the ability to process 10's of thousands of messages a second using ~250MiB of memory!DuckDB also supports a rich ecosystem of sinks and connectors!https://sql-flow.com/docs/category/tutorials/https://github.com/turbolytics/sql-flowWe were tired of running JVM's for simple stream processing, and also of bespoke one off stream processorsI would love your feedback, criticisms and/or experiences!Thank you
Enrichment
- Theme
- database infrastructure and developer tools
- Vertical
- Horizontal
- Function
- Data infrastructure
- Audience
- Developer
- AI stance
- Not AI
- Project type
- Hobby / open-source project
- Normalized one-liner
- stream processing with duckdb and kafka
- Manually corrected
- False
Could you build this?
No Building a stream processing engine integrating Kafka with embedded DuckDB at 10,000+ msgs/sec in a tiny memory footprint requires deep low-level systems and streaming engine internals expertise.
What it would actually take: The engine is likely built in Rust or Go, embedding DuckDB via C/FFI bindings, with custom streaming buffer management, Kafka consumer group rebalancing, tumbling/sliding window aggregation logic, and exactly-once/at-least-once sink delivery. The difficult parts are zero-copy buffer handoffs, backpressure handling, stateful stream crash-recovery, and high-throughput low-latency memory management.
Discussion
13 comments analyzed.
Competitors mentioned: Tributary, Flink, Snowflake, BigQuery, SQLite
Concerns raised: Unclear production-readiness and operational deployment story, Batch processing vs. true streaming semantics unclear, Lacks DevOps, testing, configuration-as-code tooling, Stream-to-stream joins not supported, Feels hacky for continuous streaming use cases
Feature requests: Stream-to-stream joins support, Apache Pulsar support (in addition to Kafka), Avro and Protobuf schema support, Kafka transactions for writes, Materialized views for streaming aggregates
Competitors
Other products that read as similar to this one — 108 launches clear the similarity bar, closest 8 shown.
Attention rank: #17 of 109 (itself plus its competitors, highest first — normalized so YC and Product Hunt are compared fairly).
Launched 40 days after the earliest competitor.
- Stratum · hn · 2026-03-12 · 12 upvotes · similarity 0.50
- StreamHouse · hn · 2026-02-25 · 10 upvotes · similarity 0.48
- Dbxlite · hn · 2025-12-12 · 5 upvotes · similarity 0.47
- duckdb-rust-workers · github · 2026-09-29 · 11 upvotes · similarity 0.45
- BlazeMQ · hn · 2026-02-10 · 5 upvotes · similarity 0.45
- Connect DuckDB to any database that has an ADBC driver · hn · 2026-07-08 · 7 upvotes · similarity 0.45
- duckdb-wasm-sample · github · 2026-09-12 · 9 upvotes · similarity 0.44
- UnisonDB · hn · 2025-11-01 · 17 upvotes · similarity 0.43
Other launches for this product
- No other launches for this product.
Same idea, different domain
Nobody's really built a data infrastructure tool for Media & entertainment yet.