Markdown is 40× more token-efficient with SQLite

Hi HN, I'm Richard, founder of Native. I'm working on open-source context ownership, to stop AI centralisation: https://github.com/withnative/nativeMy first focus has been making Markdown vaults powerful enough so that teams can self-host all of their context (messaging, docs and projects), without it being externally fragmented or rate-limited.I'm sharing some early results: Markdown vaults can be 40× more efficient with SQLite!https://www.withnative.ai/research-and-posts/markdown-contex...Here's some of the main ideas:- SQLite or Markdown folders, you pick. You can project the same authoritative event log into either form; more projection formats will follow (e.g. maybe neo4j; we do use a graph structure).- Empirically: SQLite is better for agents! On a 120 trial evaluation for 11,000 records, agents use 40× fewer tokens with SQLite vs Markdown, per correct answer.- Everything is a file or SQLite row. Messages, documents and projects: all can be modelled in your vault. You get local-first collaboration with teammates, synced to a cloud of your choice.- You can render your vault with arbitrary UI. Your WhatsApp-like UI speaks to your teammate's Slack-like UI; both are just views over the same database.A few things I've learned building this:- Agents do pretty well even without SQL if you give them strongly typed Markdown frontmatter. This was one arm of the trial I ran. I didn't have any clear expectations, but it was cool to watch: agents ended up writing Python scripts to do SQL-like queries.- Cumulatively, cached input tokens can get expensive. I feel silly in hindsight here. I'd gotten used to letting my sessions run long, "It's all cached so it's very cheap". But when I looked at my results here, over 2/3 of the token bill on long sessions was cached input...- "Markdown vaults don't scale as well as databases" isn't as self-evident as I think. This is why I decided to benchmark it... I've had some very blank stares when I try to explain how Markdown files might break at true agent-scale volume.Unfortunately, the dataset is private, because it's two months of my own work. I am considering open-sourcing it even so, but there is definitely some embarrassing stuff in there. TBD... I know that independent result verification matters!The core engine that I evaluated is open-source, with more to be released soon: https://github.com/withnative/nativeIf you're interested in trying connected context on an open engine, you can get updates from our website or email me directly: [email protected]://www.withnative.ai/