Unified multimodal memory framework, without embeddings
Get picks like this daily. Today's launch numbers, the top launches, AI and tech news, and a daily opportunity pick, straight to your inbox.
Want to reach these readers? Sponsor an issue.
Hi HN,We’ve been building memU(https://github.com/NevaMind-AI/memU), an open-source, general-purpose memory framework for AI agents. It supports dual-mode retrieval: classic RAG and LLM-based direct file reading.Most multimodal memory systems either embed everything into vectors or treat non-text data as attachments. These work, but at scale it becomes hard to explain why certain context was retrieved and what evidence it relies on.memU takes a different approach: since models reason in language, multimodal memory should converge into structured, queryable text, while remaining fully traceable to original data.---## Three-Layer Architecture- Resource Layer Stores raw multimodal data as ground truth. All higher-level memory remains traceable to this layer.- Memory Item Layer Extracts atomic facts from raw data and stores them as natural-language statements. Embeddings are optional and used only for acceleration.- Memory Category Layer Aggregates items into readable, theme-based memory files (e.g. user preferences, work logs). Frequently accessed topics stay active; low-usage content is demoted to balance speed and coverage.---## Memorization Bottom-up and asynchronous. Data flows from resources → items → category files without manual schemas. When capacity is reached, recently relevant memories replace the least used ones.## Retrieval Top-down. memU searches category files first, then items, and only falls back to raw data if needed. At the item layer, it combines BM25 + embeddings to balance exact matching and semantic recall, avoiding embedding-only imprecision.Dual-mode retrieval lets applications choose between: - low-latency embedding search, or - LLM-based direct reading of memory files.## Evolution Memory structure adapts automatically based on real usage: - Frequently accessed memories remain at the Category layer - Memories retrieved from raw data are promoted upward and linked - Organization evolves from usage patterns, not predefined rulesGoal: keep relevant memories retrievable at the Category layer and minimize latency over time.---## A Unified Multimodal Memory Pipeline memU is a text-centered multimodal memory system. Multimodal inputs are progressively converted into interpretable text memory, while staying traceable to original data. This provides stable, high-level context for reasoning, with detailed evidence available when needed—inside a memory structure that evolves through real-world use.
HN · launched Jan 7, 2026 · 7 upvotes (top 65%) · 0 views
Niche: ai agent memory infrastructure · 351 launches, 8 in the last 7 days ▼ -70% · Explore this niche →
Category and tags
- Horizontal
- Data infrastructure
- Developer
- AI-native
- Hobby / open-source project
Visit website ↗ Source listing ↗
More like this: all 351 launches in ai agent memory infrastructure →
Could you build this?
Partial Basic multimodal orchestration can be built with LLMs, but creating an efficient multimodal memory framework without embeddings requires designing novel document indexing and fast context-filtering algorithms.
What it would actually take: The system requires a hybrid architecture integrating fast OCR/vision pipelines, structured hierarchical file-system indexing, and custom prompt-driven or rule-based semantic filtering to avoid embedding costs without blowing context budgets. Hard parts include managing latency when scanning raw multimodal files and ensuring high recall across diverse formats (images, audio, code) without traditional vector search.
Discussion
2 comments analyzed.
Competitors mentioned: Vanilla vector search / standard RAG, Embedding-based memory stacks
Concerns raised: Retrieval accuracy improvement over standard RAG not quantified, Unclear performance 'lift' for migration decisions, Scalability of human-readable memory format for long-running agents
Feature requests: Benchmarks comparing retrieval accuracy to vanilla vector search, Temporal reasoning and traceability capabilities, Audit/inspection tooling for agent memory state
Competitors
Other products that read as similar to this one : 277 launches clear the similarity bar, closest 8 shown.
Attention rank: #166 of 278 (itself plus its competitors, highest first, normalized so YC and Product Hunt are compared fairly).
Launched 68 days after the earliest competitor.
- A file-based agent memory framework that works like skill · hn · 2026-01-06 · 11 upvotes · similarity 0.76
- multimodal-image-retrieval · github · 2026-11-02 · 100 upvotes · similarity 0.55
- Irpapers · hn · 2026-02-23 · 5 upvotes · similarity 0.52
- PMB · hn · 2026-06-22 · 7 upvotes · similarity 0.50
- Honcho · hn · 2026-01-27 · 8 upvotes · similarity 0.49
- LinearMemory: AI Memory Explorer · ph · 2026-09-06 · 2 upvotes · similarity 0.47
- YourMemory, agentic memory is a pruning problem, not a hoarding problem · hn · 2026-06-07 · 19 upvotes · similarity 0.47
- Breathe-Memory · hn · 2026-03-26 · 6 upvotes · similarity 0.46
The niche, labels, vibe code rating and competitor comparison on this page are analysis by Nicheloom. If you use them, please credit nicheloom.com. How you may use our data.