memU

The memory harness for proactive AI agents — structured storage, intent capture, 10x token reduction.

13,944

1,036

memU

Turn Raw Multimodal Data into Agent-Ready Structured Memory

English | 中文 | 日本語 | 한국어 | Español | Français

</div>

memU is a data-to-memory engine for AI agents.
It turns raw conversations, documents, images, audio, video, tool logs, and local files into structured, scoped memory that agents can retrieve and use directly.

Raw in: chats, docs, URLs, images, audio/video, logs, and local files
Structured out: resources, typed memory items, categories, relations, summaries, and embeddings
Agent-ready: ingest once, then load ranked context in one call with user/session/task scope

🔄 How It Works

Raw Multimodal Data → Structured Memory Store → Agent Context

Raw Input                    memorize() Pipeline           Stored Memory
─────────────────────        ─────────────────────         ─────────────────────
chat logs                →   parse + extract           →   profile / event / behavior items
documents / URLs         →   ingest + extract facts    →   knowledge / skill / tool items
images / video           →   caption + describe        →   resources + memory item summaries
audio                    →   transcribe + extract      →   resources + event / knowledge items
tool logs                →   mine usage patterns       →   tool / skill memory items
local files              →   summarize + categorize    →   resources, categories, relations

Query Input                  retrieve() Pipeline           Agent Context
─────────────────────        ─────────────────────         ─────────────────────
user / task query        →   route + rank by scope     →   relevant categories, items, resources

Ingest — store each source as a Resource with its modality and source location
Preprocess — parse text, caption images/video, transcribe audio, and normalize inputs
Extract — turn raw content into typed MemoryItems such as profile, event, knowledge, behavior, skill, or tool memories
Organize — categorize, cross-link, embed, and summarize memories into a browsable structure
Persist — write records, relations, embeddings, and category summaries through the configured backend
Retrieve — return only the relevant context for the current user, agent, session, or task

🗂️ Structured Memory Graph

memU's primary output is a structured memory graph, persisted through repository contracts and returned as dictionaries from memorize() and retrieve().

Resource
├── url, modality, local_path, caption, embedding
└── MemoryItem[]
    ├── memory_type: profile | event | knowledge | behavior | skill | tool
    ├── summary, extra, happened_at, embedding
    └── CategoryItem edges
        └── MemoryCategory
            ├── name, description, summary
            └── embedding

Record	Role	Used By
`Resource`	Preserve the original source artifact and derived caption/text	Trace context back to the source
`MemoryItem`	Store typed atomic memories with summaries and optional metadata	Inject precise facts, preferences, events, skills, and tool patterns
`MemoryCategory`	Maintain topic-level summaries over related items	Load compact context for broad queries
`CategoryItem`	Link items to categories	Navigate related memories without reprocessing the source

This gives agents a stable memory surface: they can ingest raw sources once, then request scoped and ranked context instead of rereading every source artifact.

🧩 What memU Builds

The memory graph is stored as structured records:

Layer	What It Represents	Why Agents Use It
Resource	Original source artifact: conversation, document, image, video, audio, URL, or file	Trace memory back to its source
MemoryItem	Atomic structured memory with a type and summary	Inject precise facts, preferences, events, skills, and tool patterns
MemoryCategory	Auto-generated topic category with an evolving summary	Load high-level context before drilling into details
CategoryItem	Relationship between items and categories	Navigate related memories without reprocessing the source
Embedding	Vector representation for resources, items, and categories	Retrieve relevant context with low latency

Example memorize() output:

{
  "resource": {
    "id": "res_01",
    "url": "files/launch-meeting.mp4",
    "modality": "video",
    "caption": "A product planning discussion about onboarding and launch risks."
  },
  "items": [
    {
      "id": "mem_01",
      "memory_type": "event",
      "summary": "The team decided to simplify onboarding before the next launch review."
    },
    {
      "id": "mem_02",
      "memory_type": "profile",
      "summary": "The user prefers concise implementation plans with explicit verification steps."
    },
    {
      "id": "mem_03",
      "memory_type": "tool",
      "summary": "Use repository-wide search before editing configuration files to avoid missing duplicated settings."
    }
  ],
  "categories": [
    {
      "id": "cat_01",
      "name": "product_goals",
      "summary": "Current launch priorities, onboarding decisions, and unresolved risks."
    }
  ],
  "relations": [
    { "item_id": "mem_01", "category_id": "cat_01" }
  ]
}

Then an agent can call retrieve() to get a scoped, ranked context payload:

context = await service.retrieve(
    queries=[{"role": "user", "content": {"text": "What context matters for this launch task?"}}],
    where={"user_id": "123"},
)

⭐️ Star the repository

If you find memU useful or interesting, a GitHub Star ⭐️ would be greatly appreciated.

✨ Core Features

Capability	Description
🗂️ Multimodal Ingestion	Ingest conversations, documents, images, video, audio, URLs, logs, and local files
📁 Structured Memory Graph	Persist resources, memory items, categories, relations, summaries, and embeddings
🧠 Typed Memory Extraction	Extract profile, event, knowledge, behavior, skill, and tool memories from raw sources
🧭 Automatic Organization	Build categories, relations, summaries, and embeddings without manual tagging
🤖 Agent-Ready Retrieval	Return scoped, ranked context that can be injected into any agent workflow
🧱 Pluggable Storage	Use in-memory, SQLite, or Postgres backends with the same repository contracts
🔀 Profile-Based LLM Routing	Route chat, embedding, vision, and transcription work through configurable LLM profiles

🎯 Use Cases

1. Conversation Memory

Turn chat logs into user preferences, goals, events, and relationship context.

await service.memorize(
    resource_url="examples/resources/conversations/conv1.json",
    modality="conversation",
    user={"user_id": "123"},
)

context = await service.retrieve(
    queries=[{"role": "user", "content": {"text": "What should I remember about this user?"}}],
    where={"user_id": "123"},
)

2. Workspace Context for Coding Agents

Convert docs, PR notes, logs, and design decisions into reusable project memory.

await service.memorize(resource_url="docs/architecture.md", modality="document")
await service.memorize(resource_url="examples/resources/logs/log1.txt", modality="document")

context = await service.retrieve(
    queries=[{"role": "user", "content": {"text": "How should I structure this module?"}}],
)

3. Multimodal Knowledge Layer

Extract searchable facts from documents, screenshots, images, videos, and audio notes.

await service.memorize(resource_url="examples/resources/docs/doc1.txt", modality="document")
await service.memorize(resource_url="examples/resources/images/image1.png", modality="image")
# Audio is supported for your own .mp3/.wav/.m4a files.
await service.memorize(resource_url="meeting-audio.mp3", modality="audio")

context = await service.retrieve(
    queries=[{"role": "user", "content": {"text": "What matters for the next research plan?"}}],
)

4. Tool and Agent Learning

Turn execution traces into tool memories that tell future agents when to use a tool and what mistakes to avoid.

await service.memorize(resource_url="examples/resources/logs/log1.txt", modality="document")

context = await service.retrieve(
    queries=[{"role": "user", "content": {"text": "Which tools worked for config editing?"}}],
)

🗂️ Architecture

memU's memory model is hierarchical enough for browsing and structured enough for direct retrieval:

Layer	Primary Role	Retrieval Role
Resource	Preserve source artifacts and captions	Recall original context when item/category summaries are not enough
Item	Store typed atomic memories	Load precise facts, events, preferences, skills, and tool patterns
Category	Maintain topic-level summaries	Assemble compact context for broad queries

See docs/architecture.md for the runtime view of MemoryService, workflow pipelines, storage backends, and LLM routing.

🚀 Quick Start

Option 1: Cloud Version

👉 memu.so — Hosted API for managed ingestion, structured memory, and retrieval

For enterprise deployment: info@nevamind.ai

Cloud API (v3)

Base URL	`https://api.memu.so`
Auth	`Authorization: Bearer <token>`

Method	Endpoint	Description
`POST`	`/api/v3/memory/memorize`	Ingest raw data and build structured memory
`GET`	`/api/v3/memory/memorize/status/{task_id}`	Check processing status
`POST`	`/api/v3/memory/categories`	List auto-generated categories
`POST`	`/api/v3/memory/retrieve`	Query memory for agent context

📚 Full API Documentation

Option 2: Self-Hosted

Installation

From a clone of this repository:

uv sync
# or, for the full development setup:
make install

To install the published package instead:

pip install memu-py

Requirements: Python 3.13+. The default examples use OpenAI, so set OPENAI_API_KEY or pass another provider through llm_profiles.

Run an in-memory smoke script:

export OPENAI_API_KEY=your_key
cd tests
uv run python test_inmemory.py

Run with PostgreSQL + pgvector:

uv sync --extra postgres
docker run -d --name memu-postgres \
  -e POSTGRES_USER=postgres \
  -e POSTGRES_PASSWORD=postgres \
  -e POSTGRES_DB=memu \
  -p 5432:5432 \
  pgvector/pgvector:pg16

export OPENAI_API_KEY=your_key
export POSTGRES_DSN=postgresql+psycopg://postgres:postgres@127.0.0.1:5432/memu
cd tests
uv run python test_postgres.py

Custom LLM and Embedding Providers

from memu import MemUService

service = MemUService(
    llm_profiles={
        "default": {
            "base_url": "https://dashscope.aliyuncs.com/compatible-mode/v1",
            "api_key": "your_key",
            "chat_model": "qwen3-max",
            "client_backend": "sdk"
        },
        "embedding": {
            "base_url": "https://api.voyageai.com/v1",
            "api_key": "your_key",
            "embed_model": "voyage-3.5-lite"
        }
    },
)

OpenRouter Integration

from memu import MemoryService

service = MemoryService(
    llm_profiles={
        "default": {
            "provider": "openrouter",
            "client_backend": "httpx",
            "base_url": "https://openrouter.ai",
            "api_key": "your_key",
            "chat_model": "anthropic/claude-3.5-sonnet",
            "embed_model": "openai/text-embedding-3-small",
        },
    },
    database_config={"metadata_store": {"provider": "inmemory"}},
)

📖 Core APIs

`memorize()` — Structure Raw Data

result = await service.memorize(
    resource_url="path/to/file.json",    # local file path or HTTP URL
    modality="conversation",            # conversation | document | image | video | audio
    user={"user_id": "123"},            # optional: scope to a user or agent
)
# Returns after processing completes:
# { "resource": {...}, "items": [...], "categories": [...], "relations": [...] }

Converts raw input into typed memory items
Categorizes and embeds items without manual tagging
Preserves source resources and item-category relations

`retrieve()` — Load Agent Context

# The retrieval strategy is set once on the service via retrieve_config:
#   MemoryService(retrieve_config={"method": "rag"})   # vector-first recall
#   MemoryService(retrieve_config={"method": "llm"})   # LLM-ranked recall
result = await service.retrieve(
    queries=[{"role": "user", "content": {"text": "What are their preferences?"}}],
    where={"user_id": "123"},   # scope filter
)
# Returns:
# {
#   "needs_retrieval": true,
#   "original_query": "...",
#   "rewritten_query": "...",
#   "next_step_query": "...",
#   "categories": [...],
#   "items": [...],
#   "resources": [...]
# }

`retrieve_config.method`	Behavior	Cost	Best For
`rag`	Vector-first category/item/resource recall, with optional LLM routing and sufficiency checks enabled by default	Embeddings plus LLM calls unless `route_intention` and `sufficiency_check` are disabled	Fast scoped recall with controllable reasoning
`llm`	LLM-ranked category/item/resource recall	LLM ranking at each tier	Deeper semantic ranking

💡 Example Workflows

Always-Learning Assistant

export OPENAI_API_KEY=your_key
uv run python examples/example_1_conversation_memory.py

Automatically extracts preferences, builds relationship models, and surfaces relevant context in future conversations.

Self-Improving Agent

uv run python examples/example_2_skill_extraction.py

Monitors agent actions, identifies patterns in successes and failures, auto-generates skill guides from experience.

Multimodal Context Builder

uv run python examples/example_3_multimodal_memory.py

Cross-references text, images, and documents automatically into a unified memory layer.

📊 Performance

memU achieves 92.09% average accuracy on the Locomo benchmark across all reasoning tasks.

View detailed results: memU-experiment

🧩 Ecosystem

Repository	Description
memU	Core data-to-memory engine — ingestion, extraction, retrieval
memU-server	Backend with real-time sync and webhook triggers
memU-ui	Visual dashboard for browsing and monitoring memory

Quick Links:

🤝 Partners

</div>

🤝 Contributing

# Fork and clone
git clone https://github.com/YOUR_USERNAME/memU.git
cd memU

# Install dev dependencies
make install

# Run quality checks before submitting
make check

See CONTRIBUTING.md for full guidelines.

Prerequisites: Python 3.13+, uv, Git

📄 License

Apache License 2.0

🌍 Community

GitHub Issues: Report bugs & request features
Discord: Join the community
X (Twitter): Follow @memU_ai
Contact: info@nevamind.ai

⭐ Star us on GitHub to get notified about new releases!

</div>

Repository

NevaMind-AI

NevaMind-AI/memU

Created

July 29, 2025

Updated

June 15, 2026

Language

Python

memU

memU

Turn Raw Multimodal Data into Agent-Ready Structured Memory

🔄 How It Works

🗂️ Structured Memory Graph

🧩 What memU Builds

⭐️ Star the repository

✨ Core Features

🎯 Use Cases

1. Conversation Memory

2. Workspace Context for Coding Agents

3. Multimodal Knowledge Layer

4. Tool and Agent Learning

🗂️ Architecture

🚀 Quick Start

Option 1: Cloud Version

Cloud API (v3)

Option 2: Self-Hosted

Installation

Custom LLM and Embedding Providers

OpenRouter Integration

📖 Core APIs

memorize() — Structure Raw Data

retrieve() — Load Agent Context

💡 Example Workflows

Always-Learning Assistant

Self-Improving Agent

Multimodal Context Builder

📊 Performance

🧩 Ecosystem

🤝 Partners

🤝 Contributing

📄 License

🌍 Community

Repository

Created

Updated

Language

Category

`memorize()` — Structure Raw Data

`retrieve()` — Load Agent Context