Your chat app's memory is a 50-line bandage on RAG

Almost every chat app I have seen above prototype scale eventually grows the same memory layer. It looks like this. There is a messages table with user_id, conversation_id, role, content, and created_at. There is a vector store somewhere — Pinecone, Qdrant, pgvector, whatever was easy at the time. On write, the message text is shipped to an embedding model and pushed into the vector store with a foreign key back to the message row. On read, the user’s new message is embedded, the vector store returns the top-k most similar past messages, and those messages get inlined into the system prompt before generation. Somewhere in there is a “rolling summary” prompt that periodically asks the LLM to compress old turns into a paragraph that gets injected at the top.

This stack works. It is also a bandage on RAG, not memory. The seams will open in a sequence almost everyone hits, in almost the same order. This post is about which seams, and why a dedicated memory layer — Memorg, in our case — is built around closing them rather than papering over them.

Seam 1: there is no notion of what the memory belongs to

The vector store does not know what a session is. It knows there are embeddings and there are filters. If you want to scope a search to “this user, this conversation, this topic,” you have to encode all of that into metadata at write time and replay it as a filter at read time. The application becomes the schema, and any new dimension — a project, a workspace, a thread within a thread — means a schema migration and a backfill of every embedding ever written.

Memorg starts from the other end. Storage is hierarchical: session, conversation, topic, exchange. Each level is a real entity in SQLite with foreign keys, not a string in a metadata blob. Searches can scope to any level. When you add a project dimension, you add a column to the right table; you do not re-tag a million vectors.

Seam 2: vector similarity is not memory

The vector store returns the top-k nearest neighbours of the query. That is what it does, and it is the wrong answer to “what should I remember now.” A near-duplicate of the user’s question is often less useful than a definition from three turns ago, an action you took yesterday, or the fact that the user mentioned their stack two messages back. Pure similarity over a vector index ranks all of those the same — by cosine — and the most useful items are not the most similar.

Memorg blends three signals. Semantic similarity is one of them. Recency is another: a result that happened twenty seconds ago should beat one from a month ago, all else equal. Importance is the third: items can be marked or scored as important, and the scorer respects that. The blend is deterministic; you can read the scoring code and predict what will come back. You do not need an LLM-as-a-judge stage to get a useful ranking.

This deterministic stance is deliberate. Reranking with an LLM is a real technique and we are not against it, but the moment your read path makes an LLM call to decide what to read, you have added latency, cost, non-determinism, and a debugging headache to the simplest operation in your app. Memorg keeps that LLM call out of the read path. If you want it, you can layer it on top.

Seam 3: the rolling summary lies

The rolling summary is the part of the bandage most teams are least proud of. It works fine for a while, then a user mentions a fact in turn 14 that the summary at turn 40 has flattened into oblivion, and the model now confidently states the opposite. The summary is lossy by construction; the question is whether the loss happens at the right level of detail. With a single rolling summary prompt, it usually does not.

A hierarchical store lets the summary be a feature instead of a hack. Topic-level summaries can exist alongside the exchanges that produced them and be retrieved on demand. The exchanges are still there — searchable, scorable, recoverable — when the summary is wrong. Memorg does not currently auto-summarise (the data model supports it, the policy is on you), but the structure means a summary is a value-add to the memory rather than a replacement for it.

Seam 4: token budgets are an application concern

The vector store does not know how big your model’s window is. Neither does the rolling summary prompt. You end up reasoning about token budgets in the place that generates the next prompt, which is the place least equipped to do it well — by the time you are building the prompt, you have already decided what to fetch.

Memorg pushes the token budget into the memory call. The session has a configured budget (default model windows, override per-session for clients with smaller contexts). search_context() returns results trimmed to fit. You hand the result to the LLM with confidence that the context fits. If you change the budget, you do not change four prompt templates.

Seam 5: there is no MCP story

The bandage stack does not expose memory to other tools cleanly. If Cursor wants to read what the user has stored, you write a glue server. If Claude Desktop wants to write a fact, you write another. Each integration is a small project.

Memorg ships an MCP server (memorg-mcp, built on FastMCP). MCP-aware clients can talk to it directly. The memory layer becomes an integration point, not an integration tax. If your team is already exposing tools via MCP, memory belongs in that surface.

What the dedicated layer looks like

We did not set out to build something exotic. The shape is small. SQLite via aiosqlite for the relational store. USearch for the vector index, stored in the same file. OpenAI embeddings for vectors. A four-level hierarchy because real apps need at least one level above conversation and at least one below. A deterministic scorer because non-determinism in retrieval costs more than it earns. A token-budget aware return. An MCP server because that is how memory tools are talking to clients now.

That is most of Memorg. The library is MIT-licensed, alpha on PyPI, and runs in your process — there is no managed service to register for, and the storage is a single file you can copy with cp.

When you do not need this

If you are running single-turn completions with no continuity, you do not need a memory layer. If your “conversation” is two turns and a forty-token system prompt covers everything, the bandage is fine. The bandage stops being fine somewhere around the point where you notice a summary_prompt.py file in your repo with a TODO at the top.

What changes when you remove the bandage

Three things, mostly. One: you stop owning a schema you did not want to own. Memorg gives you a hierarchy and the scoring logic; you give it text. Two: the read path becomes deterministic and debuggable. You can ask “why was this returned” and get a real answer. Three: adding new retrieval consumers — an MCP-aware editor, a CLI tool, an admin dashboard — is a configuration change, not a code change. The memory layer is the integration surface.

That is the case for treating memory as infrastructure. The bandage stack was a way to ship the first version of a chat app. The dedicated layer is the way to ship the next ten.