Production AI, explained simply. One concept, one working project, every edition. From AI School 369.
Your RAG did not notice the document changed
|
Layer by Layer · AI School 369 · Edition 0 · Start here Your RAG did not notice the document changed PLUS: Latent semantic memory, explained simply If you run RAG in production, this bug is already in your system. This edition shows the fix, with a project you can run in two minutes. In this edition:
Reading time: 7 minutes. The index is a photograph, not a window When you build a RAG index, you take a photograph of your documents at one moment. Then the documents change. Someone edits the returns policy. Someone deletes the shipping guide. The photograph does not. Your chatbot keeps answering from the old photograph, confidently, in the same tone, with the same formatting. Nothing crashes. Nobody gets an error. The answer is simply wrong, and it stays wrong until a human notices. This is the most common quiet failure in production RAG, and it has a boring, complete fix. Three habits: 1. Version stamps. Every chunk carries the version of the document it came from. When a document changes, you know exactly which chunks are stale. 2. Reindex on change, and only on change. A file hash tells you what actually changed. One edited paragraph should cost you one document of re-embedding, not the whole corpus. 3. Deletion propagation. When a document is deleted, its chunks leave the index in the same run. A deleted policy should never answer a question again. And the safety net over all three: a small set of canary questions with known answers, run after every refresh. If a refresh breaks an answer, the canaries fail loudly before your users do. The refresh loop in one picture: the change path, the delete path, and the canary guard, above the question path they protect. This diagram is the whole edition in one picture, the version worth sharing. The Build: rag-freshness This edition's project is a complete demo of the failure and the fix, in about 260 lines of Python. It indexes three tiny store documents (returns, shipping, warranty), then edits one and deletes another, and shows you both indexes answering. What actually happens when you run it (this is the real output):
Run it yourself: git clone https://github.com/urbanschool369-oss/ai-school-369-projects cd ai-school-369-projects/projects/rag-freshness pip install -r requirements.txt python main.py The one thing to notice: the demo runs the edit on a scratch copy, so you can run it as many times as you like. Try breaking it. Edit a document and ask its question before refreshing. That stale answer is what your users see when nobody owns freshness. Reading this as a forward? Layer by Layer is free, twice a week: subscribe here. Latent semantic memory, explained simply Saving an agent's conversations does not mean it can use what it learned. On the next call, something still has to decide what to bring back. Send everything, and the model rereads its whole history. Retrieve a few passages, and the useful part may be left out. A recent approach called Memorilla changes what gets retrieved. A normal RAG pipeline retrieves text and pastes it into the prompt. Memorilla never returns to text. A trained module reads across the stored document embeddings and produces just 16 memory tokens, vectors written directly in the language model's own input space. The model reads those tokens plus your question. The original documents are never shown to it. The honest numbers: on FactKG, a fact-checking benchmark, this scored 84.33% accuracy against 80.43% for giving the model the full context, while using 108x fewer tokens to represent the source. It was best on five of eight benchmarks, not all of them. And the token saving is not a total cost saving: the embeddings still need storage, and the memory module still reads across them. The limit that matters for builders: this needs access to the model's input embeddings. It is not a prompting trick, and it will not work through an ordinary text-only API. File it under ideas worth watching, not tools you can adopt this quarter. Our own take on inspectable agent memory, in plain SQL, is coming in Edition 4. Quick poll: where does your RAG hurt most? Reply to this email with the number. Results open the next edition. 1 · Freshness, stale answers like this one One-line takeaway: an index you never refresh is a photograph of a system that no longer exists. Stamp versions, reindex on change, propagate deletions, and let canaries stand guard. Next edition: Anatomy of an agent tool loop, and where it spins forever. The build is a loop with a step budget, a retry budget, and a circuit breaker. PS: What are you stuck on? Reply to this email and tell me. Reader problems become future editions, and the builds that go with them. If this was useful, forward it to one engineer who would enjoy it. That is how this grows. Shekar Kaki Layer by Layer is the AI School 369 newsletter. Production AI, explained simply. One concept, one working project, every edition. Subscribe / past editions · The projects repo · AI School 369 on YouTube |
Layer by Layer
Production AI, explained simply. One concept, one working project, every edition. From AI School 369.