Curing AI Amnesia

Problem numero uno, every new chat with an LLM starts from zero. Close the tab, and the context is gone. Ask the same model about yesterday's conversation and it has no idea what you're referring to. You get all the confidence of a person you go out on beautiful dinner date with who definitely remembers the food, the ambiance and decor, but not what your insightful, extremely interesting conversation on why hats with pockets is the best fashion idea in ages.

That statelessness (I've hit the legal limit of allowed "s"s in a word) causes a second, more visible problem: hallucination. A model has to produce an answer whether or not it actually knows one, so when it doesn't, it produces something plausible-sounding instead of something true. It's the same energy when you run into said date at Trader Joe's and they say "of course I remember yooooouuuu" a little too loud.

Retrieval-Augmented Generation - whoa, whoa expensive word alert. I know, I know but hey, this is pretty sophisticated stuff so let's take a breath and get it - also known as RAG addresses both.

Instead of relying only on what the model learned during training, RAG looks up relevant information from an external source at query time (at query time - just more time for emphasis) and feeds it into the model's context before it generates a response. Open-book test instead of closed-book guessing. Similar to asking someone you went on 4.5 very expensive dates with, a generic "How are you doing?" and using their response to guide the conversation. Instead of guessing that the flaps on their new hat is because they are a pilot. I do have a drone license but LLMs are rarely so lucky. They will confidently lie to you and complement you at the same time.

How retrieval works

RAG depends on embeddings (pronounced embeddings), which are numerical representations of text that capture meaning. (read: computers only can read numbers). Similar concepts land close together in vector space, which is what lets a retrieval system match a question to relevant content even when the wording doesn't overlap. It's the closest thing computers have to reading comprehension, and they got there through spite and linear algebra.

This is basically how you can do real-time searching on relevant concepts. For example, if from reading your profile, I learn that you are a big fan of the hit reality TV show, Love Island and I binge watch all the seasons, 2 days before we meet. When l ask to "pull you for a chat", that reference is directly from what I just very recently stuffed into my memory over a few sleepness nights. And if you are confused by that statement, we can also definitely talk about how you haven't watched the show in a long time and need to update your profile, either way, my current knowledge is grounded based on the context I recently acquired and generated (RAG!!!! - I'm just kidding. I actually told you want RAG stood for above. Pop quiz!).

A typical RAG pipeline has three stages:

  1. Index: documents are chunked, embedded, and stored in a vector database. (AKA turn words into numbers)
  2. Retrieve: the user's query is embedded, and the system finds the nearest matching chunks. (AKA turn users question into more numbers and find the numbers that stored that are similar)
  3. Generate: the retrieved chunks are added to the prompt, and the model generates an answer grounded in that context. (AKA all the numbers are together happy now we will be able to generate words from all the numbers we have)

Where this shows up in practice

  • Support chatbots that answer from a company's actual documentation instead of guessing at policy.
  • Internal tools that let you query your own notes, docs, or codebase directly.
  • Search-augmented assistants that cite sources instead of asserting facts from memory alone.
  • KeithBot

The common thread is that retrieval narrows what the model has to guess at. It doesn't eliminate hallucination, but it gives the model something concrete to point to before it answers.

Its basically like if I find out, through talking to you, that Roses are your favorite flower and I remember that. Then while thinking of something thoughtful, and hopefully memorable, to bring to our next date, I use the information you told me and retrieve (👀) it from my brain computer to bring you a boutique of Roses to which you said was the "most beautiful and thoughtful gift you've ever received", thats RAG IRL. Now if I tell you that candy is my favorite food and you bring me a used dog toy for my non-existent pet, that's hallucination. RAG would have helped out in that situation.

Hopes this helps, if not remember, all wounds heal in time and more bombshells can always enter the villa. Especially if the bombshells are adorable puppies who love their new toy.

Resources:

About

© 2026 Keith Williams. All rights reserved.