Introduction

In Step 1, we built a simple grounded LLM application. The application could answer questions using our documents, but there was a fundamental problem: we sent all documents to the model for every question.

That works for a handful of documents. It doesn’t work for a real knowledge base. So the next question is: How do we give the LLM only the information it actually needs? This leads us to retrieval and the moment we learn about RAG.

As usual, you can find the code for this example here: https://github.com/genoiucosmin/AIApp1/tree/Step2

From all documents to relevant documents

Step 1

Question + ALL documents → LLM → Answer

Step 2

 Question -> Find relevant chunks -> Selected context -> LLM -> Answer

Let’s explain why this is cheaper, faster and more scalable conceptually.

Chunking and our first retrieval mechanism

In the previous example we sent in all documents to the LLM. This is clearly not something scalable, so we need to define a way of sending only the relevant data to the LLM. This is actually RAG – and we will implement a very primitive version of it, by just splitting the documents we had into chunks of 300 characters.

Then we find out how many times each word from our question appears in each chunk. We then select the top three chunks based on this score, and send only those to the LLM. That is it.

So our Step 2 doesn’t use a vector database or an embedding model yet. Instead, we created a very simple version of the retrieval component yourself.

Question -> Compare words from question with every chunk -> Calculate relevance score -> Sort -> Take top N

For us calculating the relevance score just means how many words are in each chunk. This is intentionally a simplified implementation, we will see more advanced ways of doing this in the future lessons. In a production RAG system, that component might instead look like:

Retrieval-Augmented Generation

The simplest way to understand RAG is to stop thinking of it as one magical AI operation. It is a pipeline containing at least two fundamentally different tasks:

This separation is fundamental to RAG. The original RAG architecture is explicitly based around a retriever and a generator: the retriever returns relevant passages, and the LLM generates text conditioned on those passages and the user’s question.

1. Retrieval = finding information

The retriever’s job is not to answer the question but to analyze: “Given this question, which pieces of my knowledge base are likely to contain the answer?”

For example, suppose your knowledge base contains:

Document A:Our refund policy allows customers to request a refund within 30 days of purchase.
Document B:Premium customers receive priority technical support.
Document C:Subscriptions can be cancelled from the account settings.

If the user asks: “How long do I have to ask for my money back?”, the retriever should just return: Document A. It doesn’t need to formulate: “You have 30 days.”, it simply needs to find the relevant information. Real systems can use keyword search, vector/semantic search, hybrid search, reranking, metadata filtering, etc.

2. Generation = producing the answer

Now the LLM receives something like:

System instructions: Answer using the supplied context. Retrieved context:"Our refund policy allows customers to request a refund within 30 days of purchase. "User:"How long do I have to ask for my money back?"

The LLM’s job is now completely different: Understand the retrieved information and formulate a useful natural-language response. It might produce: “You can request a refund within 30 days of purchase.” So to summarize:

Retriever = information discovery

Generator = language/reasoning/output

So why our implementation is still primitive? Our retrieval mechanism has important limitations:

• synonyms are not understood;
• different wording can produce poor results;
• exact word matching is simplistic;
• fixed-size character chunks are crude;
• there is no semantic similarity;
• there is no vector database;
• there is no evaluation framework.

And this is intentional. The goal is to understand the mechanism before introducing specialized infrastructure.

What comes next?

The obvious problem is now: what if the relevant information uses different words than the question? For example:

Question:
“How can I terminate my subscription?”

Document:
“Customers may cancel their membership at any time.”

Keyword matching may struggle. Humans understand that these concepts are related so we need a representation of meaning. That leads us to embeddings and semantic search, that we will discuss in depth in the next chapter.

What you learned in step #2:

• documents can be chunked into retrievable units;
• retrieval should happen before generation;
• RAG separates retrieval from generation;
• simple lexical retrieval has significant limitations;
• the next step is semantic retrieval using embeddings.

What we have here is actually a major interview concept, and from my experience as both interviewer and interviewee, a question that gets asked a lot in the beginning of the discussion around AI understanding for programmers is:

“How would you troubleshoot a RAG system that gives bad answers?” To answer don’t immediately say: “Change the prompt.” Think about the larger picture and propose a solution depending on the context:

Interview Questions

To make sure you understood the topic presented here, try answering the questions below:

• What is RAG?
• Why do we chunk documents?
• What is retrieval?
• What is the difference between retrieval and generation?
• What is lexical search?
• What are the limitations of keyword-based retrieval?
• Why might semantic search perform better?

Next Step

In Step 3 we replace keyword matching with embeddings and vector similarity, allowing the application to retrieve information based on meaning rather than exact word matches.