Retrieval-Augmented Generation

Retrieval-augmented generation (RAG) combines a language model with a retrieval step: relevant documents are fetched at query time and injected into the prompt so answers cite real sources instead of confabulating.

RAG is most useful when the answer depends on private, changing, or large bodies of knowledge that do not fit reliably in a model’s training data or static prompt. It does not make a model “know” the documents. It gives the model selected evidence and asks it to answer within that evidence.

When to use it

Use RAG when the system needs to answer from a controlled knowledge base, such as product documentation, support tickets, policies, contracts, research notes, or internal engineering records.

It is a poor fit when the task mainly needs deterministic lookup, exact aggregation, or transactional state changes. In those cases, use a database query, search endpoint, or tool call directly, then use the model only if natural-language synthesis is valuable.

Pipeline stages

Ingest, normalize source documents, preserve permissions, extract metadata, split content into chunks, embed those chunks, and store them in an index.
Retrieve, search for candidate chunks using vector similarity, lexical search, metadata filters, or a hybrid of these approaches.
Rerank, reorder candidates with a stronger model or domain-specific signal so the best evidence reaches the context window first.
Pack context, deduplicate, trim, and order retrieved chunks so the prompt contains enough evidence without drowning the model in noise.
Generate, ask the LLM to answer from the provided context, cite sources, and say when the evidence is insufficient.
Evaluate, measure retrieval quality, answer faithfulness, coverage, latency, and cost as the corpus changes.

Retrieval design choices

Chunk size controls recall and precision. Smaller chunks are easier to match but can lose context; larger chunks preserve meaning but may hide the exact answer.
Metadata filters prevent plausible but wrong matches, especially across tenants, products, versions, dates, languages, or permission boundaries.
Hybrid search combines semantic matching with keyword matching. It helps with exact identifiers, error messages, product names, and terms that embeddings may blur.
Reranking improves quality when the first retrieval pass returns many acceptable candidates but only a few are truly useful.
Context packing decides what the model actually sees. Ranking alone is not enough if repeated, stale, or low-signal chunks consume the window.
Freshness strategy defines how quickly source changes reach the index. Some systems can reindex in batches; others need event-driven updates.

Minimal implementation shape

async function answerWithRag(question) {
   // Retrieval is scoped before generation; filters enforce corpus and permission rules.
   const chunks = await retrieveRelevantChunks(question, {
      limit: 8,
      filters: { published: true },
   });

   // No evidence means no answer. This prevents the model from filling gaps.
   if (chunks.length === 0) {
      return {
         answer: "I do not have enough source material to answer that.",
         citations: [],
      };
   }

   // Numbered chunks give the model stable citation handles like [1] and [2].
   const context = chunks
      .map((chunk, index) => `[${index + 1}] ${chunk.text}`)
      .join("\n\n");

   return generateGroundedAnswer({
      question,
      context,
      instruction: "Answer only from the supplied context and cite the numbered sources.",
      citations: chunks.map((chunk) => chunk.sourceUrl),
   });
}

type RetrievedChunk = {
   id: string;
   text: string;
   sourceUrl: string;
   score: number;
};

async function answerWithRag(question: string) {
   // Retrieval is scoped before generation; filters enforce corpus and permission rules.
   const chunks = await retrieveRelevantChunks(question, {
      limit: 8,
      filters: { published: true },
   });

   // No evidence means no answer. This prevents the model from filling gaps.
   if (chunks.length === 0) {
      return {
         answer: "I do not have enough source material to answer that.",
         citations: [],
      };
   }

   // Numbered chunks give the model stable citation handles like [1] and [2].
   const context = chunks
      .map((chunk: RetrievedChunk, index: number) => `[${index + 1}] ${chunk.text}`)
      .join("\n\n");

   return generateGroundedAnswer({
      question,
      context,
      instruction: "Answer only from the supplied context and cite the numbered sources.",
      citations: chunks.map((chunk) => chunk.sourceUrl),
   });
}

public sealed record RetrievedChunk(
   string Id,
   string Text,
   string SourceUrl,
   double Score
);

public async Task<GroundedAnswer> AnswerWithRag(string question)
{
   // Retrieval is scoped before generation; filters enforce corpus and permission rules.
   var chunks = await RetrieveRelevantChunks(question, new RetrievalOptions
   {
      Limit = 8,
      Filters = new Dictionary<string, object> { ["published"] = true }
   });

   // No evidence means no answer. This prevents the model from filling gaps.
   if (chunks.Count == 0)
   {
      return new GroundedAnswer(
         Answer: "I do not have enough source material to answer that.",
         Citations: []
      );
   }

   // Numbered chunks give the model stable citation handles like [1] and [2].
   var context = string.Join(
      "\n\n",
      chunks.Select((chunk, index) => $"[{index + 1}] {chunk.Text}")
   );

   return await GenerateGroundedAnswer(new GroundedAnswerRequest
   {
      Question = question,
      Context = context,
      Instruction = "Answer only from the supplied context and cite the numbered sources.",
      Citations = chunks.Select(chunk => chunk.SourceUrl).ToList()
   });
}

from dataclasses import dataclass


@dataclass
class RetrievedChunk:
    id: str
    text: str
    source_url: str
    score: float


async def answer_with_rag(question: str):
    # Retrieval is scoped before generation; filters enforce corpus and permission rules.
    chunks = await retrieve_relevant_chunks(
        question,
        limit=8,
        filters={"published": True},
    )

    # No evidence means no answer. This prevents the model from filling gaps.
    if not chunks:
        return {
            "answer": "I do not have enough source material to answer that.",
            "citations": [],
        }

    # Numbered chunks give the model stable citation handles like [1] and [2].
    context = "\n\n".join(
        f"[{index + 1}] {chunk.text}"
        for index, chunk in enumerate(chunks)
    )

    return await generate_grounded_answer(
        question=question,
        context=context,
        instruction="Answer only from the supplied context and cite the numbered sources.",
        citations=[chunk.source_url for chunk in chunks],
    )

This code shows the runtime path for a single RAG answer:

Retrieve a small set of candidate chunks for the user’s question, using filters to avoid pulling from the wrong corpus.
Handle empty retrieval explicitly instead of asking the model to guess without evidence.
Pack context by numbering each chunk, which gives the prompt stable citation handles.
Generate a grounded answer with an instruction that limits the model to the supplied context and returns the source URLs as citations.

The important boundary is between retrieval and generation. Retrieval should be tested as an information-retrieval system; generation should be tested as a grounded synthesis step.

Common pitfalls

Chunks too large or too small for the domain.
No metadata filters, retrieval returns plausible but wrong documents.
Treating embeddings as the only search mechanism when exact terms matter.
Indexing documents without preserving authorization rules.
Passing too much context and making the prompt noisy.
Asking the model to cite sources without tracking which chunks came from which documents.
Skipping evaluation, quality drifts silently as content changes.

Evaluation

Evaluate RAG in layers:

Retrieval recall, did the retriever find the document or chunk that contains the answer?
Retrieval precision, are the top chunks relevant, current, and allowed for this user?
Groundedness, does the answer stay within the retrieved evidence?
Answer usefulness, is the response complete enough for the user’s task?
Operational quality, latency, index freshness, cost, and failure rates.

Golden question sets are useful, but production monitoring matters too. Content changes, model changes, and embedding model changes can all shift quality without any application code changing.

Relationship to prompts and agents

RAG often works with prompt engineering: the prompt tells the model how to use retrieved context, cite sources, and handle missing evidence. It can also support AI agents by giving a multi-step workflow access to current knowledge before it decides which tool to call next.