LLMs have two problems that make them frustrating for knowledge-intensive applications. First, they hallucinate - they generate plausible-sounding text that’s factually wrong. Second, their knowledge is frozen at their training cutoff - they don’t know about things that happened after they were trained.

Retrieval-Augmented Generation (RAG) addresses both problems by giving the model access to relevant documents at query time. Instead of relying on knowledge baked into its weights, the model can base its answer on documents you provide. This reduces hallucination and allows the system to answer questions about any documents you give it, regardless of training cutoff.

The Core Idea

A RAG system works in two phases:

Retrieval: given a user’s question, find the most relevant documents or passages from your knowledge base.

Augmented generation: include those retrieved passages in the prompt as context, and ask the LLM to answer the question based on that context.

User: "What was the revenue in Q3 2025?"

1. Retrieve: find relevant passages from your financial documents
   → "Q3 2025 revenue was $4.2B, up 12% YoY..."

2. Generate:
   Prompt: "Based on the following documents:
   [Q3 2025 revenue was $4.2B, up 12% YoY...]
   Answer: What was the revenue in Q3 2025?"

   Response: "Revenue in Q3 2025 was $4.2 billion, representing
   a 12% year-over-year increase."

The model isn’t recalling this from training - it’s reading documents you provided and summarizing them. If the document doesn’t contain the answer, the model can say so.

Embeddings and Vector Search

The hard part of RAG is retrieval: given a question, how do you quickly find relevant documents from potentially millions of candidates?

Keyword search (like a traditional search engine) works poorly for semantic questions. “What was the company’s financial performance last quarter?” should match documents about “Q3 revenue” and “quarterly earnings” even though none of those words appear in the question.

The solution: embeddings. An embedding model converts text into a high-dimensional vector (a list of floating point numbers) where semantically similar texts produce similar vectors. “Revenue” and “earnings” produce similar vectors because they’re used in similar contexts in the training data.

from openai import OpenAI

client = OpenAI()

def get_embedding(text: str) -> list[float]:
    response = client.embeddings.create(
        model="text-embedding-3-small",
        input=text
    )
    return response.data[0].embedding
    # Returns a list of ~1536 floats

To find relevant documents: embed the query, compare it against the embeddings of all documents, return the ones with the highest similarity (cosine similarity is standard). This is called a vector search or nearest-neighbor search.

Vector Databases

Running a similarity search over millions of embeddings with a naive approach (compare query against every document) is too slow. Vector databases are specialized for this: they index embeddings and support approximate nearest-neighbor search efficiently.

Popular options:

• pgvector: Postgres extension. If you’re already on Postgres, this is the simplest path.
• Pinecone: managed vector database, no infrastructure to run.
• Weaviate, Qdrant, Chroma: open-source vector databases with various tradeoffs.
• Milvus: open-source, designed for very large scale.

A basic RAG pipeline with pgvector:

import psycopg2
from openai import OpenAI

client = OpenAI()
db = psycopg2.connect(DATABASE_URL)

def search_documents(query: str, top_k: int = 5) -> list[dict]:
    query_embedding = get_embedding(query)

    cursor = db.cursor()
    cursor.execute("""
        SELECT id, content, 1 - (embedding <=> %s::vector) AS similarity
        FROM documents
        ORDER BY embedding <=> %s::vector
        LIMIT %s
    """, (query_embedding, query_embedding, top_k))

    return [
        {"id": row[0], "content": row[1], "similarity": row[2]}
        for row in cursor.fetchall()
    ]

def answer_question(question: str) -> str:
    relevant_docs = search_documents(question)
    context = "\n\n".join(doc["content"] for doc in relevant_docs)

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {
                "role": "system",
                "content": "Answer questions based only on the provided context. "
                           "If the context doesn't contain the answer, say so."
            },
            {
                "role": "user",
                "content": f"Context:\n{context}\n\nQuestion: {question}"
            }
        ]
    )
    return response.choices[0].message.content

Chunking

Documents are usually too long to embed as a single unit. A 50-page PDF needs to be split into chunks that are small enough to embed meaningfully and fit in the context window when retrieved.

Chunking strategies:

• Fixed size: split every N characters or tokens with some overlap (so important context isn’t split across chunk boundaries)
• Sentence/paragraph: split on natural language boundaries
• Semantic: split based on topic shifts (harder to implement, often better quality)

Chunk overlap matters. If you split at 500 tokens with no overlap, a sentence that spans the boundary will be split in half. A 50-100 token overlap catches these edge cases.

def chunk_document(text: str, chunk_size: int = 500, overlap: int = 50) -> list[str]:
    words = text.split()
    chunks = []
    start = 0
    while start < len(words):
        end = start + chunk_size
        chunks.append(" ".join(words[start:end]))
        start += chunk_size - overlap
    return chunks

Chunk size involves a tradeoff: smaller chunks retrieve more precisely but may miss broader context; larger chunks provide more context but reduce retrieval precision.

What Makes RAG Fail

Retrieval misses: the relevant document exists but doesn’t get retrieved. Causes: poor embedding quality, wrong chunk size, too many documents for approximate search to be accurate, the query doesn’t semantically match the document language.

Context length overflow: retrieved documents plus the question exceeds the model’s context window. Fix: retrieve fewer documents, use smaller chunks, or summarize documents before including them.

The model ignores context: if the retrieved context contradicts the model’s training knowledge, some models will favor their training data. Use system prompts that explicitly instruct the model to rely on provided context.

Garbage in, garbage out: low quality source documents (PDFs with poor text extraction, HTML with navigation noise, duplicate content) produce low quality answers. Document preprocessing quality matters.

Semantic drift: the query is about one topic, the retrieved documents are about a superficially similar but different topic. Hybrid search (combining vector search with keyword search) often improves retrieval quality.

The Bigger Picture

RAG is a pragmatic solution to a real problem, but it shifts complexity rather than eliminating it. Instead of worrying about hallucination in the base model, you worry about retrieval quality, chunking strategy, embedding model choice, and whether your documents are up to date.

For internal knowledge bases, customer support, and document Q&A, RAG works well and produces significantly better results than asking an LLM to recall facts from training. For tasks where retrieval quality is hard to guarantee - very large document sets, highly ambiguous questions, specialized domains - the quality depends heavily on the retrieval layer, which requires its own investment to tune.