Most language models fail the same way: they sound completely confident while being factually wrong, or they simply do not know what happened after their training cutoff. Fine-tuning helps with style and task format, but it is a poor solution for grounding models in specific, updatable facts. The knowledge baked into weights during training is static the moment training ends.
Retrieval-augmented generation addresses this directly. Instead of relying on memorized knowledge, a RAG system fetches relevant documents at inference time and provides them as context alongside the query. The model reads, then answers. It is less like asking an expert from memory and more like asking a researcher who has just looked something up.
The idea is straightforward. The engineering is not. A poorly built RAG pipeline retrieves the wrong documents, stuffs too much irrelevant text into the context window, and produces answers that are wrong in subtler, harder-to-detect ways than a vanilla model would. This guide covers what actually works, where pipelines break, and how to know whether yours is any good.
What Is the RAG Architecture, Exactly?
The original formulation — Lewis et al., 2020, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — combined a dense retriever with a seq2seq generator, training them jointly. Production RAG today is more modular: a retriever, a vector store, optional reranking, and a generation model that are typically assembled from independent components rather than trained end-to-end.
The pipeline has five distinct stages:
- Ingestion — source documents are cleaned, split into chunks, and embedded into vectors that are stored in an index.
- Query encoding — the user query is embedded using the same (or a compatible) model.
- Retrieval — the query vector is compared against the index to find the most similar chunk vectors.
- Reranking — a second, more expensive model rescores the top-k retrieved chunks for relevance.
- Generation — the reranked chunks are assembled into a prompt context; the language model generates a response grounded in that context.
Each stage has failure modes. None of them are optional.
How Should You Chunk Your Documents?
Chunking is the first place RAG pipelines go wrong, and it receives far less attention than embedding model selection.
The goal is to split documents into units that are semantically coherent, small enough that a single chunk does not dilute retrieval signal, and large enough that the retrieved text actually contains a complete thought. These constraints are in tension.
Fixed-size character chunking (e.g., 512 or 1024 characters with overlap) is the default in most tooling. It is easy to implement and often works well enough for homogeneous corpora — API documentation, product manuals — where paragraph boundaries are consistent. It fails when sentences straddle chunk boundaries or when a concept spans a full section.
Recursive character splitting respects paragraph breaks before falling back to sentence breaks before falling back to character limits. This is a pragmatic improvement over naive fixed-size splitting and should be the default for most text corpora.
Semantic chunking uses embedding similarity between adjacent sentences to identify natural topic shifts and splits there. It produces more coherent chunks at the cost of variable chunk sizes and slower ingestion. It is worth the overhead for long-form documents where topic density varies substantially — research papers, legal documents, books.
Parent-document retrieval stores two levels of chunks: small chunks for precise retrieval, larger parent chunks for generation context. The retriever finds a small chunk; the system returns the parent chunk to the language model. This decouples retrieval precision from context richness and often outperforms single-level chunking without additional complexity.
Overlap between chunks — typically 10–20% of chunk size — is standard practice to avoid splitting a critical sentence at a chunk boundary. Overlap does not solve the coherence problem; it reduces the frequency of the worst cases.
Empirically, chunk sizes of 256–512 tokens with 10–15% overlap are a reasonable starting point for most corpora. Adjust based on the average length and internal structure of your source documents, not on benchmarks from a different domain.
Which Embedding Model Should You Use?
The embedding model determines the quality of the semantic index. A retriever can only surface what the embedding space makes distinguishable.
Several dimensions matter:
Domain fit. General-purpose text embeddings (trained on web text) perform well on general queries but degrade on technical, legal, or biomedical text where vocabulary and usage patterns differ substantially from training data. For specialized domains, embeddings fine-tuned on in-domain text — or evaluated specifically on domain-representative queries — consistently outperform general models.
Embedding dimensionality and storage. Higher-dimensional embeddings capture more nuance but increase index size and query latency. 768-dimensional embeddings (common in BERT-family models) and 1536-dimensional embeddings (common in larger models) are standard. Dimensionality reduction via PCA or learned projections can recover much of the retrieval quality at lower storage cost, but requires care to avoid compressing away signal.
Query-document asymmetry. Queries and documents are different text types. Models trained with asymmetric objectives — where query encoders and document encoders are separate — generally outperform symmetric models for retrieval tasks. Evaluate on query-document pairs representative of your actual workload, not on symmetric sentence similarity benchmarks.
The MTEB (Massive Text Embedding Benchmark) is among the most widely used benchmarks for retrieval-relevant embedding evaluation. It covers retrieval, reranking, and classification tasks across dozens of datasets. Treat any public leaderboard as a starting filter; always validate on your own query distribution before committing to a model, since standings shift frequently and benchmark coverage may not match your domain.
What Is Reranking and Why Does It Matter?
Approximate nearest-neighbor search over embeddings is fast but imprecise. Embedding retrieval at top-k returns documents that are broadly topically similar; it does not guarantee that the top result is the most relevant to the specific query.
Reranking applies a second model — typically a cross-encoder — to score each retrieved document-query pair jointly. Cross-encoders attend to the full pair simultaneously rather than encoding query and document independently, which allows them to model fine-grained relevance that bi-encoder retrieval misses. They are substantially slower than embedding lookup (which is why they are applied only to the top-k retrieved candidates, not the full index).
In practice, reranking a top-20 retrieval set down to top-5 or top-3 for context assembly consistently improves generation quality on complex or ambiguous queries. The latency cost — typically tens to a few hundred milliseconds for a cross-encoder over 20 candidates — is usually acceptable.
Reciprocal rank fusion (RRF) is an alternative when running multiple retrieval strategies in parallel (e.g., dense embedding retrieval plus sparse BM25 retrieval). RRF combines ranked lists without requiring a learned model, is robust to score scale differences between retrieval methods, and often matches or exceeds learned rerankers at lower implementation cost. Hybrid retrieval (dense + sparse) with RRF fusion is now a standard production pattern for heterogeneous document collections.
Context Assembly: What Goes Into the Prompt?
After reranking, the surviving chunks are assembled into the context that the language model receives. This step determines whether the model can actually use what was retrieved.
Several practical constraints apply:
Context window limits. Even large-context models have finite windows. Stuffing the window with marginally relevant text forces the model to attend over more noise. Empirically, retrieval quality degrades faster than context length increases — retrieving more is rarely better.
Ordering effects. Language models exhibit a primacy and recency bias: content near the beginning and end of the context is processed more reliably than content in the middle. Place the most relevant chunks near the query, not buried in the middle of a long context.
Context dilution. If retrieved chunks span multiple subtly different topics, the model may hedge, blend answers incorrectly, or fail to identify which portion of the context is authoritative for the query. Reranking aggressively to fewer, more relevant chunks mitigates this; instruction prompting that directs the model to cite specific passages helps further.
Citation and grounding instructions. Prompts that instruct the model to base its answer only on the provided context, and to indicate when it cannot find an answer in the retrieved documents, meaningfully reduce hallucination rates compared to open-ended prompts. This is a prompt engineering concern, not a retrieval concern, but it affects the output quality of the full pipeline.
RAG vs. Fine-Tuning: When Does Each Apply?
These are not competing approaches — they solve different problems. The decision tree is simpler than most discussions imply.
Use RAG when:
- The knowledge domain is large, dynamic, or updated frequently.
- Answers must be traceable to specific source documents.
- The model’s base capabilities (reasoning, instruction following, language generation) are adequate; only the knowledge is missing.
- You need to serve multiple knowledge domains without retraining.
Use fine-tuning when:
- The task format or output style is substantially different from what the base model produces.
- The model needs to internalize a specialized vocabulary, reasoning pattern, or domain convention that does not fit in a context window.
- Latency constraints preclude retrieval (though retrieval latency is often lower than assumed).
- You have high-quality labeled examples of the target behavior and the knowledge is stable.
The frequent confusion: fine-tuning a model on a large document corpus does not reliably transfer specific facts. Models fine-tuned on documents learn writing style and general domain patterns; they do not reliably retrieve specific facts from training documents the way a retrieval system does. If the question is “does the model need to know specific facts,” the answer is RAG, not fine-tuning.
RAG and fine-tuning compose well: a fine-tuned model with domain-appropriate reasoning patterns, augmented by retrieval at inference time for specific facts, is a common and effective architecture for specialized applications.
What Are the Most Common RAG Failure Modes?
Understanding the transformer architecture — covered in detail in our guide to transformer architecture — helps clarify why these failures occur at the model level. The RAG failures themselves, however, are mostly retrieval and pipeline failures, not model failures.
Retrieval miss. The correct document exists in the index but is not returned in top-k. Causes: chunking that splits the answer across multiple chunks, embedding model that does not represent the query domain well, index that has grown stale relative to the document corpus, or a query that is too short or ambiguous to produce a discriminative embedding. Mitigation: query expansion (generating multiple phrasings of the query before retrieval), hybrid retrieval (dense + BM25), and regular index freshness audits.
Context dilution. Retrieved documents are topically related but do not contain the answer. The model, faced with plausible-sounding context, generates a plausible-sounding answer that is not grounded in the retrieved text. This is the failure mode that most resembles hallucination and is hardest to detect without per-query evaluation. Mitigation: aggressive reranking, smaller context windows with higher relevance thresholds, and prompts that instruct the model to abstain when the context is insufficient.
Hallucination despite retrieval. The correct documents are retrieved and included in context, but the model generates an answer inconsistent with them. This occurs when the model’s prior from pretraining is stronger than the retrieved context signal, particularly for queries where the retrieved context contradicts a common-knowledge belief. Mitigation: grounding instructions that explicitly prioritize context over prior knowledge, and faithfulness evaluation (checking whether the generated answer is entailed by the retrieved context).
Latency and cost at scale. Dense retrieval over large indexes requires approximate nearest-neighbor search (HNSW, IVF, or similar), which trades recall for speed. At very large scales, the embedding and reranking steps dominate latency. Caching embeddings for repeated queries, indexing in tiers (hot/warm/cold), and aggressive top-k reduction are standard mitigations.
How Do You Evaluate a RAG System?
Evaluation is the hardest part, and the most frequently skipped.
A RAG pipeline has two separable components to evaluate: retrieval quality and generation quality. Conflating them obscures where the pipeline is failing.
Retrieval evaluation measures whether the system retrieves relevant documents. Standard metrics: recall@k (what fraction of relevant documents appear in top-k results), mean reciprocal rank (MRR), normalized discounted cumulative gain (NDCG). These require a labeled set of (query, relevant document) pairs. Building this labeled set is expensive; using LLM-assisted labeling with human spot-checking is a pragmatic approach for most teams.
Generation evaluation measures whether the model produces accurate, grounded responses given retrieved context. Two dimensions:
- Faithfulness — is the answer entailed by the retrieved context? (Can be estimated with NLI models or LLM-as-judge prompts.)
- Answer correctness — is the answer factually right? (Requires ground-truth answers; much more expensive to evaluate at scale.)
End-to-end evaluation asks whether the full pipeline — retrieval plus generation — produces answers users would rate as correct and useful. This is the metric that matters in production and the hardest to automate reliably.
RAGAS (Retrieval-Augmented Generation Assessment) is an open framework that formalizes several of these metrics and automates evaluation using an LLM judge. It is a reasonable starting point but should be validated against human judgments on your specific query distribution before being used as the primary signal.
The minimum viable evaluation setup: a curated set of 100–200 representative queries with known correct answers, automated faithfulness scoring, and manual review of a sample of failures each week. Everything beyond this is a refinement on that baseline.
Frequently Asked Questions
When should I use RAG instead of fine-tuning?
Use RAG when the problem is knowledge access — the model needs facts it does not have or that change over time. Use fine-tuning when the problem is behavior — the model’s output format, reasoning style, or task approach needs to change. If you need both, fine-tune first for behavior, then add retrieval for knowledge. The two approaches are complementary, not mutually exclusive.
What chunk size works best for RAG?
There is no universal answer. Start with 256–512 tokens and 10–15% overlap, then evaluate retrieval recall on a sample of your real queries. Smaller chunks improve retrieval precision but may omit necessary context; larger chunks improve context completeness but introduce noise that dilutes relevance signals. The right size depends on the typical density and structure of your source documents.
Why does RAG still hallucinate if the right document was retrieved?
Retrieval does not guarantee grounding. A model will sometimes generate an answer consistent with its pretraining prior rather than the retrieved context, particularly when the two conflict. Grounding failures increase when context windows are long (more for the model to ignore), when the query is ambiguous, and when the retrieved text requires inference rather than direct lookup. Explicit prompting to rely on context and automated faithfulness scoring both reduce this failure mode.
How do I evaluate whether my RAG pipeline is working?
Separate retrieval evaluation from generation evaluation. For retrieval, measure recall@k on a labeled query-document set. For generation, measure faithfulness (is the answer grounded in the context?) and correctness (is the answer right?). Build a curated evaluation set of 100–200 representative queries with known answers and run it regularly — not just at launch. Pipeline regressions after index updates or prompt changes are common and only visible with systematic evaluation.
Is hybrid retrieval (dense + sparse) worth the added complexity?
Usually yes, for heterogeneous corpora. Dense retrieval excels at semantic similarity; BM25 excels at exact keyword and entity matching. Queries about specific product codes, proper names, or technical identifiers often fail dense-only retrieval because the embedding space does not preserve exact string similarity well. Hybrid retrieval with reciprocal rank fusion combines both signals without a learned fusion model and is a reliable improvement over either method alone for most real-world document collections.