How to Optimize RAG Retrieval: Diagnose First, Then Tune

If your RAG pilot gives wrong or incomplete answers, you have to decide whether to tune retrieval in-house, bring in outside help, or hold off. The reader in mind is the CTO, head of data or AI lead who has to make that call. It shows how to diagnose where retrieval fails, which fix to test first, what each fix costs in latency and effort, and when not to bother.

Most of the useful work happens before any tuning. A team that cannot say whether the right passage was retrieved, ranked low or never indexed will spend weeks changing settings and still not know why answers are wrong. That is a measurement problem, and it is cheaper to solve first.

Published figures are cited to their original sources with the conditions they were measured under. Suggested workflows are engineering recommendations to adapt and test on your own data.

Diagnose before you tune

Start by finding which stage fails. Take a set of questions where you know the correct source passage and check what happened to that passage.

What you observe	Likely cause	First thing to test
The right passage is not among the retrieved candidates	Chunking, embedding, indexing, candidate depth, filters or query formulation	Inspect how the source was split; check that it was indexed, that no filter excluded it and how deep the candidate pool is
The right passage is retrieved but ranked low	Ranking problem	Add or tune a reranking stage
Exact codes, dates, clause numbers or names are missed	Lexical gap in vector-only search	Add keyword search alongside vector search (hybrid retrieval)
Answers cite superseded content	Freshness or lifecycle problem	Check how updates and deletions reach the index
Questions that need several documents fail	Single-pass retrieval limit	Try query decomposition or multi-step retrieval
The right passage is in the final context but the answer is wrong	Generation or grounding problem	Check grounding, citation correctness and the prompt
The system shows content the user should not see	Access control, not tuning	Fix authorization first; see the governance section

If this table shows a gap your current team cannot close, read Build in-house or get help first. Otherwise keep reading, because each fix below has a price.

Measure a baseline first

Tuning without a baseline produces opinions, not evidence. Build a small labeled set before changing anything:

1. Collect real user questions, including hard ones, ambiguous ones and some the corpus cannot answer.
2. For each answerable question, record the source passage that should be retrieved.
3. Run retrieval and record where each expected passage lands in the ranked results.
4. Report retrieval recall (was the expected passage in the top k?) separately from answer quality and citation correctness. A fluent answer can hide a retrieval miss.
5. Record the configuration, source versions and run date, then change one variable at a time.

Do not adopt a universal target such as "recall above 90%". Acceptable levels depend on the task and the cost of a wrong answer. The data preparation guide covers building the evaluation set and the permission and deletion tests that belong in it.

Worked example: a fictional 20-query diagnosis

Suppose a team labels 20 questions and records where each expected passage appears. These numbers are invented to show the arithmetic. They are not a Lucent client result or a benchmark. "Retrieved" below means inside the top 50 candidates, and "ranked low" means below the top 5 passed to the generator.

Stage	Queries with the expected passage	Rate
In the top 50 candidates from first-stage retrieval	17 of 20	85%
In the top 5, before reranking	11 of 20	55%
In the top 5, after adding a reranker	15 of 20	75%

Candidate recall    = 17 / 20 = 85%
Top-5 recall before = 11 / 20 = 55%
Top-5 recall after  = 15 / 20 = 75%
Outside the top 50                     = 20 - 17 = 3 queries
Retrieved but ranked low (before)      = 17 - 11 = 6 queries
Retrieved but still ranked low (after) = 17 - 15 = 2 queries

The reading: most failures here are ranking failures, so a reranker is a reasonable first test. It cannot recover the 3 queries whose passage was outside the 50 candidates it received, because a reranker only reorders its input. Those need a deeper or better first stage, better chunking, a lexical search path or a check that the source was indexed. The 11 and 15 are net counts, since a reranker can also push a passage out of the top 5. After reranking, 3 of the 5 remaining failures are first-stage failures, so the next test moves to the first stage.

Fix options and trade-offs

Each option below addresses a different symptom. Cost, effort and timeline are described by direction, because the actual numbers depend on your corpus, traffic, labeling capacity, re-indexing time and vendor terms.

Fix	Helps when	Latency and cost direction	Tuning effort	Main risk
Better chunking	Right answer is split across chunk boundaries	Usually indexing cost only; re-indexing on change	Medium: needs corpus-specific testing	A new strategy can regress other queries
Hybrid retrieval	Exact terms, codes or dates are missed	Extra search path per query	Low to medium	Fusion settings need tuning on your queries
Reranking	Right passage is retrieved but ranked low	Adds a rerank step per query; cost grows with candidate count, and fewer chunks reaching the generator can offset some of it	Medium	Latency budget; candidate-count limits in some services
Contextual retrieval	Chunks lose meaning without their document	Extra model calls at indexing time	Medium to high	Indexing cost and re-processing on updates
Agentic retrieval	Questions need several retrieval steps	Multiple model and search calls per question	High	Latency, cost, cascading errors

Chunking

Chunking is often an afterthought. Chroma's chunking evaluation found that some strategies outperformed others by up to 9% in recall, as reported by Chroma. That is a best case across the strategies it compared, but large enough to be worth testing. The same study notes that a simple recursive splitter often performs well in practice when its parameters are set appropriately. The lesson is not that one method wins. It is that you should compare candidates on your own documents and questions.

For long documents with nested structure, hierarchical or parent-document retrieval matches small units and returns broader parent context. That can improve answer completeness, but any broader context you return must be authorized for the user as well as the matched chunk.

Hybrid retrieval

Vector search finds meaning but can miss exact terms such as product codes, clause numbers, dates and names. Keyword search (typically BM25) is better at those. Hybrid retrieval runs both and merges the results.

Reciprocal Rank Fusion (RRF) is a common way to merge them. It does not compare raw scores. Azure AI Search documents the formula as 1/(rank + k), where rank is a document's position in each list and k is a constant, with 60 given as an example value. Because it uses rank positions only, vector similarity scores and BM25 scores never need to be put on the same scale. The trade-off is that score magnitude is discarded.

Hybrid retrieval is not guaranteed to win on every corpus. Benchmark it on your own queries, especially the ones with exact identifiers.

Reranking

A common pattern retrieves a larger candidate set cheaply, then reorders it with a more precise model before passing the top results to the generator. The first stage typically uses a bi-encoder, a keyword index or both. With a bi-encoder, documents are embedded ahead of time and the query is embedded at request time, then the two vectors are compared. That is fast, but each side is compressed into a single vector without seeing the other. A cross-encoder reranker reads the query and a candidate together, which is slower per comparison and usually more precise.

Candidate counts are a tuning choice, not a standard. Anthropic's contextual retrieval tests retrieved 150 candidates and reranked them down to 20. Azure AI Search's semantic ranker, as documented, reranks the top 50 matches, so managed services can cap how many candidates you may rerank. Treat the following as a starting point, not a standard: tens to about 150 candidates, reranked to roughly 5 to 20 passages, within your service's cap. Test it against your recall measurements and latency budget.

Rerankers are offered by several vendors and open-source projects, including Cohere Rerank, Voyage rerank, Jina Reranker and BAAI's BGE-Reranker. Model versions change quickly, so check each vendor's current release before choosing.

Reranking adds a rerank step to every query, and there is a trade-off between reranking more candidates for accuracy and fewer for lower latency and cost. If your first-stage recall is already good and ranking is not your problem, it adds cost without adding accuracy.

Contextual retrieval

Anthropic's contextual retrieval prepends a short, model-generated description of where a chunk sits in its document before embedding it. In Anthropic's tests, the reduction in top-20 retrieval failures, measured as 1 minus recall@20, was:

• 35% from contextual embeddings alone;
• 49% when combined with contextual BM25;
• 67% when reranking was added as well.

All three are measured against the same 5.7% baseline failure rate. The 67% is the result of the whole pipeline, not of reranking alone. The figures are averages across several datasets, including codebases, fiction and scientific papers, with a specific embedding model, and the reranking step used 150 candidates reranked to 20.

The technique adds a model call per chunk at indexing time. Anthropic's 2024 post estimated a one-time cost of about $1.02 per million document tokens, assuming prompt caching, 800-token chunks, 8k-token documents, 50 tokens of context instructions and 100 tokens of context per chunk. The cost recurs when you re-index, so check those assumptions against your own chunk sizes and current model pricing before budgeting.

Agentic retrieval

Some questions need several retrieval steps: "compare this quarter's results with the contract terms and flag the differences." A single retrieval pass may not gather everything. Agentic retrieval lets a model break the question into sub-questions, run several retrievals, call tools and combine the results.

That is one way to handle multi-step questions, not the only way. Query decomposition with a fixed number of steps may be enough and is easier to test.

That holds when the number of hops is predictable; if it varies widely, fixed steps can miss questions. An agentic loop adds latency and cost because each step is another model and search call, and an error early in the chain can carry through to the final answer. Set limits on the number of steps, log every retrieval and restrict which tools and data the agent can reach. Add human review for high-stakes decisions.

Governance, freshness and prompt injection

Retrieval tuning can improve answers and still leave serious gaps. Three need attention before launch:

• Permissions at retrieval time. Filter by the user's permissions before ranking, and propagate permission changes and revocations to the index and any caches. A cache keyed only on the query and document hashes can serve one user's answer to another, so include the caller's permission scope (for example a tenant plus an ACL version) in the cache key and invalidate entries when permissions change.
• Freshness. Updated and deleted documents must reach the index. Track which chunks belong to which document so superseded chunks can be retired. The data preparation guide covers the update and deletion workflow.
• Malicious instructions in retrieved content. OWASP's LLM01 guidance states that RAG and fine-tuning do not fully mitigate prompt injection vulnerabilities. A document a user is allowed to retrieve can still contain instructions that try to manipulate the model. Test for this, and limit what the model can do with retrieved content.

For high-risk AI systems, the EU AI Act requires that the systems technically allow automatic event logging (Article 12). Providers must keep those logs, to the extent they are under their control, for a period appropriate to the intended purpose of at least six months unless other law provides otherwise (Article 19). Deployers must keep the logs the system generates, also for at least six months unless other law provides otherwise (Article 26). Logging which sources were retrieved can help with traceability. Whether a given system is high-risk, which role your organization has and what applies to it are questions for legal counsel.

Build in-house or get help

Lucent Innovation provides AI and ML services, so we have a commercial interest in this decision. Use the criteria below to decide whether outside help fits the actual gap.

An in-house approach may be enough when your team already owns the ingestion pipeline, the search infrastructure and an evaluation set, and the diagnosis points to a fix your engineers can test.

Consider specialist help when the diagnosis exposes a capability you cannot build or maintain internally, such as designing the evaluation set, integrating a reranker within your latency budget, or enforcing permissions across several source systems. Name the gap before you buy. A partner can tune a pipeline, but it cannot decide for your business which document is the authoritative answer.

When not to do this

Do not add more retrieval machinery when:

• The corpus is small. Anthropic's 2024 post said a knowledge base under about 200,000 tokens (roughly 500 pages) can go directly in the prompt with no need for RAG, and it tied that to prompt caching. Treat that as a starting point, not a rule. Context limits differ by model, every request pays for the whole corpus unless it is cached, and cache lifetimes are short. Everything in the prompt is visible to the model, so this only suits content every user may see. Test accuracy, cost and latency against RAG.
• You have too few test questions. Results from a handful of queries are anecdotes. Build the labeled set first.
• The baseline already meets the task. If retrieval is good enough for the risk of the use case, further tuning is cost without benefit.
• The source data is the problem. Duplicates, obsolete versions, flattened tables and missing metadata cannot be fixed by a better reranker.
• The questions are single lookups. Agentic retrieval adds cost and failure modes that a one-step lookup does not need.
• The business case is unproven. Confirm the value before funding more engineering; see how to measure RAG ROI.

Questions to ask any vendor, including Lucent

These cover retrieval tuning. For permission, revocation, cache, handover and ownership questions, use the data preparation guide. For scope and funding questions, use the RAG ROI guide.

1. What will you measure first, and what baseline will we review before you change anything?
2. Which failures do you expect the proposed change to fix, which will it leave untouched, and what result would make you recommend no further work?
3. What do the added latency and per-query cost look like at our expected traffic?
4. How will you test for malicious instructions inside retrievable documents?

Compare the evidence in the answers, not only the number of techniques proposed.

Take your retrieval diagnosis to a conversation

If your diagnosis shows a gap you cannot close internally, such as evaluation set design or reranking integration, you can discuss your RAG requirements with Lucent. Bring your query set, current retrieval results and the failure pattern you found. Ask what work Lucent would propose, what evidence it would produce, what it would cost and how long it would take.

Share anonymized examples only, through your organization's approved process. Do not send credentials, personal records or confidential documents through an unapproved channel.

For related guidance, see the explanation of how RAG works, the overview of core RAG components, the enterprise RAG implementation guide and the article on common challenges in RAG implementation.