RAG Is Not Just Search: Chunking, Ranking, Reranking, and Hallucination Control

"RAG is just search plus a prompt" is the sentence that produces most disappointing RAG systems. Plain semantic search gets you a demo; it does not get you answers users trust. The gap between the two is a stack of unglamorous techniques careful chunking, hybrid ranking, a reranking pass, and explicit hallucination controls that turn "vaguely related paragraphs" into "the right evidence, in the right order, with the model honest about its limits." This guide is about that gap and how to close it.

Why naive vector search disappoints

A single embedding search returns the k chunks whose vectors sit closest to the question's vector. That sounds right, but it fails in predictable ways: it misses exact terms (product codes, error numbers, names) that semantic similarity blurs over; it returns near-duplicates that crowd out diverse evidence; and it has no notion of authority or freshness, so a stale draft can outrank the current policy. The model then dutifully answers from mediocre context and sounds confident doing it.

Question: "What's the refund window for error code E-402?"

Naive top-k may return:
  ✓ a paragraph about refunds (semantically close)
  ✗ three near-duplicate marketing blurbs about "hassle-free returns"
  ✗ misses the support doc that literally lists "E-402 → 14 days"
       because that doc is terse and embeds far from the chatty query

Chunking: the foundation everything rests on

If a chunk does not contain a coherent, self-contained idea, no amount of clever ranking rescues it. Chunking is where most quality is won or lost, and the right strategy depends on your content.

Strategies, from blunt to smart

Hybrid ranking: keywords and vectors together

Semantic search understands meaning; keyword search (BM25) nails exact terms. They fail in opposite ways, which is exactly why combining them wins. Hybrid retrieval runs both and fuses the results — commonly with Reciprocal Rank Fusion (RRF), which blends rankings without needing the two scoring scales to be comparable. The result catches both "what they meant" and "the literal code they typed."

# Reciprocal Rank Fusion: combine a keyword and a vector ranking.
def rrf(rankings, k=60):
    scores = {}
    for ranking in rankings:                 # each is an ordered list of doc ids
        for rank, doc_id in enumerate(ranking):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

fused = rrf([bm25_results, vector_results])  # best of both worlds

Reranking: the cheap precision multiplier

Initial retrieval optimises for speed across millions of chunks, so it is deliberately approximate. Reranking adds a second, slower-but-smarter pass over just the top candidates: retrieve, say, the top 50 cheaply, then use a cross-encoder reranker to score how well each one actually answers the query, and keep the best 5. Because it only runs on a handful of candidates, it is affordable — and it routinely produces the largest jump in answer quality of anything in the pipeline.

Stage 1  Retrieve top 50 (fast, approximate)   ── recall-oriented
Stage 2  Rerank those 50 with a cross-encoder   ── precision-oriented
Stage 3  Pass the top 5 to the model             ── grounded generation

Cost: rerank runs on 50 items, not the whole corpus → cheap, high impact.

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

def retrieve_and_rerank(question: str, k: int = 5):
    candidates = hybrid_search(question, top_k=50)        # cheap, recall-oriented

    # The cross-encoder scores how well each candidate answers THIS query.
    pairs = [(question, c.text) for c in candidates]
    for cand, score in zip(candidates, reranker.predict(pairs)):
        cand.score = score

    ranked = sorted(candidates, key=lambda c: c.score, reverse=True)
    return ranked[:k]                                     # precision-oriented top 5

Assembling context the model can use

Once you have the best chunks, how you present them matters. Deduplicate near-identical passages so the model sees diverse evidence. Order by relevance, and be aware that models attend more strongly to the start and end of long contexts so do not bury the best chunk in the middle. Label each chunk with a source id so the model can cite it, and keep the total tight: more context is not better context.

Hallucination control

Even with perfect retrieval, a model can embellish. Controlling that is a layered job, not a single setting.

Measure, or you are guessing

Every technique here is a knob, and knobs need a dial. Maintain an evaluation set of real questions with known-good evidence and answers. Measure retrieval (did the right chunk make the final cut?) and generation (is the answer faithful, relevant, and correctly cited?) separately, and re-run the suite on every change. Without it, "we added reranking" is a vibe, not a result.