Production RAG 2026: Lifting Search Quality with Hybrid Search + Reranking

Why retrieval is the bottleneck in RAG today

Any team that has shipped RAG (Retrieval-Augmented Generation) to production hits the same wall: the model is plenty smart, yet it confidently gives wrong answers "grounded on the wrong documents." The 2026 consensus is clear: most RAG failures happen in retrieval, not generation. The more naive the pipeline — pure vector search only — the more its failures concentrate in the retrieval step.

The root cause is the structural limit of any single retrieval method. Dense vector search is strong at "what does this sentence mean?" but weak at exact matches like a product code (KERN-4421), an internal acronym, or a precise function name. Keyword search (BM25) is the opposite: strong on exact matches, but it misses synonyms and context. The two methods fail in complementary ways. That is why production RAG has converged on a two-stage structure: cast a wide net with hybrid search (dense + sparse), then narrow precisely with a reranker.

This post walks through a production pipeline that fuses BM25 + dense vectors with RRF (Reciprocal Rank Fusion) and precisely re-orders only the top documents with a cross-encoder reranker — with code. We use pgvector as the baseline vector store and cover Qdrant as an alternative.

Background: strengths/weaknesses per strategy and the 2026 options

First, a comparison of three retrieval strategies.

Here is the hybrid support status of major vector stores as of 2026.

The point is not "which DB is best" but that the hybrid + reranking pattern itself drives search quality. IntentCode standardizes production RAG on Postgres + pgvector to keep operational complexity low, complementing it with BM25 via Postgres full-text search (or at the application level).

Deep dive: the two-stage retrieval pipeline

The overall flow looks like this.

query → [dense search top-20] ┐
        → [sparse search top-20] ┴→ RRF fuse top-20 → cross-encoder rerank top-5 → LLM

The principle is one line: "cast wide (optimize recall), then narrow precisely (optimize precision)." Stage 1 retrieves broadly (e.g. top-20) for recall; the reranker distills it to top-5.

1) Fusing two result sets with RRF

RRF safely merges two result sets with totally different score scales (cosine similarity vs BM25 score) using rank only. The formula is simple.

$$\text{RRF}(d) = \sum_{r \in \text{rankings}} \frac{1}{k + \text{rank}_r(d)}$$

k is a constant, usually 60. k=60 is known to balance the influence of top and bottom items well across many datasets (e.g. rank 1 → 1/61 ≈ 0.0164, rank 10 → 1/70 ≈ 0.0143).

def reciprocal_rank_fusion(
    rankings: list[list[str]], k: int = 60
) -> list[tuple[str, float]]:
    """Fuse multiple result sets (lists of doc ids) with RRF.

    rankings: each retriever's list of doc ids, in rank order
    returns: (doc_id, score) descending
    """
    scores: dict[str, float] = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)

A practical advantage of RRF is that it works without score normalization (min-max, etc.). You can add or swap retrievers without touching the fusion logic.

2) pgvector + BM25 hybrid search

Run dense search with pgvector and sparse search with Postgres full-text (ts_rank), then merge with the RRF above. First, the schema and indexes.

-- extension and table
CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE chunks (
  id          bigserial PRIMARY KEY,
  doc_id      text NOT NULL,
  content     text NOT NULL,
  embedding   vector(1024) NOT NULL,           -- match your embedding dim
  tsv         tsvector GENERATED ALWAYS AS (to_tsvector('simple', content)) STORED
);

-- dense: HNSW (cosine)
CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops);
-- sparse: full-text GIN
CREATE INDEX ON chunks USING gin (tsv);

At the application level, run both searches and fuse with RRF.

import asyncpg

DENSE_SQL = """
  SELECT id::text
  FROM chunks
  ORDER BY embedding <=> $1          -- <=> : cosine distance
  LIMIT $2
"""

SPARSE_SQL = """
  SELECT id::text
  FROM chunks
  WHERE tsv @@ plainto_tsquery('simple', $1)
  ORDER BY ts_rank(tsv, plainto_tsquery('simple', $1)) DESC
  LIMIT $2
"""

async def hybrid_search(
    pool: asyncpg.Pool, query: str, query_vec: list[float], top_k: int = 20
) -> list[str]:
    async with pool.acquire() as conn:
        dense_rows = await conn.fetch(DENSE_SQL, str(query_vec), top_k)
        sparse_rows = await conn.fetch(SPARSE_SQL, query, top_k)

    dense_ids = [r["id"] for r in dense_rows]
    sparse_ids = [r["id"] for r in sparse_rows]
    fused = reciprocal_rank_fusion([dense_ids, sparse_ids])
    return [doc_id for doc_id, _ in fused[:top_k]]

Note: tune pgvector's HNSW search breadth with SET hnsw.ef_search = 100;. Raise it if recall is short, lower it if latency is the problem.

With Qdrant, the same pattern is handled via a first-class API (BM25 sparse + dense fused with RRF in one call).

from qdrant_client import QdrantClient, models

client = QdrantClient(url="http://qdrant:6333")

results = client.query_points(
    collection_name="chunks",
    prefetch=[
        models.Prefetch(query=query_vec, using="dense", limit=20),
        models.Prefetch(query=models.Document(text=query, model="Qdrant/bm25"),
                        using="sparse", limit=20),
    ],
    query=models.FusionQuery(fusion=models.Fusion.RRF),  # first-class RRF
    limit=20,
)

3) Cross-encoder reranking

Re-order the RRF top-20 with a cross-encoder. A bi-encoder (embeddings) encodes the query and document separately, but a cross-encoder encodes the (query, document) pair jointly and scores relevance directly. Precision rises sharply at a cost — so apply it only to the top candidates, not the whole set. The open-source bge-reranker-v2-m3 balances accuracy and latency well and fits comfortably into a sub-500ms end-to-end RAG.

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("BAAI/bge-reranker-v2-m3", max_length=512)

def rerank(query: str, candidates: list[dict], top_n: int = 5) -> list[dict]:
    """candidates: [{'id':..., 'content':...}, ...] (RRF top-20)"""
    pairs = [(query, c["content"]) for c in candidates]
    scores = reranker.predict(pairs)          # per-pair relevance score
    ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
    return [c for c, _ in ranked[:top_n]]

Reranking is widely cited as the single highest-impact optimization in production RAG. Narrowing top-20 → top-5 markedly improves the signal-to-noise ratio of the context fed to the LLM.

Practical guide: assembling end to end

Tie the pieces above into one retrieval function.

async def retrieve(pool, embed_fn, query: str) -> list[dict]:
    # 1) embed the query
    query_vec = await embed_fn(query)
    # 2) hybrid search (RRF top-20)
    ids = await hybrid_search(pool, query, query_vec, top_k=20)
    # 3) load bodies
    rows = await pool.fetch(
        "SELECT id::text, content FROM chunks WHERE id::text = ANY($1)", ids
    )
    candidates = [{"id": r["id"], "content": r["content"]} for r in rows]
    # 4) cross-encoder rerank (top-5)
    return rerank(query, candidates, top_n=5)

Manage operational parameters per environment.

# config/retrieval.yaml
retrieval:
  dense_top_k: 20
  sparse_top_k: 20
  rrf_k: 60
  rerank_top_n: 5
  hnsw_ef_search: 100
reranker:
  model: BAAI/bge-reranker-v2-m3
  max_length: 512
  batch_size: 32

"Measuring" quality (RAGAS)

Catch regressions with numbers, not vibes. Here are the key RAGAS metrics and practical thresholds.

from ragas import evaluate
from ragas.metrics import (
    faithfulness, answer_relevancy, context_precision, context_recall,
)

result = evaluate(
    dataset,   # columns: question / answer / contexts / ground_truth
    metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
)
print(result)  # CI gate: fail the build if below threshold

Fix a golden set of 50–100 queries and run these scores as a regression test on every PR. You can then objectively track quality shifts whenever you change the retriever, prompt, or chunking strategy.

Troubleshooting

1) Hybrid is on, but precision dropped. Usually stage-1 recall is insufficient, so the reranker is picking "the best of bad candidates." Raise dense_top_k/sparse_top_k from 20 → 40 to secure recall first, and keep rerank_top_n the same. A reranker only works when good candidates are actually in the candidate set.

2) Reranking latency spikes at p95. Check you aren't running the cross-encoder over the full result set. Apply it only to the RRF top candidates (e.g. 20) and batch the inference with batch_size. If you only have CPU, drop to ONNX/quantized or a smaller reranker; under heavy load, split the reranker into a separate service and scale it horizontally.

3) Exact code/acronym queries still get missed. You are likely riding only the dense path. Check that the BM25-side plainto_tsquery tokenizes properly (special characters, case) and apply the simple dictionary plus dedicated normalization for identifier-like tokens. Log the result ids from both searches to confirm the sparse contribution actually reaches RRF.

Conclusion: the IntentCode view

The answer for production RAG in 2026 is not a bigger model but better retrieval. The two-stage structure of hybrid (dense + sparse) + RRF + cross-encoder reranking is no longer optional — it is table stakes. IntentCode standardizes production RAG on Postgres + pgvector to lower operational complexity, then adds BM25 complementary search, reranking, and a RAGAS-based regression gate to port "measurable search quality" into customer environments. When retrieval wobbles, every agent on top of it wobbles — RAG must be built solid from retrieval up.