Advanced RAG Pipeline: Multi-Stage Retrieval & Reranking

Category: AI Coding Difficulty: Advanced Updated: 2026-05-28

Go beyond basic RAG with advanced techniques: multi-stage retrieval, cross-encoder reranking, hybrid search (BM25 + vector), query transformation, and self-querying retrieval.

Why Advanced RAG?

Basic RAG works for simple Q&A, but production systems need more. Users ask complex questions, documents are noisy, and a single retrieval pass often misses relevant information. Advanced RAG techniques solve these problems with multi-stage pipelines.

Technique 1: Hybrid Search (Vector + Keyword)

Vector search excels at semantic similarity but misses exact keyword matches. BM25 (keyword search) excels at exact terms but misses synonyms. Hybrid search combines both for the best of both worlds.

from langchain.retrievers import BM25Retriever, EnsembleRetriever
from langchain_chroma import Chroma

# Vector retriever
vectorstore = Chroma(embedding_function=embeddings, persist_directory="./chroma_db")
vector_retriever = vectorstore.as_retriever(search_kwargs={"k": 10})

# Keyword retriever
keyword_retriever = BM25Retriever.from_documents(documents)
keyword_retriever.k = 10

# Ensemble (weighted hybrid)
ensemble_retriever = EnsembleRetriever(
    retrievers=[vector_retriever, keyword_retriever],
    weights=[0.5, 0.5]
)
results = ensemble_retriever.invoke("API rate limits and authentication")
print(f"Retrieved {len(results)} results from both methods")

Technique 2: Cross-Encoder Reranking

Initial retrieval uses fast bi-encoders to get candidate documents. Then a slower but more accurate cross-encoder reranks them. This gives you the speed of vector search with the accuracy of neural reranking.

from sentence_transformers import CrossEncoder
from langchain.retrievers import ContextualCompressionRetriever
from langchain.retrievers.document_compressors import CrossEncoderReranker

# Load a cross-encoder model
reranker = CrossEncoderReranker(
    model=CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2"),
    top_n=5
)

compression_retriever = ContextualCompressionRetriever(
    base_compressor=reranker,
    base_retriever=vectorstore.as_retriever(search_kwargs={"k": 20})
)

# Only the top 5 most relevant chunks survive reranking
compressed_docs = compression_retriever.invoke("How to configure authentication?")

Technique 3: Query Transformation

Users often ask vague or multi-part questions. Query transformation rewrites the question before retrieval for better results.

Query rewriting: "Tell me about it" → "Tell me about the authentication system configuration"
Multi-query: Generate 3-5 variations of the same question, retrieve for each, deduplicate results
Step-back prompting: "How do I fix error X?" → "What causes error X and what are the common fixes?"

The Complete Advanced Pipeline

User Query
    |
    v
[Query Transformer] --> Rewrite for better retrieval
    |
    v
[Hybrid Search] --> Vector (k=20) + BM25 (k=20)
    |
    v
[Cross-Encoder Reranker] --> Pick top 5
    |
    v
[LLM with Prompt] --> Context + Question --> Final Answer