Appearance
Day 3 -- Retrieval Quality & Ranking
By Day 2 you had a working vector store with chunked documents. The system retrieves something when you query it. But is it retrieving the right things? Vector similarity alone misses acronyms, product codes, and rare terms. The top-10 results might be only loosely relevant. And when retrieval fails entirely, your system needs to handle it without hallucinating.
Today covers the techniques that turn a working retrieval pipeline into a reliable one: hybrid search that combines vector and keyword matching, cross-encoder re-ranking to sort results by actual relevance, metrics for measuring retrieval quality, query rewriting with LLMs, and fallback strategies for when nothing matches.
What you'll have by the end of today
- A hybrid search pipeline that fuses vector similarity with PostgreSQL full-text search using reciprocal rank fusion
- A re-ranking layer that improves result relevance using cross-encoder models
- A retrieval evaluation suite that measures recall@k, MRR, and NDCG against a test set
- Query rewriting techniques including multi-query retrieval and HyDE (hypothetical document embeddings)
- Graceful failure handling: fallback chains, empty-result responses, and confidence thresholds
Lessons
- Hybrid Search -- combining vector similarity with keyword search using pgvector full-text and reciprocal rank fusion. When vector search fails on acronyms, product codes, and rare terms, and why BM25-style keyword matching catches what embeddings miss.
- Re-Ranking -- using cross-encoder models to re-rank the top-k retrieval results for better relevance. The cost-quality tradeoff: when to re-rank and when vector similarity is enough.
- Evaluating Retrieval -- measuring what your retrieval pipeline actually returns. Building query-document test sets. Computing recall@k, MRR, and NDCG. Automating evaluation so you can iterate on retrieval without guesswork.
- Query Rewriting -- using an LLM to expand, decompose, or rephrase queries before retrieval. Multi-query retrieval with fusion. HyDE: generating hypothetical documents to search against.
- Handling Failures -- when retrieval returns nothing or only irrelevant results. Fallback strategies. Admitting ignorance instead of guessing. Empty result handling and confidence thresholds.