Vector similarity is approximate, so people inflate top-k to avoid missing the answer. A cheap reranking pass lets you fetch many candidates but send only the few that matter to the expensive LLM.
Add a Reranker and a Hard Token Budget: Retrieve 20 Candidates, Send Only the Best 3
🔒 Pro tip · Intermediate
Unlock this tip — and 37 more
This is one of 38 advanced, fact-checked tactics reserved for Pro. Get the full 60-tip library, a searchable archive, and a new tip every morning for $9/mo.
Prefer to browse? The 22 Beginner tips are free forever.
More in Retrieval & RAG
🔎Retrieval & RAG
Eliminates the embedding API call and vector search on cache hits. The saved cost tracks your hit rate; it cuts embedding/retrieval spend, not the tokens sent to the LLM.
Cache the Context, Not Just the Answer
Cache the retrieved chunk set keyed by a normalized query, so popular or repeated questions skip the embedding call and vector search and reuse the same context block instead of rebuilding it every time.
🔎Retrieval & RAG
70-95% on document-heavy prompts
Stop Pasting Whole Documents: Retrieve the 3 Chunks That Actually Answer the Question
Dumping a full PDF or knowledge base into every prompt bills you for thousands of tokens the model never needed. Retrieve only the passages relevant to the question instead.
🔎Retrieval & RAG
20-40% fewer retrieved tokens at equal accuracy
Chunk on Structure, Not Character Count, So You Retrieve Fewer (and Smaller) Chunks
Naive fixed-length chunking splits ideas mid-sentence, forcing you to retrieve more chunks (and more overlap) to capture one answer. Chunk on semantic boundaries to send fewer tokens per query.