Advanced Query Strategies

Learn HyDE, step-back prompting, sub-question retrieval and iterative multi-step retrieval for questions that a single search cannot answer.

20 minAdvancedCode Examples
The Core Idea: Some questions cannot be answered by one search with one phrasing. The question may look nothing like the answer, may be too specific to match general text, or may need facts that depend on each other. Advanced query strategies use a language model to plan the retrieval: write a hypothetical answer, ask a broader question first, split the question, or search in several dependent steps.

1. Intermediate: When Basic Rewriting Is Not Enough

Three kinds of mismatch survive simple rewriting.
Question versus answer
The question uses words like get my money back, while the document says refunds are issued to the payment method. They are about the same thing but sound different.
Specific versus general
A very specific question may not match a document that explains the general principle behind it.
Dependent facts
Who manages the team that owns the billing service needs two facts, and the second cannot be searched before the first is known.

2. Intermediate: HyDE, Hypothetical Document Embeddings

In HyDE, a language model first writes a short, plausible answer to the question, even without seeing any documents. That hypothetical passage is then embedded and used as the search query. It works because an answer looks like other answers: its vector sits closer to the real answer passage than the short question does.
1. Ask the model to write a passage that would answer the question.
2. Embed that passage.
3. Search the index with it.
4. Use only the retrieved real documents as evidence.
Never use the fake answer as evidence: it may contain invented facts. It exists only to find real passages. HyDE costs an extra model call and helps most when questions and documents are phrased very differently.

3. Intermediate: Step-Back Prompting

Step-back prompting asks a broader question before the specific one. For a question about a particular invoice amount, the step-back question might be how invoices are issued and what they include. Retrieving background material gives the model the principles it needs to reason about the specific case.
The usual pattern retrieves for both: the original specific question and its more general step-back version. The results are merged, so the model receives both concrete facts and the general rules.
Good fit: troubleshooting, why questions, and any case where the answer depends on a general policy or concept more than on one specific sentence.

4. Intermediate: Sub-Question Retrieval

Sub-question retrieval breaks a complex question into several simpler questions that can be searched independently, often in parallel. Each sub-question gets its own evidence, and the model combines the pieces into one answer.
StrategySub-questions depend on each other?Run
Sub-question retrievalNo, they are independentIn parallel
Multi-step retrievalYes, later steps need earlier resultsIn sequence
A comparison question such as how do the Basic and Pro plans differ is a good example of the parallel kind: one search for each plan, then a combined answer.

5. Advanced: Multi-Step and Iterative Retrieval

Multi-step retrieval is needed when one fact leads to the next. The system searches, reads what it found, forms the next query from it, and repeats. Each round is a small loop: search, read, decide whether the question is answered, and if not, ask for what is still missing.
Search. Retrieve with the current query.
Read. Extract the useful fact, such as the name of the owning team.
Decide. If the question is answered, stop. If not, write the next query.
Limit. Stop after a maximum number of rounds or when no new information appears.
Costs and risks: every round adds latency and model calls, and an early mistake carries into every later round. Cap the number of steps, keep a trace of every query and result, and compare against a simple single-search baseline before adopting this pattern.
Panel A contrasts HyDE and step-back. Panel B follows a two-step chain from the code below. Panel C is a strategy explorer.
Panel A · HyDE vs Step-BackBoth change what is searched for
HyDE
Search with a fake answer
Question
↓
Model writes a hypothetical answer
↓
Embed the answer text
↓
Finds answer-like passages
Step-back
Search a broader question too
Specific question
↓
Model asks the general question
↓
Retrieve both
↓
Brings background and facts
Panel B · A Two-Step ChainFrom the code in section 7
Who manages the team that owns billing?→Find owner: payments team→Find manager of payments team→Dana Lee
The second query could not be written until the first search finished
Panel C · Strategy ExplorerClick a situation to open it
Click to see why you should start simple▼
Each strategy adds model calls and delay. Many questions are solved by good chunking and hybrid search. Add a planning strategy only for the question types where measurements show a single search fails.
How to read it: Panel A shows two single-shot tricks, Panel B shows the chained approach, and Panel C helps you pick one by the symptom.

7. Python Implementation: HyDE, Step-Back and a Two-Step Chain

The program has five short documents and a word-match search. A tiny lookup table plays the role of the language model that writes a hypothetical answer or a broader question. The first two experiments show a failing question succeeding once it is replaced by a hypothetical answer or a step-back question. The third experiment is a two-step chain: it finds the team that owns the billing service, extracts the team name, and uses it to find the manager.
query_strategies.pyPython 3.10+ · Standard library only
stop_words = set(["how", "can", "i", "my", "the", "a", "that", "who", "is", "of", "by", "to", "get", "do", "does", "for", "and"]) documents = [    "Refunds are issued to the original payment method within 5 business days.",    "Invoices are issued monthly and list the plan fee and taxes.",    "The billing service is owned by the payments team.",    "The payments team is managed by Dana Lee.",    "Dana Lee joined the company in 2019.",] def words(text):    return set(w for w in "".join(ch if ch.isalnum() or ch == " " else " " for ch in text.lower()).split() if w not in stop_words) def search(query):    best = max(documents, key=lambda d: len(words(query) & words(d)))    return best if words(query) & words(best) else None # A pretend language model: it writes a plausible answer, even without any documentshypothetical = dict([("money back", "You can get a refund. Refunds are issued to your payment method after the return is processed.")])broader = dict([("49 dollars", "How are invoices issued and what do they include?")]) print("1. HyDE")question = "How can I get my money back?"print("  Question search       :", search(question))fake_answer = hypothetical["money back"]print("  Hypothetical answer   :", fake_answer)print("  Search with that text :", search(fake_answer)) print("")print("2. Step-back")question = "Why was my invoice 49 dollars?"print("  Question search :", search(question))step_back = broader["49 dollars"]print("  Broader question:", step_back)print("  Broader search  :", search(step_back)) print("")print("3. Iterative, multi-step retrieval")def after(text, marker):    return text.split(marker)[1].rstrip(".") if marker in text else None first = search("billing service owned by team")team = after(first, " owned by the ")print("  Step 1 finds:", first, "->", team)second = search(team + " managed by")manager = after(second, " managed by ")print("  Step 2 finds:", second, "->", manager)print("  Final answer: the billing service team is managed by", manager)
Expected output
1. HyDE  Question search       : None  Hypothetical answer   : You can get a refund. Refunds are issued to your payment method after the return is processed.  Search with that text : Refunds are issued to the original payment method within 5 business days. 2. Step-back  Question search : None  Broader question: How are invoices issued and what do they include?  Broader search  : Invoices are issued monthly and list the plan fee and taxes. 3. Iterative, multi-step retrieval  Step 1 finds: The billing service is owned by the payments team. -> payments team  Step 2 finds: The payments team is managed by Dana Lee. -> Dana Lee  Final answer: the billing service team is managed by Dana Lee
The direct questions return nothing because they share no content words with the documents. Searching with the hypothetical answer or with the broader question finds the right passage. In the chain, the second search is only possible after reading the result of the first. The hypothetical answer and the step-back question were scripted here, while a real system would generate them with a model, which is where the extra cost and risk come from.

Knowledge Check

Q1. What is the hypothetical answer in HyDE used for?
Q2. When is iterative multi-step retrieval the right tool?
Q3. How do sub-question retrieval and multi-step retrieval differ?

Key Points

✓
Advanced strategies use a model to plan the search when one phrasing is not enough.
✓
HyDE searches with a hypothetical answer, which looks more like the target passages than the question does.
✓
A hypothetical answer is a search aid only and must never be treated as evidence.
✓
Step-back prompting also retrieves for a broader question to bring in background rules.
✓
Sub-question retrieval handles independent parts in parallel; multi-step retrieval handles dependent parts in sequence.
✓
Iterative retrieval needs stop conditions, a step limit and a trace of every query.
✓
Add these strategies only for query types where measurements show a single search fails.

The Big Picture

These strategies move part of the thinking from answer time to search time. They can unlock questions that plain retrieval cannot touch, at the price of latency, cost and new failure modes, so treat them as targeted tools rather than defaults.
Understand the question → Plan the searches → Retrieve → Check progress → Repeat or stop
Up Next: Context Construction and Grounded Generation. You will learn how to select, order and compress retrieved chunks, build a grounded prompt, and handle missing or conflicting information.