Naive RAG embeds the user's exact query and searches directly. This works for simple questions but fails for complex ones that require information from multiple sources, ambiguous phrasings that do not match the corpus vocabulary, or multi-step reasoning. Query decomposition and rewriting transform the user's original question into one or more search-friendly queries before retrieval, dramatically improving coverage and recall for complex cases.

Why Naive Retrieval Fails on Complex Queries

Consider the question: "How does the latest version of our authentication service compare to the old one in terms of token expiration and session management?" Naive RAG embeds this whole question and searches for similar chunks. The problem is that no single document in your knowledge base is likely to cover both the latest version and the old version, both token expiration and session management. The retrieval returns chunks that are mildly relevant to several aspects of the query but miss the specifics of each.

Multi-part questions, comparisons, and multi-hop reasoning all suffer from this issue. The embedding of a compound question lands in an average of its component topics, which is not where the actual answers live.

Query Decomposition

Query decomposition breaks a complex query into simpler sub-queries that can be answered independently. The pipeline:

1. An LLM receives the original query and produces a list of sub-queries.
2. Each sub-query is run through the standard retrieval pipeline independently.
3. The results from all sub-queries are combined and passed to the LLM for final synthesis.

For the authentication example, decomposition might produce:

• "latest version of authentication service token expiration"
• "latest version of authentication service session management"
• "old version of authentication service token expiration"
• "old version of authentication service session management"

Each sub-query finds specific relevant chunks. The final LLM answer then draws from all of them to produce the comparison.

Decomposition works best on questions that naturally split into parts, comparisons, multi-entity queries, and questions with multiple aspects. It adds latency (multiple retrieval calls) but dramatically improves coverage for these query types.

Query Rewriting

Query rewriting is simpler than decomposition: transform the user's query into a form better suited for retrieval without splitting it. Common rewrite patterns:

1. Expansion: Add synonyms or related terms. "How do I fix connection errors?" becomes "How do I fix connection errors, timeouts, network failures, or authentication problems?"
2. Specification: Make vague queries more specific. "Why is my app slow?" becomes "Why is my application experiencing high latency in database queries, API responses, or UI rendering?"
3. Formalization: Convert conversational queries to more document-like phrasings. "can u help me with login" becomes "authentication troubleshooting and login issue resolution".
4. Historical context injection: For multi-turn conversations, rewrite the current query to include context from previous turns. "What about the second one?" becomes "What about the second configuration option mentioned earlier?"

Rewriting is faster than decomposition (only one retrieval call) and addresses query-vocabulary mismatch issues effectively. It is the right choice when queries are single-topic but phrased poorly for retrieval.

HyDE: Hypothetical Document Embeddings

HyDE (Hypothetical Document Embeddings) is a clever variant of query rewriting. Instead of rewriting the query, HyDE asks the LLM to generate a hypothetical answer to the query, then uses the embedding of that hypothetical answer for retrieval.

The intuition: document embeddings are trained to be similar to other documents, not to questions. A query like "what is X?" does not embed near documents about X because documents about X do not usually start with "what is X?". But a hypothetical answer to the query ("X is a...") embeds right next to real documents about X.

HyDE workflow:

1. Pass the query to the LLM with instruction: "Write a short paragraph that would answer the following question."
2. Embed the generated paragraph.
3. Use that embedding for retrieval.
4. Return the retrieved documents as usual.

HyDE adds an LLM call per query (the hypothetical generation) but improves retrieval on domain-specific queries where embedding models have not been trained on query-document pairs. It is particularly effective for RAG over technical documentation.

When to Use Each Technique

Guidance:

• Query rewriting: Default starting point for improving beyond naive RAG. Low cost, modest improvements, handles most query-vocabulary mismatch issues.
• HyDE: Use when queries are very different in style from documents (formal vs casual, question vs explanation). Adds one LLM call per query.
• Decomposition: Use for genuinely multi-part queries. Adds multiple retrieval calls and synthesis overhead but critical for multi-hop questions.
• All three: Production systems often combine them. Use rewriting as default, detect multi-part queries and apply decomposition, use HyDE for specific query types identified by a classifier.