HyDE in RAG: When the Query Doesn't Say What You're Looking For

HyDE in RAG: When the Query Doesn't Say What You're Looking For

Let me start with a confession: for a long time I thought retrieval in a RAG system was "just" embeddings and cosine similarity. Embed the query, fetch the closest chunks, done. It works… until the query is short, ambiguous, or doesn't even share vocabulary with the documents.

That's where HyDE entered my stack. I implemented it as part of my RAG course, and this post is the summary of what I learned, with the actual code from the notebook.

What is HyDE (Hypothetical Document Embeddings)

HyDE (Hypothetical Document Embeddings) is a retrieval technique that flips the question around. Instead of embedding the user's query directly and searching with it, you first ask an LLM to generate a hypothetical document: a plausible answer to that query.

Then you embed that document — not the query — and use it to search your vector store. The idea is that the embedding of a complete answer is usually closer to the relevant documents than the embedding of a short question.

def get_hyde_doc(query):
    template = """Imagine you are an expert writing a detailed explanation on the topic: '{query}'
    create a hypothetical answer for the topic"""
    system_message_prompt = SystemMessagePromptTemplate.from_template(template=template)
    chat_prompt = ChatPromptTemplate.from_messages([system_message_prompt])
    messages = chat_prompt.format_prompt(query=query).to_messages()
    response = llm.invoke(messages)
    return response.content
Copy code

Why it works

HyDE builds a bridge between user intent and relevant content. In my experience it shines in three scenarios:

1. Short queries: "Steve Jobs fired" says very little; a hypothetical document develops the idea and gives the embedding context.
2. Language or vocabulary mismatch: the question uses one set of words, the document another. The hypothetical document acts as a semantic translator.
3. Retrieving by answer content: you want documents that answer the question, not documents that contain its exact words.

Step-by-step implementation

The flow I built in the notebook has five phases:

1. Load and split the data

I used WikipediaLoader with the Steve Jobs page and RecursiveCharacterTextSplitter with 300-token chunks and 100 overlap:

loader = WikipediaLoader(query="Steve Jobs", load_max_docs=5)
documents = loader.load()

text_splitter = RecursiveCharacterTextSplitter(chunk_size=300, chunk_overlap=100)
docs = text_splitter.split_documents(documents=documents)
Copy code

2. Build the vector store

I embedded the chunks with HuggingFace's all-MiniLM-L6-v2 and stored them in Chroma (I also tried FAISS):

embeddings = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")
db = Chroma.from_documents(documents=docs, embedding=embeddings, persist_directory="output/steve_jobs_for_hyde.db")
base_retriever = db.as_retriever(search_kwargs={"k": 5})
Copy code

3. Configure the LLM

To generate the hypothetical document I used a Groq model, gemma2-9b-it, via LangChain's init_chat_model:

llm = init_chat_model("groq:gemma2-9b-it")
Copy code

4. Retrieve with HyDE

The moment of truth: instead of searching with the query, I search with the generated hypothetical document.

query = "When was Steve Jobs fired from Apple?"
hypo_doc = get_hyde_doc(query=query)
matched_doc = base_retriever.invoke(hypo_doc)
Copy code

The results improved noticeably compared to searching with the raw query, especially on questions where the query words barely appeared in the chunks.

The LangChain version: HypotheticalDocumentEmbedder

LangChain already ships this technique as HypotheticalDocumentEmbedder, and the best part is that it plugs in like any other embedding function. Here's how I used it with the default web search prompt:

from langchain.chains.hyde.base import HypotheticalDocumentEmbedder

hyde_embedding_function = HypotheticalDocumentEmbedder.from_llm(
    llm=llm,
    base_embeddings=base_embeddings,
    prompt_key="web_search"
)
Copy code

One detail I liked: it bundles several ready-made prompts in PROMPT_MAP, tuned for different domains:

• web_search
• sci_fact
• arguana
• trec_covid
• fiqa
• dbpedia_entity
• trec_news
• mr_tydi

If none of them fits your domain, you can pass your own prompt with custom_prompt.

The full RAG pipeline

With the HyDE embedder I built the whole RAG pipeline: I index the dataset with those embeddings, retrieve with similarity_search, and generate the answer with create_stuff_documents_chain:

vectorstore = Chroma.from_documents(
    documents=chunks,
    embedding=hyde_embedding_function,
    persist_directory="output/langchain"
)

rag_chain = create_stuff_documents_chain(llm=llm, prompt=rag_prompt)

def hyde_rag_pipeline(query):
    matched_docs = vectorstore.similarity_search(query, k=4)
    return rag_chain.invoke({"input": query, "context": matched_docs})
Copy code

With a query like "What memory modules does LangChain provide?" the pipeline retrieved relevant context that classic search was missing.

Custom prompt

If the generic prompt isn't for you, you can define your own. I tried a more direct one:

custom = PromptTemplate.from_template(
    "Generate a concise hypothetical answer for this topic: {query}"
)

hyde_embedding_function = HypotheticalDocumentEmbedder.from_llm(
    llm=llm,
    base_embeddings=base_embeddings,
    custom_prompt=custom
)
Copy code

When to use HyDE (and when not to)

My honest recommendation:

• Use it if your users ask short questions, if your documents are in another language or register, or if keyword-based search keeps failing you.
• Think twice if latency matters a lot or token cost is a concern: every query means an extra LLM call.
• Always measure: HyDE depends on LLM quality and can hallucinate content. Before shipping it to production, compare it against your baseline with a set of real questions.

Conclusion

HyDE isn't magic — it's an elegant way to move LLM understanding into the embedding space. If you're building a RAG system and notice it retrieves poorly on short or ambiguous queries, trying HyDE should be on your list before touching chunking or switching embedding models.

In my RAG course I treat it as one of the essential query enhancement techniques, alongside query expansion and query decomposition. If you want to take it to a real project, I can help you design the retrieval and evaluate it with data from your own domain.