Bridging the Zero-Shot Gap: A Deep Dive into Hypothetical Document Embeddings (HyDE)
Bridging the Zero-Shot Gap: A Deep Dive into Hypothetical Document Embeddings (HyDE)
In the world of Information Retrieval (IR), we face a persistent challenge known as the "Zero-Shot Gap."
Imagine trying to match a short, punchy question like "How do plants make food?" to a dense, academic paragraph about photosynthesis. These two pieces of text look entirely different structurally and linguistically, even though they are semantically identical. This is an asymmetric retrieval problem: the query is a request for information, while the document is the information itself.
Enter HyDE (Hypothetical Document Embeddings). Instead of trying to force a square peg (a query) into a round hole (a document), HyDE uses the generative power of Large Language Models (LLMs) to create a "bridge."
The Core Intuition: From Asymmetric to Symmetric
The fundamental thesis of HyDE is simple: It is easier to match a document to a document than a query to a document.
Rather than encoding the user's query directly, HyDE asks an LLM to imagine what a perfect answer would look like. This "hypothetical document" may contain hallucinations or factual errors, but it possesses the semantic pattern and vocabulary of a real answer.
By transforming the query into a fake document, we turn an asymmetric search task into a symmetric document-to-document match, allowing contrastive encoders to operate in their "sweet spot."
The High-Level Workflow
The Technical Blueprint
The Mathematical Framework
HyDE moves through four distinct mathematical stages:
-
Hypothetical Generation: An LLM $g$ takes a query $q$ and an instruction $\text{INST}$ to produce a hypothetical document $\hat{d}$: $$\hat{d}{ij} = g(q{ij}, \text{INST})$$
-
Hypothetical Encoding: A contrastive encoder $f$ maps this hypothetical text into a dense vector space: $$\mathbf{v}{q{ij}} = f(\hat{d}_{ij})$$
-
Vector Search: We perform a Minimum Inner Product Search (MIPS) against the pre-computed embeddings of the real corpus $D$: $$\text{Retrieved Documents} = \text{arg max}{d \in D_i} \langle \mathbf{v}{q_{ij}}, f(d) \rangle$$
-
Similarity: The final ranking is determined by the cosine similarity between the hypothetical vector and the real document vectors: $$\text{Similarity}(q, d) = \text{enc}_q(q)^T \text{enc}_d(d)$$
Implementation Guide
Below is a production-ready conceptual implementation. While we use gpt2 and DPR for demonstration purposes, in a production environment, you would replace these with Llama-3/GPT-4 and Contriever/BGE-M3.
import torch
import torch.nn.functional as F
from typing import List, Tuple
from transformers import AutoTokenizer, AutoModel, pipeline
class HyDERetriever:
def __init__(self, llm_model_name: str = "gpt2", encoder_model_name: str = "facebook/dpr-ctx_encoder-single-nq-base"):
print(f"Loading LLM: {llm_model_name}...")
self.llm = pipeline("text-generation", model=llm_model_name, device=-1)
print(f"Loading Encoder: {encoder_model_name}...")
self.tokenizer = AutoTokenizer.from_pretrained(encoder_model_name)
self.encoder = AutoModel.from_pretrained(encoder_model_name)
self.encoder.eval()
def generate_hypothetical_document(self, query: str) -> str:
"""Step 1: Use LLM to imagine a relevant document."""
prompt = f"Write a detailed passage to answer the question: {query}\n\nAnswer:"
output = self.llm(
prompt,
max_new_tokens=100,
num_return_sequences=1,
truncation=True,
pad_token_id=self.tokenizer.eos_token_id
)
return output[0]['generated_text'].replace(prompt, "").strip()
def encode(self, text: str) -> torch.Tensor:
"""Step 2: Map text to a dense vector using the contrastive encoder."""
inputs = self.tokenizer(text, return_tensors="pt", padding=True, truncation=True, max_length=512)
with torch.no_grad():
outputs = self.encoder(**inputs)
# Use [CLS] token as the representation
embedding = outputs.last_hidden_state[:, 0, :]
return F.normalize(embedding, p=2, dim=1)
def retrieve(self, query: str, corpus: List[str], top_k: int = 2) -> List[Tuple[int, float]]:
"""Full HyDE Pipeline: Generate -> Encode -> Search."""
# 1. Generate hypothetical document
hypothetical_doc = self.generate_hypothetical_document(query)
# 2. Encode hypothetical document
query_emb = self.encode(hypothetical_doc)
# 3. Encode corpus (In production, use FAISS for this step)
corpus_embs = torch.cat([self.encode(doc) for doc in corpus], dim=0)
# 4. Compute Cosine Similarity
scores = torch.matmul(query_emb, corpus_embs.T).squeeze(0)
# 5. Rank
top_results = torch.topk(scores, k=min(top_k, len(corpus)))
return [(idx.item(), score.item()) for score, idx in zip(top_results.values, top_results.indices)]
# --- Execution ---
if __name__ == "__main__":
corpus = [
"The Great Wall of China is a series of fortifications built across northern borders.",
"Photosynthesis is a process used by plants to convert light energy into chemical energy.",
"The Eiffel Tower is a wrought-iron lattice tower in Paris, France.",
"Quantum entanglement occurs when particles share spatial proximity.",
"The Roman Empire was characterized by government headed by emperors."
]
hyde = HyDERetriever(llm_model_name="gpt2")
query = "How do plants make food?"
results = hyde.retrieve(query, corpus)
print(f"\nQuery: {query}")
for rank, (idx, score) in enumerate(results, 1):
print(f"{rank}. [Score: {score:.4f}] {corpus[idx]}")Key Takeaways & Analysis
Why this works (even with hallucinations)
One might worry that if the LLM generates a wrong answer, the retriever will find wrong documents. However, HyDE relies on semantic neighborhoods. Even if the LLM hallucinates a specific date or name, it will still use terms like "chlorophyll," "stomata," and "carbon dioxide" when answering a question about plants. These terms pull the embedding vector into the correct region of the vector space, where the real documents reside.
Summary of Contributions
- Zero-Shot Capability: Eliminates the need for expensive, query-document pair training data.
- Symmetry Transformation: Converts an asymmetric retrieval task into a symmetric one.
- LLM-Augmented Retrieval: Leverages the internal world knowledge of LLMs to improve the precision of dense retrievers.
Complexity Trade-off
| Feature | Standard Dense Retrieval | HyDE Retrieval |
|---|---|---|
| Latency | Low (Single Encoding) | Higher (LLM Gen + Encoding) |
| Accuracy | Moderate (Zero-shot gap) | High (Bridged gap) |
| Compute | Low | Moderate/High |
HyDE is a powerful tool for RAG (Retrieval-Augmented Generation) pipelines where accuracy is paramount and a slight increase in initial latency is an acceptable trade-off for significantly higher retrieval recall.