Query Dependent Chunking

Query-Dependent RAG Chunking

Skip to Main Menu

Skip to Main Content

Skip to Footer

Back to Blog

-->

Back to Blog

TL;DR

Different queries need different chunk sizes, but RAG systems commit to one size upfront. In this blog, we show that indexing the same corpus at multiple chunk sizes (e.g., 100, 200, 500 tokens) and aggregating results with Reciprocal Rank Fusion (RRF) improves retrieval by 1–37% across benchmarks, without retraining models. Oracle experiments reveal 20-40% headroom when selecting optimal chunk size per query. [ See it in code ]

Is chunk size query-dependent?

Every RAG system faces the same chunking dilemma: smaller chunks preserve fine-grained details but lose context; larger chunks capture broader meaning but dilute specific facts. The conventional solution is to find a “sweet spot” chunk size (typically 500-800 tokens) that performs well on average.

But what if there is no universal sweet spot? What if different queries fundamentally need different chunk sizes?

This tradeoff is fundamental to fixed-dimensional embeddings. As chunk size grows, a fixed-size vector must compress increasingly diverse information, inevitably losing fine-grained details. Conversely, smaller chunks preserve specificity but sacrifice the surrounding context needed to understand meaning.

This raises a natural question: Is there a single chunk size that serves all queries equally well?

As we show below, the answer is no.

To investigate this, we evaluated retrieval performance across multiple chunk sizes and datasets. The results reveal a consistent pattern: the best-performing chunk size varies substantially across queries, even when querying the same corpus.

This observation motivates a shift in perspective. Instead of committing to a single segmentation strategy, we ask whether multiple representations of the same text can be combined in a principled way to improve retrieval robustness.

Prior work

Several approaches have tackled this problem. Anthropic’s contextual retrieval enriches chunks with document-level context. Jina AI’s late chunking captures both coarse and fine-grained signals by chunking in the latent space. RAPTOR builds hierarchical summaries over fixed-size chunks.

However, all these methods still commit to a fixed chunk size. They also add complexity: some require retraining embedding models; others use LLMs to generate synthetic context, potentially introducing noise.

In contrast, our approach keeps chunking simple and instead exposes multiple resolutions at retrieval time.

The hypothesis

Standard practice treats chunk size as a one-time tuning parameter: choose a value, measure average performance, iterate until satisfied, then deploy. This optimization targets the mean. But averages can be misleading.

If queries truly have different optimal chunk sizes, then optimizing for average performance necessarily means underperforming on many individual queries. The question becomes: how much performance are we leaving on the table?

This, combined with the inherent limitations of embedding models , led us to the following hypothesis:

Different queries benefit from different chunk sizes, and selecting an appropriate chunk size at inference time can substantially improve retrieval performance.

Before proposing a practical method, we first test whether this hypothesis holds empirically.

Oracle experiments: is chunk size query-dependent?

To isolate the effect of chunk size, we ran a controlled experiment across several retrieval benchmarks.

Experimental setup

We evaluated on three diverse benchmarks:

– QMSum : Meeting transcripts with queries about specific discussion points. Documents average 5,000+ tokens with information distributed across speakers and topics.

– NarrativeQA : Full-length stories and book chapters where queries require understanding interconnected plot elements at varying levels of detail.

– Seinfeld (custom) : Trivia questions over TV episode transcripts, testing retrieval of both specific facts and broader contextual understanding.

These datasets span structured meeting notes, narrative fiction, and conversational dialogue, representing different information distributions and query types.

For each dataset, we:

1. Created multiple indices of the same corpus, each with a different sliding-window chunk size (e.g., 50, 100, 200, 500, 1000, 2000 tokens).

2. For each query, retrieved top-k chunks independently from each index.

3. Evaluated document-level recall@K: did any chunk from the correct document appear in the top-k results? This metric serves as a proxy for downstream success: retrieval must first surface the correct document before any generation step can succeed.

4. Introduced an oracle aggregator that, for each query, selects whichever chunk size achieved the highest recall (assuming access to ground truth).

The oracle represents an upper bound: if we could perfectly predict the best chunk size per query, what performance could we achieve?

Results

The results strongly support our hypothesis. Across all datasets, a consistent pattern emerges:

Figure 1: Chunk size performance vs. oracle upper bound. Oracle’s superior performance across all K values and across all datasets confirms that optimal chunk size varies by query.

Figure 1: Chunk size performance vs. oracle upper bound. Oracle’s superior performance across all K values and across all datasets confirms that optimal chunk size varies by query.

Key findings:

As can be seen from the chart, no single chunk size dominates: the “Oracle” is better than any single chunk size. This supports the notion that what works for one question often fails for another.

The oracle substantially outperforms all fixed choices. Gaps of 20-30% in recall@1 are common, reaching 40%+ in some datasets.

This headroom is systematic, not dataset-specific. The pattern holds across very different text types: meeting transcripts (QMSum), TV scripts (Seinfeld), and narrative stories (NarrativeQA).

This confirms the hypothesis: optimal chunk size is strongly query-dependent , and the choice materially affects retrieval quality.

The oracle experiment is deliberately unrealistic. At inference time, we do not know which chunk size will work best for a given query.

This leads to a practical question: Can we approximate the oracle without explicitly predicting chunk size, retraining embedding models, or introducing query-specific...