RAG Chunking Strategies: Size, Overlap, and Metadata

Vocabulary flashcards covering chunk sizes, overlap trade-offs, storage calculations, and metadata types in RAG systems.

Small Chunks ($100$ to $300$ tokens)

A chunk size configuration that yields high retrieval precision and higher recall, but loses context due to idea fragmentation.

Large Chunks ($500$ to $1,500$ tokens)

A chunk size configuration that preserves more context and requires fewer chunks for storage, but suffers from lower precision and introduces more noise per chunk.

General Sweet Spot Chunk Size

A baseline guideline of approximately $400$ to $800$ tokens that balances precision, context, and storage across general retrieval use cases.

Q&A Chunk Size

An optimal chunk size range of $200$ to $400$ tokens designed to yield precise answers in question-and-answer retrieval systems.

Summarization Chunk Size

A recommended chunk size range of $500$ to $1,000$ tokens, chosen because summarization tasks require full context.

Code Chunk Size

A recommended chunk size range of $300$ to $500$ tokens, balancing high precision with structural code context.

Legal and Technical Document Chunk Size

A recommended baseline range of $400$ to $800$ tokens, selected because content in these documents is typically clause-sized.

Chunk Overlap

The repetition of tokens across adjacent chunk boundaries designed to prevent information loss and avoid splitting ideas across split points.

Storage Increase Formula

The calculation for extra vector storage costs resulting from chunk overlap: $\frac{\text{chunks with overlap}}{\text{chunks without overlap}} - 1$.

Rule of Thumb for Overlap Storage

A large-document heuristic stating that a $10\%$ overlap increases storage by $\approx 11\%$, a $20\%$ overlap by $\approx 25\%$, and a $50\%$ overlap doubles storage ($2\times$ or $100\%$ increase).

Boundary Effect

The reason small documents experience lower overlap storage penalties than the rule of thumb, caused by the first chunk covering tokens without overlap and fixed boundary savings representing a larger fraction of total chunks.

Recommended Overlap Percentage

A standard baseline recommendation of $10\%$ to $20\%$ overlap of the chunk size (such as $20$ to $40$ tokens for a $200$-token chunk).

Structural Metadata

Metadata describing where a chunk resides in a document, including source name, file path, section, subsection, and positional index (e.g., chunk $N$ of $M$).

Semantic Metadata

Metadata describing what a chunk is about, including model-generated summaries, keywords, and named entities.

Retrieval Metadata

Metadata used for chunk indexing and search operations, such as vector embeddings and creation timestamps (created_at).

Pre-filtering (Filtering Before Search)

Using metadata attributes to narrow down candidate chunks prior to vector similarity matching, avoiding full scans across all chunks.

Hybrid Retrieval

A retrieval strategy that combines semantic search with keyword and metadata filtering to produce more precise matches.

Source Attribution

The capability of a large language model to reference exact document sources and sections (e.g., policy document and section number) using attached chunk metadata.