Four Chunking Strategies, Same Document

Chunking is the highest impact, lowest visibility decision in a retrieval pipeline.

How you split a document decides what your search can ever find. Take one document, chunk it four different ways, and you get four different search experiences.

Fixed-size, say 512 tokens, is simple and predictable. It also slices through the middle of sentences, cuts code blocks in half, and breaks tables apart. Your embedding ends up holding half a thought.

Semantic chunking splits on topic shifts, detected by embedding similarity. Every chunk is coherent and captures one complete idea. The sizes swing hard, though. Some come out at 100 tokens, others at 2,000.

Hierarchical chunking nests small child chunks inside large parent chunks. You search the children for precision and pull the parent for context, so you get fine-grained matching without losing the surrounding meaning. The price is a more complex index.

Document-aware chunking splits on headings and keeps code blocks and tables intact. The chunks follow the document's own structure. A table stays a table, a function stays whole. It just needs format-specific parsing.

Rough guide: fixed-size for homogeneous text or a fast prototype, semantic for long-form articles and papers, hierarchical for Q&A and RAG where context carries the answer, document-aware for technical docs full of code and tables.

Most RAG failures aren't model failures. They're chunking failures.