How you split decides what you find

The retrieval decision nobody reviews in design docs: where to cut.

Documents get split into chunks before embedding, and that split decides everything downstream. Cut every 512 tokens and you slice sentences in half, so the embedding captures half a thought. Cut on topic shifts and every chunk is coherent but the sizes swing wildly. Cut on document structure (headings, tables, code blocks) and chunks match how the author actually organized meaning.

Here's the part that should bother you: a study this month found plain fixed-size chunks beating clever semantic chunking on real benchmarks. The sophisticated option isn't automatically the better one. It depends on your documents, and the only way to know is to measure.

We made chunking a configurable stage in Mixpeek rather than a buried constant, because we kept seeing the same failure: teams tune models for months while a bad splitter quietly caps their ceiling.

Your retrieval can't find what your chunker cut in half.