Why Does My Docs Chatbot Return Half a Code Example?

Why does my docs chatbot return half a code example?

The page was cut into chunks by length before it was embedded, and a cut landed inside a code block. Search then finds the chunk holding the first half, and the model answers from what it was given. A splitter that reads markdown structure keeps each fenced block in one chunk and labels it with the headings above it, so the code arrives whole and the chatbot knows which section it came from.

We ran six common splitter settings over the 823 pages of Mixpeek's public documentation on 2026-10-20. Splitting by length cut between 10 and 27 percent of the 1,539 code blocks. Splitting by headings kept the blocks together and stripped their indentation. Splitting by markdown structure cut none.

What happens when the answer stops mid-function?

Most RAG tutorials split text into windows of a fixed size, often 1,000 characters with 200 characters of overlap. The splitter has no idea a code block has started. If a 40-line example begins 700 characters into a window, the window ends partway through it and the rest lands in the next chunk. Overlap repeats the last 200 characters in the next chunk, which rescues a sentence that straddles the boundary. A code block longer than the overlap still ends up in two pieces, and each piece is retrieved and ranked on its own.

Two more things go wrong at the same time:

The code loses its context. The chunk holding the snippet often has no heading in it, so the

chatbot cannot tell whether it configures the Python SDK, the REST API or a CLI.

Whitespace gets normalized. Some splitters strip or collapse whitespace. Prose survives that.

Python and YAML do not, because their indentation is the syntax.

When the model receives half an example, it writes the other half itself. Ethan Steininger, Mixpeek's founder, puts the general problem this way: "LLMs alone hallucinate over unstructured data."

How often does each splitter cut a code block?

First-party measurement, run 2026-10-20 on every .mdx page of the Mixpeek docs.



How we counted. The corpus is 823 .mdx files, 189 of which contain code: 1,539 non-empty fenced blocks with a median of 11 lines. A block counts as cut when no single chunk contains its full text, and as "indentation lost" when a chunk contains it only after whitespace is collapsed. LangChain is langchain-text-splitters 0.3.8 with its default separators. The Mixpeek rows run the production chunker code.

106 of the blocks are longer than 1,000 characters, so no 1,000-character window could hold them. Leaving those out, the recursive splitter still cut 52 of the 1,433 blocks that would have fit (3.6%), and fixed windows cut 305 (21.3%). Two of our own modes are in the table on purpose: the character and word modes cut code too, which is why the markdown mode exists.

How do I keep code blocks whole when I chunk markdown?

1. Split on structure first. Break the page at headings and at fences, and treat each fenced code block as one unit that is never split. 2. Put the heading path on every chunk, such as "Retrievers > Stages > Deduplicate", as text at the top of the chunk or as metadata you can filter on. 3. Split long prose at paragraph boundaries, and repeat the section heading at the top of each part so every chunk reads on its own. 4. Decide what happens to very long blocks. Keep them whole if your embedding model's input limit fits them (about 512 tokens for many BERT-style models, about 8,000 for OpenAI's text-embedding-3 models). Past that, split at blank lines between functions and repeat the heading in each part. 5. Never collapse whitespace inside code.

At query time, a "how do I" question is best answered by a code chunk plus the prose section above it. Because both share a heading path, you can fetch the neighbour by that field after retrieval.

Can LangChain or LlamaIndex do this?

Partly. LangChain's MarkdownHeaderTextSplitter splits on headings and records them as metadata. In version 0.3.8 with default settings it strips leading whitespace from each line, so check indented code after splitting. A common pattern runs a recursive splitter over long sections afterwards, and that step can cut code again. The language-aware splitter prefers markdown boundaries and still cut 9.9% of our blocks at 1,000 characters. LlamaIndex ships a MarkdownNodeParser that also splits on headings. Whichever you use, run the check below on your own pages.

How do I check my own index for cut code blocks?

Run this over each source page and the chunks you produced from it. Any block it returns was cut.

import re

FENCE = re.compile(r"^ {0,3}(`{3,}|~{3,})")

def code_blocks(markdown):
    blocks, lines, i = [], markdown.split("\n"), 0
    while i < len(lines):
        opened = FENCE.match(lines[i])
        if not opened:
            i += 1
            continue
        marker, body, i = opened.group(1), [], i + 1
        while i < len(lines) and not lines[i].strip().startswith(marker):
            body.append(lines[i])
            i += 1
        i += 1
        if "".join(body).strip():
            blocks.append("\n".join(body))
    return blocks

def cut_blocks(markdown, chunks):
    return [b for b in code_blocks(markdown) if not any(b in c for c in chunks)]

To catch whitespace loss as well, compare with " ".join(b.split()) against the same normalization of each chunk: a block found only that way kept its words and lost its indentation.

Doing it in Mixpeek

Mixpeek reads your files where they already live, in S3, GCS or any bucket, and turns what's in them into data your software can search, classify and moderate. For documentation, set split_by to "markdown" in the text extractor:

{
  "feature_extractor": {
    "feature_extractor_name": "text_extractor",
    "version": "v1",
    "input_mappings": { "text": "page_markdown" },
    "parameters": {
      "split_by": "markdown",
      "chunk_size": 300
    }
  }
}

Each heading section becomes one document and each fenced code block another. Code is never split. Every chunk carries heading_path (outermost heading first, joined with " > "), chunk_kind ("section" or "code") and, for code, code_language from the fence. Section prose splits at paragraph boundaries once it passes chunk_size words, and each part keeps its heading. Front matter is dropped and its title becomes the outermost heading. The ColBERT extractor accepts the same mode.

Who it fits: teams with docs, runbooks or knowledge bases in markdown that already live in a bucket. Who it does not: if your docs fit in a single prompt, skip retrieval and send the page. Build starts at $25 a month for up to 1M vectors (pricing).

Frequently Asked Questions

What chunk size should I use for documentation with code?

Where the cuts fall matters more than the size. Split at headings and fences first, keep prose parts around 200 to 400 words, and let each code block be its own chunk at whatever length it has, up to your embedding model's input limit.

Does chunk overlap stop code blocks from being split?

Only for blocks shorter than the overlap. With 200 characters of overlap, LangChain's recursive splitter still cut 158 of 1,539 code blocks in our run, 10.3%.

Why does my chatbot make up the rest of the code?

It was given half of it. The model fills the gap with something plausible, which is how a correct first half ends in an API call that does not exist.

How does the chatbot know which API a snippet belongs to?

From the headings above the snippet. Store the heading path on every chunk, for example "Python SDK > Collections > Create", so a retrieved code block carries its section with it.

What is a fenced code block?

A block of code opened by a line of at least three backticks or tildes and closed by a matching line, as defined in the CommonMark spec. The text after the opening fence, such as "python", names the language.

Related

Multimodal chunking strategies

Why does my document search miss the answer that is in a chart or table?

Best document parsing tools for RAG

Best rerankers for RAG