Document Chunk Splitter¶
The Document Chunk Splitter breaks large documents into smaller pieces that can be embedded and searched individually. Place it after a document source and before the vector store to control how much context each search result contains.
- Category: Text Splitter
- Purpose: Splits documents into smaller, searchable chunks
- Inputs:
documents- Documents to split - Outputs:
chunks- Resulting chunksstats- Split statisticspreview- Sample of the outputmetadata_report- Metadata summary- Configuration:
- Chunk Size: 100–10000, default 1000. Target characters per chunk. Not a hard limit — the splitter looks for a natural break point nearby
- Overlap: 0–5000, default 200. Characters repeated from the previous chunk so sentences are not cut in half. Ten to twenty percent of the chunk size works well
- Separator: Default
\n\n, meaning paragraph boundaries - Keep Separator:
trueorfalse. Defaulttrue, which preserves paragraph spacing and keeps the text readable - Length Function:
len(character count),tokenizer(token count),custom. Token counting is more accurate when managing a model's context window - Use Regex Separator:
falseortrue. Treats the separator as a regular expression. Useful for splitting on patterns such as numbered clauses