Skip to content

Document Chunk Splitter

The Document Chunk Splitter breaks large documents into smaller pieces that can be embedded and searched individually. Place it after a document source and before the vector store to control how much context each search result contains.

  • Category: Text Splitter
  • Purpose: Splits documents into smaller, searchable chunks
  • Inputs: documents - Documents to split
  • Outputs:
  • chunks - Resulting chunks
  • stats - Split statistics
  • preview - Sample of the output
  • metadata_report - Metadata summary
  • Configuration:
  • Chunk Size: 100–10000, default 1000. Target characters per chunk. Not a hard limit — the splitter looks for a natural break point nearby
  • Overlap: 0–5000, default 200. Characters repeated from the previous chunk so sentences are not cut in half. Ten to twenty percent of the chunk size works well
  • Separator: Default \n\n, meaning paragraph boundaries
  • Keep Separator: true or false. Default true, which preserves paragraph spacing and keeps the text readable
  • Length Function: len (character count), tokenizer (token count), custom. Token counting is more accurate when managing a model's context window
  • Use Regex Separator: false or true. Treats the separator as a regular expression. Useful for splitting on patterns such as numbered clauses