agrag.chunking.chunker.Chunker

Chunker(size:int = DEFAULT_SIZE, min_size:int | None = None, tokenizer:str = DEFAULT_TOKENIZER) -> None

Packs the sections of a Document into chunks.

The chunker walks the sections in reading order. It packs the units of a section into a chunk until the next unit would pass size tokens. It splits a unit that is over size on its own, at paragraph, sentence, clause or word boundaries. When a section ends and the open chunk holds fewer than min_size tokens, the chunk goes on into the next section. A table never mixes with text: it becomes one chunk, or row groups that each repeat the header row.

A chunk joins the text of its pieces with a blank line. Size counts that joined text: every piece plus every blank line between pieces. A document whose units all carry text offsets gives text provenance over the span of its units. A document whose units carry no offsets gives page provenance.

Attributes:

• size (int) – The most tokens in a chunk.
• min_size (int | None) – A chunk with fewer tokens than this goes on into the next section. None, the default, means a quarter of size.
• tokenizer (str) – The tokenizer that counts tokens. A name that chonkie accepts.

Functions:

• chunk – Split a document into chunks with their placements.
• count_tokens – Return the number of tokens in a text, counted with tokenizer.
• settings – Return the settings as plain data.
• split – Split a text into pieces of at most size tokens.

chunk

chunk(document:Document) -> ChunkedDocument

Split a document into chunks with their placements.

Parameters:

• document (Document) – The document to split.

Returns:

• ChunkedDocument – The chunks in reading order with one placement per chunk.

Raises:

• ChunkingError – The document mixes units that carry text offsets with units that carry none, or the splitter changed or dropped text.

count_tokens

count_tokens(text:str) -> int

Return the number of tokens in a text, counted with tokenizer.

effective_min_size

effective_min_size: int

Return the merge threshold: min_size, or a quarter of size.

fingerprint

fingerprint: str

Return the hash of the settings. Equal settings give equal hashes.

min_size

min_size: int | None = None

settings

settings() -> dict[str, Any]

Return the settings as plain data.

The splitter and table rendering versions are included, so an upgrade that changes the chunks also changes the fingerprint.

size

size: int = DEFAULT_SIZE

split

split(text:str) -> list[TextPiece]

Split a text into pieces of at most size tokens.

Parameters:

• text (str) – The text to split.

Returns:

• list[TextPiece] – The pieces in order. Each piece's text is a slice of the input text.

Raises:

• ChunkingError – A piece does not match its slice, or the pieces do not join back into the text.

tokenizer

tokenizer: str = DEFAULT_TOKENIZER