# `Arcana.Chunker.Default`
[🔗](https://github.com/georgeguimaraes/arcana/blob/main/lib/arcana/chunker/default.ex#L1)

Default text chunker using the text_chunker library.

Supports multiple formats (plaintext, markdown, etc.) and can size chunks
by characters or tokens.

## Options

  * `:chunk_size` - Maximum chunk size (default: 450)
  * `:chunk_overlap` - Overlap between chunks (default: 50)
  * `:format` - Text format: `:plaintext`, `:markdown`, `:elixir`, etc. (default: :plaintext)
  * `:size_unit` - How to measure size: `:characters` or `:tokens` (default: :tokens)
  * `:chars_per_token` - Characters assumed per token when `size_unit` is
    `:tokens` (default: 4). A positive integer; anything else raises an
    `ArgumentError`
  * `:max_chunk_chars` - Ceiling on a chunk's length in bytes. A chunk over
    it is split rather than emitted (default: none). Splits land on
    grapheme boundaries, so a single grapheme longer than the budget - a
    ZWJ emoji sequence runs to 25 bytes - is emitted whole rather than
    cut into invalid UTF-8. Any budget worth setting is far above that.
    A positive integer or `nil`; anything else raises an `ArgumentError`

## Sizing by tokens is an estimate, not a guarantee

With `size_unit: :tokens` the size is converted to characters by
multiplying by `:chars_per_token`, which defaults to 4. That is a fair
average for flowing English prose and **not an upper bound**. Dense text
runs closer to 3 characters per token, and tables of numbers can approach
2, so the default 450 tokens can produce chunks an embedder with a 512
token window rejects:

    413 Input validation error: 'inputs' must have less than 512 tokens. Given: 573

Nothing here can tell you that has happened, because the chunker does not
run the model's tokenizer. Two ways to stay inside the budget:

  * lower `:chars_per_token` for a corpus you know is dense — product data
    sheets, code, chemical names, digit-heavy tables
  * set `:max_chunk_chars` as a backstop, so a pathological chunk is split
    instead of being handed to the embedder oversized

Remember any prefix your embedder adds (E5-style `passage: ` / `query: `)
comes out of the same budget.

## Examples

    Arcana.Chunker.Default.chunk("Hello world", chunk_size: 100)
    Arcana.Chunker.Default.chunk(markdown_text, format: :markdown, chunk_size: 512)
    Arcana.Chunker.Default.chunk(text, size_unit: :tokens, chunk_size: 256)

---

*Consult [api-reference.md](api-reference.md) for complete listing*