# `Arcana.Chunker`
[🔗](https://github.com/georgeguimaraes/arcana/blob/main/lib/arcana/chunker.ex#L1)

Behaviour for text chunking providers used by Arcana.

Arcana accepts any module that implements this behaviour.
Built-in implementations are provided for:

- `Arcana.Chunker.Default` - Default chunking using text_chunker library

## Configuration

Configure your chunking provider in `config.exs`:

    # Default: text_chunker-based chunking
    config :arcana, chunker: :default

    # Default chunker with custom options
    config :arcana, chunker: {:default, chunk_size: 512, chunk_overlap: 100}

    # Custom function
    config :arcana, chunker: fn text, opts -> [%{text: text, chunk_index: 0, token_count: 10}] end

    # Custom module implementing this behaviour
    config :arcana, chunker: MyApp.SemanticChunker
    config :arcana, chunker: {MyApp.SemanticChunker, model: "..."}

## Implementing a Custom Chunker

Create a module that implements this behaviour:

    defmodule MyApp.SemanticChunker do
      @behaviour Arcana.Chunker

      @impl true
      def chunk(text, opts) do
        # Custom chunking logic...
        # Return list of chunk maps
        [
          %{text: "chunk 1", chunk_index: 0, token_count: 50},
          %{text: "chunk 2", chunk_index: 1, token_count: 45}
        ]
      end
    end

Then configure:

    config :arcana, chunker: {MyApp.SemanticChunker, model: "..."}

## Chunk Format

Each chunk returned must be a map with at minimum:

  * `:text` - The chunk text content (required)
  * `:chunk_index` - Zero-based index of this chunk (required)
  * `:token_count` - Estimated token count (required)
  * `:metadata` - Optional map stored on the chunk as-is

Any other key is folded into the stored metadata too. Metadata
round-trips through JSONB, so keys are stringified on the way in:
a chunk of `%{text: ..., chunk_index: 0, token_count: 8, page: 3}`
reads back as `metadata["page"] == 3`.

`Arcana.Chunker.Default` uses this to report each chunk's
`"start_byte"`/`"end_byte"` range in the source text.

# `chunk`

```elixir
@callback chunk(text :: String.t(), opts :: keyword()) :: [map()]
```

Splits text into chunks.

Returns a list of chunk maps, each containing at minimum `:text`,
`:chunk_index`, and `:token_count`.

## Options

Options are implementation-specific. Common options include:

  * `:chunk_size` - Maximum chunk size
  * `:chunk_overlap` - Overlap between chunks
  * `:format` - Text format hint (`:plaintext`, `:markdown`, etc.)

# `chunk`

Chunks text using the configured chunker.

The chunker is a `{module, opts}` tuple where module implements
this behaviour.

# `chunk`

Chunks text using the configured chunker, merging additional options.

Useful when you need to override chunker defaults at call time.

# `metadata_for`

Builds the metadata to store for a chunk map.

A chunker's own `:metadata` map is the canonical place for extra keys
(that's what `Arcana.Chunker.Default` uses for its byte offsets), and
anything else it hands back is folded in too, as the behaviour
promises. Keys are stringified since chunk metadata round-trips
through JSONB; a declared `:metadata` entry wins over a top-level one.

Every path that inserts chunks goes through here, so a document
recovered by `Arcana.Maintenance.reembed/2` carries the same metadata
the original ingest would have given it.

---

*Consult [api-reference.md](api-reference.md) for complete listing*