# `Arcana.Ingest`
[🔗](https://github.com/georgeguimaraes/arcana/blob/main/lib/arcana/ingest.ex#L1)

Document ingestion for Arcana.

Handles chunking, embedding, and storing documents with optional
GraphRAG entity/relationship extraction.

# `ingest`

Ingests text content, creating a document with embedded chunks.

## Options

  * `:repo` - The Ecto repo to use (required)
  * `:source_id` - An optional identifier for grouping/filtering
  * `:metadata` - Optional map of metadata to store with the document
  * `:chunk_size` - Maximum chunk size in characters (default: 1024)
  * `:chunk_overlap` - Overlap between chunks (default: 200)
  * `:collection` - Collection name (string) or map with name and description (default: "default")
  * `:graph` - Enable GraphRAG extraction (default: from config)
  * `:replace` - When true, atomically replaces any prior document with the
    same `(collection, source_id)` once the new ingest completes. Requires
    `:source_id`. See "Replacing documents" below.
  * `:on_chunk_error` - What a chunk that fails to embed does to the rest
    of the document: `:abort` (default) or `:skip`. See "When a chunk
    fails to embed" below.

## When a chunk fails to embed

Every chunk is embedded before anything is written, so a failure never
leaves a partly-stored document behind.

With the default `:abort`, one rejected chunk fails the whole ingest and
writes nothing:

    {:error, {:embedding_failed, %{chunk_index: 12, reason: reason}}}

The `chunk_index` is there so you can find the offending input instead of
guessing which of forty chunks the endpoint refused.

With `:skip`, the chunks that did embed are stored and the rest are
reported:

    {:ok, document, %{skipped_chunks: 1, reasons: [...], failed: [%{chunk_index: 12, reason: ...}]}}

For retrieval, 39 of 40 chunks beats losing the document. The reason is
also written to `document.error`, so a partial ingest is still explicable
from the row alone. Skipped chunks leave a gap in `chunk_index` rather
than renumbering, since a chunk's `"start_byte"`/`"end_byte"` metadata is
what locates it in the source.

If *every* chunk fails, `:skip` returns
`{:error, {:all_chunks_failed, failures}}` and writes nothing: a document
with no chunks is invisible to search but still counts in listings, which
is the state this option exists to avoid.

## Replacing documents

With `replace: true`, `source_id` becomes a stable document identity:
re-ingesting the same identity supersedes earlier documents instead of
accumulating next to them. The new document is ingested first and
predecessors (including `:failed`/`:processing` leftovers from crashed
attempts) are deleted only after it completes, so the old chunks stay
searchable until the replacement lands and their rows cascade away with
the document.

The swap runs in a transaction under a per-identity advisory lock, so
callers don't need their own mutex for correctness. If two ingests for the
same identity run concurrently, the first to complete wins and the other
returns `{:error, :replaced_by_concurrent_ingest}` (or may fail while
storing chunks whose document was already replaced).

A replacement that fails to embed leaves the predecessor untouched: the
swap only runs once the new document's chunks are all in hand.

A custom graph store cannot join the database transaction. If retiring the
predecessor's graph data fails after the swap commits, ingestion returns
`{:error, {:post_commit_graph_cleanup_failed, context}}`. The replacement is
already the published document in that case. The context contains the
failure reason, retired chunk IDs, their publication snapshot, and the
collection ID needed to retry graph cleanup without re-ingesting the source.

# `ingest_binary`

Ingests in-memory bytes, parsing them the same way `ingest_file/2`
parses a file on disk.

`:filename` is required: its extension picks the parser and its value
is stored as the document's `file_path` for provenance. Nothing is
written to disk.

## Options

  * `:filename` - Name the bytes came from, e.g. `"report.docx"` (required)
  * `:repo` - The Ecto repo to use (required)
  * `:source_id` - An optional identifier for grouping/filtering
  * `:metadata` - Optional map of metadata to store with the document
  * `:chunk_size` - Maximum chunk size in characters (default: 1024)
  * `:chunk_overlap` - Overlap between chunks (default: 200)
  * `:collection` - Collection name to organize the document (default: "default")
  * `:graph` - Enable GraphRAG extraction (default: from config)
  * `:replace` - When true, atomically replaces any prior document with the
    same `(collection, source_id)` once the new ingest completes. Requires
    `:source_id`. See "Replacing documents" in `ingest/2`.
  * `:on_chunk_error` - `:abort` (default) or `:skip`. See "When a chunk
    fails to embed" in `ingest/2`.

## Parsers that need a path

A parser only handles binaries when it says so via
`c:Arcana.FileParser.supports_binary?/0`. The built-in PDF parser
shells out to `pdftotext` and needs a real file, so ingesting PDF bytes
returns `{:error, {:binary_unsupported, Arcana.FileParser.PDF.Poppler}}`
— write them to a temp file and use `ingest_file/2` instead.

A parser that is *also* unavailable reports that instead, since a retry
through `ingest_file/2` would fail just the same: with `pdftotext`
missing, PDF bytes come back as `{:error, :poppler_not_available}`.

# `ingest_file`

Ingests a file, parsing its content and creating a document with embedded chunks.

Handles plain text, markdown, and PDF natively, plus any format with a
parser registered under `config :arcana, :file_parsers` — see
`Arcana.Parser` for resolution and `Arcana.FileParser` for the
behaviour. The document's `content_type` comes from the resolved
parser.

## Options

  * `:repo` - The Ecto repo to use (required)
  * `:source_id` - An optional identifier for grouping/filtering
  * `:metadata` - Optional map of metadata to store with the document
  * `:chunk_size` - Maximum chunk size in characters (default: 1024)
  * `:chunk_overlap` - Overlap between chunks (default: 200)
  * `:collection` - Collection name to organize the document (default: "default")
  * `:graph` - Enable GraphRAG extraction (default: from config)
  * `:replace` - When true, atomically replaces any prior document with the
    same `(collection, source_id)` once the new ingest completes. Requires
    `:source_id`. See "Replacing documents" in `ingest/2`.
  * `:on_chunk_error` - `:abort` (default) or `:skip`. See "When a chunk
    fails to embed" in `ingest/2`.

## Chunk metadata

Every chunk stores the `"start_byte"`/`"end_byte"` range it occupies in
the extracted text. When the parser also reports page positions (the
built-in PDF parser does), chunks additionally carry `"page_start"` and
`"page_end"`, so `Arcana.search/2` results can cite a page:

    [result | _] = Arcana.search("refund policy", repo: Repo)
    result.metadata["page_start"]
    #=> 4

---

*Consult [api-reference.md](api-reference.md) for complete listing*