# `Arcana`
[🔗](https://github.com/georgeguimaraes/arcana/blob/main/lib/arcana.ex#L1)

RAG (Retrieval Augmented Generation) library for Elixir.

Arcana provides document ingestion, embedding, and vector search
capabilities that you can embed into any Phoenix/Ecto application.

## Usage

    # Ingest a document
    {:ok, document} = Arcana.ingest("Your text content", repo: MyApp.Repo)

    # Search for relevant chunks
    {:ok, results} = Arcana.search("your query", repo: MyApp.Repo)

    # Ask questions with RAG
    {:ok, answer} = Arcana.ask("What is X?", repo: MyApp.Repo, llm: my_llm)

    # Delete a document
    :ok = Arcana.delete(document.id, repo: MyApp.Repo)

## Modules

  * `Arcana.Config` - Configuration management
  * `Arcana.Ingest` - Document ingestion
  * `Arcana.Search` - Vector and hybrid search
  * `Arcana.Ask` - RAG question answering
  * `Arcana.Graph` - GraphRAG functionality

# `ask`

Asks a question using retrieved context from the knowledge base.
See `Arcana.Ask.ask/2` for options.

# `chunker`

Returns the configured chunker as a `{module, opts}` tuple.
See `Arcana.Config` for configuration options.

# `config`

Returns the current Arcana configuration.

# `count_documents`

Counts documents matching the `list_documents/1` filters.
See `Arcana.Documents.count_documents/1`.

# `delete`

Deletes a document and all its chunks.

When the graph is enabled, also sweeps the document's collection for
orphaned graph data: entities left with zero mentions are deleted and
communities that referenced them are marked dirty so the next
summarize pass regenerates them.

Returns `:ok`, `{:error, :not_found}`, `{:error, {:sweep_failed, reason}}`
when the graph store *returns* an error from its sweep, or
`{:error, reason}` for a database failure such as a foreign key violation,
or a not-null violation from another table pointing at the document.

A sweep that raises a *database* error reports as a database failure rather
than as `:sweep_failed` — so match `{:error, _}` as well if you route on the
sweep case. `sweep_orphans/2` deletes through `repo.delete_all/1`, which
skips changeset constraint mapping, so a host row referencing an entity the
sweep wants to remove arrives that way.

When a non-Ecto graph store, including the built-in `:memory` store, is
cleaned after the database delete commits, a returned error or exception is
wrapped as
`{:error, {:post_commit_graph_cleanup_failed, context}}`. The context carries
`:reason`, `:chunk_ids`, `:published_chunk_ids`, and `:collection_id`, so the
graph deletion and sweep can be retried even though the document row is
already gone. Calling `delete/2` inside an existing Ecto transaction when an
a non-Ecto graph cleanup is required returns
`{:error, :external_graph_store_requires_post_commit_delete}` before deleting.

A concurrent delete reports `{:error, :not_found}`, the same as losing that
race by a moment more would have.

Infrastructure failures — a lost connection, a pool timeout — still raise.
There is nothing useful to hand back and the transaction's fate is unknown,
so those are left to the caller's supervisor rather than flattened into a
tuple that would read like a clean refusal.

Called on its own, the delete and the orphan sweep run in one transaction,
so a `:sweep_failed` leaves the document in place: nothing happened and the
call can be retried. That is worth knowing if you are upgrading, because
this case used to delete the document and report the failed cleanup
afterwards. Inside a transaction of your own it behaves differently — see
below.

The transaction covers what runs on `:repo`. A graph store holding its data
anywhere else is outside it — that includes the built-in `:memory` backend,
whose sweep is a `GenServer.call`, not only custom stores. So the two can
disagree in both directions: a store that fails partway through its own
sweep may have applied some of it even though the document survives, and a
store that sweeps successfully before the repo side rolls back has dropped
graph data for a document that is still there.

## Inside a transaction of your own

It does not open one, and it does not roll anything back: rolling back a
nested Ecto transaction aborts the outermost one, which would kill your
transaction while handing you an error that looks recoverable.

So the guarantee above is weaker here. A `:sweep_failed` comes back as a
tuple with your transaction intact, but **the delete has been applied** and
stays in your transaction's scope — undoing it is yours to do. That is the
one place where `:sweep_failed` does not mean "nothing happened".

A database failure is harsher: Postgres aborts to the nearest savepoint, and
with none open that is the whole transaction, so the error ends it whatever
gets returned. If you need to survive one, set your own savepoint around the
call:

    Repo.query!("SAVEPOINT before_delete")

    case Arcana.delete(id, repo: Repo) do
      {:error, _reason} -> Repo.query!("ROLLBACK TO SAVEPOINT before_delete")
      :ok -> Repo.query!("RELEASE SAVEPOINT before_delete")
    end

That recovers every failure the built-in stores can produce: a failed
`repo.delete`, a `:sweep_failed`, and a sweep that raises.

A custom graph store can still defeat it, by opening a nested `Ecto`
transaction of its own around work that then fails — DBConnection marks the
connection aborted, and the `ROLLBACK TO SAVEPOINT` is refused along with
everything else. Calling `delete/2` outside your transaction avoids the
question entirely.

Sweeping is optional for custom graph stores: one that doesn't
implement `c:Arcana.Graph.GraphStore.sweep_orphans/2` returns `:ok` and
leaves the orphans alone.

## Options

  * `:repo` - The Ecto repo to use (required)
  * `:graph` - Sweep orphaned graph data after deletion (default: from config)

# `embedder`

Returns the configured embedder as a `{module, opts}` tuple.
See `Arcana.Config` for configuration options.

# `get_document`

Fetches a document by id. See `Arcana.Documents.get_document/2`.

# `get_document_metadata`

Fetches sparse metadata for published documents in one explicitly scoped read.
See `Arcana.Documents.get_document_metadata/2`.

# `graph_enabled?`

Returns whether GraphRAG is enabled.

# `ingest`

Ingests text content, creating a document with embedded chunks.
See `Arcana.Ingest.ingest/2` for options.

# `ingest_binary`

Ingests in-memory bytes, routing on the required `:filename` option's
extension. See `Arcana.Ingest.ingest_binary/2` for options.

# `ingest_file`

Ingests a file, parsing its content and creating a document with embedded chunks.
See `Arcana.Ingest.ingest_file/2` for options.

# `list_documents`

Lists documents, newest first. See `Arcana.Documents.list_documents/1`.

# `rewrite_query`

Rewrites a query using a provided rewriter function.
See `Arcana.Search.rewrite_query/2` for options.

# `search`

Searches for chunks similar to the query.
See `Arcana.Search.search/2` for options.

---

*Consult [api-reference.md](api-reference.md) for complete listing*