# `Arcana.Pipeline`
[🔗](https://github.com/georgeguimaraes/arcana/blob/main/lib/arcana/pipeline.ex#L1)

Pipeline-based agentic RAG for Arcana.

Compose steps via pipes with a context struct flowing through each transformation:

    Arcana.Pipeline.new(question, llm: llm_fn)
    |> Arcana.Pipeline.search()
    |> Arcana.Pipeline.answer()

## Context

The `Arcana.Pipeline.Context` struct flows through the pipeline, accumulating
results at each step. Each step transforms the context and passes it on.

## Steps

- `new/1,2` - Initialize context with question and options
- `search/2` - Execute search, populate results
- `answer/1` - Generate final answer from results

## Configuration

Set defaults in your config to avoid passing options every time:

    config :arcana,
      repo: MyApp.Repo,
      llm: &MyApp.LLM.complete/1

## Example

    ctx =
      Arcana.Pipeline.new("What is Elixir?")
      |> Arcana.Pipeline.search()
      |> Arcana.Pipeline.answer()

    ctx.answer
    # => "Generated answer"

## How agentic is this?

`Arcana.Pipeline` is a composed sequence of steps you wire together
once. It's not a fully autonomous loop where an LLM decides its next
action (like ReAct) — that's what `Arcana.Loop` is for. The "agentic"
parts of the pipeline live inside specific steps:

  * `gate/2` — LLM decides whether retrieval is needed at all
  * `search(self_correct: true)` — retries with rewritten queries when
    results look insufficient
  * `reason/2` — multi-hop loop that asks the LLM for follow-up queries
    when current results don't answer the question
  * `answer(self_correct: true)` — regenerates the answer when grounding
    detects unsupported claims

These match the "Modular" and "Corrective" patterns in Singh et al.'s
survey rather than the full single-agent loop. You compose the pipeline
once at call time; you don't hand control to an LLM to drive it.

## References

  * [Agentic Retrieval-Augmented Generation: A Survey](https://arxiv.org/abs/2501.09136)
    (Singh et al., 2025) — canonical taxonomy of agentic RAG patterns.
    Arcana implements the Modular and Corrective patterns; not the full
    single-agent or multi-agent variants.
  * [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://arxiv.org/abs/2310.11511)
    (Asai et al., ICLR 2024) — inspires `gate/2` and `ground/2`.
  * [Corrective Retrieval Augmented Generation (CRAG)](https://arxiv.org/abs/2401.15884)
    (Yan et al., 2024) — inspires `search(self_correct: true)` and
    `answer(self_correct: true)`.
  * [HopRAG: Multi-Hop Reasoning for Logic-Aware RAG](https://arxiv.org/abs/2502.12442)
    (Liu et al., ACL 2025) — inspires `reason/2` (multi-hop search).
  * [What Is Agentic RAG?](https://weaviate.io/blog/what-is-agentic-rag)
    (Weaviate, 2024) — vendor-neutral overview of the pattern.

# `answer`

Generates the final answer from search results.

Collects all chunks from results, deduplicates by ID, and prompts the LLM
to generate an answer based on the context.

## Options

- `:answerer` - Custom answerer module or function (default: `Arcana.Pipeline.Answerer.LLM`)
- `:prompt` - Custom prompt function `fn question, chunks -> prompt_string end`
- `:llm` - Override the LLM function for this step
- `:self_correct` - Enable self-correcting answers (default: false)
- `:max_corrections` - Max correction attempts (default: 2)

## Example

    ctx
    |> Pipeline.search()
    |> Pipeline.answer()

    ctx.answer
    # => "The answer based on retrieved context..."

## Custom Answerer

    # Module implementing Arcana.Pipeline.Answerer behaviour
    Pipeline.answer(ctx, answerer: MyApp.TemplateAnswerer)

    # Inline function
    Pipeline.answer(ctx, answerer: fn question, chunks, opts ->
      llm = Keyword.fetch!(opts, :llm)
      prompt = "Q: " <> question <> "
Context: " <> inspect(chunks)
      Arcana.LLM.complete(llm, prompt, [], [])
    end)

# `decompose`

Breaks a complex question into simpler sub-questions.

Uses the LLM to analyze the question and split it into parts that can
be searched independently. Simple questions are returned unchanged.

## Options

- `:decomposer` - Custom decomposer module or function (default: `Arcana.Pipeline.Decomposer.LLM`)
- `:prompt` - Custom prompt function `fn question -> prompt_string end`
- `:llm` - Override the LLM function for this step

## Example

    ctx
    |> Pipeline.decompose()
    |> Pipeline.search()
    |> Pipeline.answer()

The sub-questions are stored in `ctx.sub_questions` and used by `search/2`.

## Custom Decomposer

    # Module implementing Arcana.Pipeline.Decomposer behaviour
    Pipeline.decompose(ctx, decomposer: MyApp.KeywordDecomposer)

    # Inline function
    Pipeline.decompose(ctx, decomposer: fn question, _opts ->
      {:ok, [question]}  # No decomposition
    end)

# `expand`

Expands the query with synonyms and related terms.

Uses the LLM to add related terms and synonyms that may help
find more relevant documents. The expanded query is used by `search/2`
if present.

## Options

- `:expander` - Custom expander module or function (default: `Arcana.Pipeline.Expander.LLM`)
- `:prompt` - Custom prompt function `fn question -> prompt_string end`
- `:llm` - Override the LLM function for this step

## Example

    ctx
    |> Pipeline.expand()
    |> Pipeline.search()
    |> Pipeline.answer()

The expanded query is stored in `ctx.expanded_query` and used by `search/2`.

## Custom Expander

    # Module implementing Arcana.Pipeline.Expander behaviour
    Pipeline.expand(ctx, expander: MyApp.ThesaurusExpander)

    # Inline function
    Pipeline.expand(ctx, expander: fn question, _opts ->
      {:ok, question <> " programming development"}
    end)

# `gate`

Decides whether retrieval is needed for the question.

Uses the LLM to determine if the question can be answered from general
knowledge or if it requires searching the knowledge base. Questions
about basic facts, math, or general knowledge can skip retrieval.

Sets `skip_retrieval: true` on the context if retrieval can be skipped,
which causes `answer/2` to generate a response without context.

## Options

- `:prompt` - Custom prompt function `fn question -> prompt_string end`
- `:llm` - Override the LLM function for this step

## Example

    ctx
    |> Pipeline.gate()      # Decides if retrieval is needed
    |> Pipeline.search()    # Skipped if skip_retrieval is true
    |> Pipeline.answer()    # Uses no-context prompt if skip_retrieval

# `ground`

Checks if the generated answer is grounded in the retrieved context.

Uses NLI scoring to check each sentence in the answer against the
retrieved context, producing a grounding score and hallucinated spans.

By default uses `Arcana.Grounder.Hallmark` (Vectara HHEM via Bumblebee).

## Options

- `:grounder` - Custom grounder module or function (default: `Arcana.Grounder.Hallmark`).
  Pass `Arcana.Grounder.LLMJudge` for LLM-as-judge faithfulness instead of NLI scoring.

## Example

    ctx
    |> Pipeline.search()
    |> Pipeline.answer()
    |> Pipeline.ground()

    ctx.grounding.score
    # => 0.95

    ctx.grounding.hallucinated_spans
    # => [%{text: "invented in 2010", start: 42, end: 59, score: 0.87}]

## Custom Grounder

    # Module implementing Arcana.Grounder behaviour
    Pipeline.ground(ctx, grounder: MyApp.LLMGrounder)

    # Inline function
    Pipeline.ground(ctx, grounder: fn answer, chunks, opts ->
      {:ok, %Arcana.Grounding.Result{score: 1.0, hallucinated_spans: []}}
    end)

# `new`

Creates a new pipeline context.

## Options

- `:repo` - The Ecto repo to use (defaults to `Application.get_env(:arcana, :repo)`)
- `:llm` - Function that takes a prompt and returns `{:ok, response}` or `{:error, reason}`
  (defaults to `Application.get_env(:arcana, :llm)`)
- `:limit` - Maximum chunks to retrieve (default: 5)
- `:threshold` - Minimum similarity threshold (default: 0.5)

## Example

    # With config defaults
    config :arcana, repo: MyApp.Repo, llm: &MyApp.LLM.complete/1

    Pipeline.new("What is Elixir?")

    # Or with explicit options
    Pipeline.new("What is Elixir?", repo: MyApp.Repo, llm: &my_llm/1)

# `reason`

Evaluates if search results are sufficient and searches again if not.

This step implements multi-hop reasoning by:
1. Asking the LLM if current results can answer the question
2. If not, getting a follow-up query and searching again
3. Repeating until sufficient or max iterations reached

Tracks `queries_tried` to prevent searching the same query twice.

## Options

- `:max_iterations` - Maximum additional searches (default: 2)
- `:prompt` - Custom prompt function `fn question, chunks -> prompt_string end`
- `:llm` - Override the LLM function for this step
- `:searcher` - Searcher for the follow-up searches. Defaults to the one
  `search/2` used, so a scoped searcher keeps applying without repeating
  the option. Falls back to `Arcana.Searcher.Arcana` when `search/2` never
  ran.
- `:searcher_opts` - Extra options for follow-up searches. These merge over
  the original search options when the searcher is inherited. Replacing the
  searcher starts from repo, limit, and threshold so backend-specific options
  do not leak between implementations.

Follow-up searches run against `ctx.collections`, which `search/2` records,
so the collection scope keeps applying too.

## Example

    ctx
    |> Pipeline.search()
    |> Pipeline.reason()    # Multi-hop if needed
    |> Pipeline.rerank()
    |> Pipeline.answer()

# `rerank`

Re-ranks search results to improve quality before answering.

Scores each chunk based on relevance to the question, filters by threshold,
and re-sorts by score. Uses `Arcana.Reranker.LLM` by default.

## Options

- `:reranker` - Custom reranker module or function (default: `Arcana.Reranker.LLM`)
- `:threshold` - Minimum score to keep (default: 7, range 0-10)
- `:prompt` - Custom prompt function for LLM reranker `fn question, chunk_text -> prompt end`

## Example

    ctx
    |> Pipeline.search()
    |> Pipeline.rerank()
    |> Pipeline.answer()

## Custom Reranker

    # Module implementing Arcana.Reranker behaviour
    Pipeline.rerank(ctx, reranker: MyApp.CrossEncoderReranker)

    # Inline function
    Pipeline.rerank(ctx, reranker: fn question, chunks, opts ->
      {:ok, my_rerank(question, chunks)}
    end)

The reranked results replace `ctx.results`, and scores are stored in `ctx.rerank_scores`.

# `rewrite`

Rewrites conversational input into a clear search query.

Uses the LLM to remove conversational noise (greetings, filler phrases)
while preserving the core question and all important terms.

This step should run before `expand/2` and `decompose/2` to clean up
the input before further transformations.

## Options

- `:rewriter` - Custom rewriter module or function (default: `Arcana.Pipeline.Rewriter.LLM`)
- `:prompt` - Custom prompt function `fn question -> prompt_string end`
- `:llm` - Override the LLM function for this step

## Example

    ctx
    |> Pipeline.rewrite()   # "Hey, tell me about Elixir" → "about Elixir"
    |> Pipeline.expand()
    |> Pipeline.search()
    |> Pipeline.answer()

## Custom Rewriter

    # Module implementing Arcana.Pipeline.Rewriter behaviour
    Pipeline.rewrite(ctx, rewriter: MyApp.RegexRewriter)

    # Inline function
    Pipeline.rewrite(ctx, rewriter: fn question, _opts ->
      {:ok, String.downcase(question)}
    end)

# `search`

Executes search and populates results in the context.

Uses `sub_questions` if present (from decompose step), otherwise uses the original question.

## Collection Selection

Collection scope is determined in this priority order:
1. `:collection` option passed to this function
2. `ctx.collections` (set by `select/2` if LLM selection was used)
3. Falls back to `"default"` collection

The resolved scope is recorded on `ctx.collections`, so later steps that
retrieve (like `reason/2`) search the same scope instead of picking one
of their own.

This allows you to explicitly specify a collection without using LLM-based selection:

    # Search a specific collection
    ctx |> Pipeline.search(collection: "technical_docs")

    # Search multiple specific collections
    ctx |> Pipeline.search(collection: ["docs", "faq"])

## Options

- `:searcher` - Custom searcher module or function (default: `Arcana.Searcher.Arcana`)
- `:collection` - `:all`, a collection name, or a list of collection names
- `:self_correct` - Enable self-correcting search (default: false)
- `:max_iterations` - Max retry attempts for self-correct (default: 3)
- `:sufficient_prompt` - Custom prompt function `fn question, chunks -> prompt_string end`
- `:rewrite_prompt` - Custom prompt function `fn question, chunks -> prompt_string end`

## Examples

    # Basic search (uses default collection)
    ctx |> Pipeline.search() |> Pipeline.answer()

    # Search specific collection
    ctx |> Pipeline.search(collection: "products") |> Pipeline.answer()

    # With pipeline options
    ctx
    |> Pipeline.expand()
    |> Pipeline.search(collection: "docs", self_correct: true)
    |> Pipeline.answer()

## Custom Searcher

    # Module implementing Arcana.Searcher behaviour
    Pipeline.search(ctx, searcher: MyApp.ElasticsearchSearcher)

    # Inline function
    Pipeline.search(ctx, searcher: fn question, collection, opts ->
      {:ok, my_search(question, collection, opts)}
    end)

## Self-correcting search

When `self_correct: true`, the agent will:
1. Execute the search
2. Ask the LLM if results are sufficient
3. If not, rewrite the query and retry
4. Repeat until sufficient or max_iterations reached

# `select`

Selects which collection(s) to search for the question.

By default, uses the LLM to decide which collection(s) are most relevant.
You can provide a custom selector module or function for deterministic routing.

Collection descriptions are automatically fetched from the database
and passed to the selector.

## Options

- `:collections` (required) - List of available collection names
- `:selector` - Custom selector module or function (default: `Arcana.Pipeline.Selector.LLM`)
- `:prompt` - Custom prompt function for LLM selector
- `:context` - User context map passed to custom selectors

## Example

    # LLM-based selection (default)
    ctx
    |> Pipeline.select(collections: ["docs", "api", "support"])
    |> Pipeline.search()

    # Custom selector module
    ctx
    |> Pipeline.select(
      collections: ["docs", "api"],
      selector: MyApp.TeamBasedSelector,
      context: %{team: user.team}
    )

    # Inline selector function
    ctx
    |> Pipeline.select(
      collections: ["docs", "api"],
      selector: fn question, _collections, _opts ->
        if question =~ "API", do: {:ok, ["api"], "API query"}, else: {:ok, ["docs"], nil}
      end
    )

The selected collections are stored in `ctx.collections` and used by `search/2`.

---

*Consult [api-reference.md](api-reference.md) for complete listing*