# `Arcana.Graph` [🔗](https://github.com/georgeguimaraes/arcana/blob/main/lib/arcana/graph.ex#L1) GraphRAG (Graph-enhanced Retrieval Augmented Generation) for Arcana. This module provides the public API for GraphRAG functionality: - Building knowledge graphs from documents - Graph-based search and retrieval - Fusion search combining vector and graph results - Community summaries for global context ## Installation GraphRAG is optional and requires separate installation: $ mix arcana.graph.install $ mix ecto.migrate Add the NER serving to your supervision tree: children = [ MyApp.Repo, Arcana.Embedder.Local, Arcana.Graph.NERServing # For entity extraction ] ## Configuration GraphRAG is disabled by default. Enable it in your config: config :arcana, graph: [ enabled: true, # Community detection community_levels: 1, # Hierarchy depth for Leiden algorithm (1 = flat) resolution: 1.0, # Leiden granularity (lower = fewer, larger communities) min_size: 1, # Minimum community size # RRF fusion (combining vector + graph search results) rrf_k: 60, # Ranking constant (higher = less weight to top ranks) rrf_pool_multiplier: 2, # Over-fetch multiplier before RRF combine # Query-time graph traversal query_depth: 0, # Hops to expand matched entities (0 = direct mentions only) query_depth_decay: 0.5, # Score decay per hop for chunks reached via traversal # Community summaries in ask pipeline community_summary_limit: 5, # Max summaries injected as background context community_summary_level: 0, # Level(s) to pull summaries from: integer, list, range or :all # Community summarization prompt limits summary_max_entities: 50, # Top N entities by connection count per summary summary_max_relationships: 100, # Top N relationships per summary # Entity matcher (pluggable) entity_matcher: :embedding, # :embedding (default), :ner, or custom module entity_embedding_threshold: 0.3, # Threshold for the :embedding matcher # Structured context in ask pipeline (GraphRAG Local Search) context_entity_limit: 10, # Max entity descriptions in LLM context context_relationship_limit: 20 # Max relationships in LLM context ] Or enable per-call: Arcana.ingest(text, repo: MyApp.Repo, graph: true) Arcana.search(query, repo: MyApp.Repo, graph: true) ## Usage # Build a graph from chunks {:ok, graph_data} = Arcana.Graph.build(chunks, entity_extractor: &MyApp.extract_entities/2, relationship_extractor: &MyApp.extract_relationships/3 ) # Convert to queryable format graph = Arcana.Graph.to_query_graph(graph_data, chunks) # Search the graph results = Arcana.Graph.search(graph, entities, depth: 2) # Fusion search combining vector and graph results = Arcana.Graph.fusion_search(graph, entities, vector_results) ## Components GraphRAG consists of several modules: * `Arcana.Graph.EntityExtractor` - Behaviour for entity extraction * `Arcana.Graph.EntityExtractor.NER` - Built-in NER implementation (default) * `Arcana.Graph.RelationshipExtractor` - Behaviour for relationship extraction * `Arcana.Graph.RelationshipExtractor.LLM` - Built-in LLM implementation (default) * `Arcana.Graph.RelationshipExtractor.Cooccurrence` - Local co-occurrence (no LLM) * `Arcana.Graph.CommunityDetector` - Behaviour for community detection * `Arcana.Graph.CommunityDetector.Leiden` - Built-in Leiden implementation (default) * `Arcana.Graph.CommunitySummarizer` - Behaviour for community summarization * `Arcana.Graph.CommunitySummarizer.LLM` - Built-in LLM implementation (default) * `Arcana.Graph.GraphQuery` - Queries the knowledge graph * `Arcana.Graph.FusionSearch` - Combines vector and graph search with RRF * `Arcana.Graph.GraphBuilder` - Orchestrates graph construction ## Custom Implementations All core extractors and detectors support the behaviour pattern for extensibility: # Custom entity extractor config :arcana, :graph, entity_extractor: {MyApp.SpacyExtractor, endpoint: "http://localhost:5000"} # Custom relationship extractor config :arcana, :graph, relationship_extractor: {MyApp.PatternExtractor, patterns: [...]} # Custom community detector config :arcana, :graph, community_detector: {MyApp.LouvainDetector, resolution: 0.5} # Custom community summarizer config :arcana, :graph, community_summarizer: {MyApp.ExtractiveSum, max_sentences: 3} # `build` Builds graph data from document chunks. Delegates to `Arcana.Graph.GraphBuilder.build/2`. ## Options - `:entity_extractor` - Function to extract entities from text - `:relationship_extractor` - Function to extract relationships ## Example {:ok, graph_data} = Arcana.Graph.build(chunks, entity_extractor: fn text, _opts -> Arcana.Graph.EntityExtractor.NER.extract(text, []) end, relationship_extractor: fn text, entities, _opts -> Arcana.Graph.RelationshipExtractor.extract(text, entities, my_llm) end ) # `build_and_persist` Builds and persists graph data from chunk records during ingest. Processes chunks incrementally, persisting after each chunk so progress is saved continuously. Accepts an optional `:progress` callback that receives `{current_chunk, total_chunks}` after each chunk is processed. ## Options * `:progress` - Callback function `fn current, total -> ... end` called after each chunk ## Examples # With progress logging Arcana.Graph.build_and_persist(chunks, collection, repo, progress: fn current, total -> IO.puts("Processed chunk #{current}/#{total}") end ) # `community_summaries` Gets community summaries from the graph. Community summaries provide high-level context about clusters of related entities, useful for global queries. ## Options - `:level` - Filter by hierarchy level (0 = finest) - `:entity_id` - Filter by communities containing entity ## Example # Get all top-level summaries summaries = Arcana.Graph.community_summaries(graph, level: 0) # `config` Returns the current GraphRAG configuration. ## Example Arcana.Graph.config() # => %{enabled: false, community_levels: 1, resolution: 1.0} # `enabled?` Returns whether GraphRAG is enabled globally. Check this before performing graph operations: if Arcana.Graph.enabled?() do # Build graph during ingest end # `expand_entity_ids` Expands entity ids through the relationships table for `depth` hops. Walks `arcana_graph_relationships` breadth-first (both directions) from the given entity ids, returning a map of hop distance to the entity ids first discovered at that hop: `%{0 => direct_ids, 1 => neighbors, ...}`. Each entity appears once, at its minimal hop distance. `collection_ids` follows the usual scoping semantics: `nil` is unscoped, a list restricts discovered neighbors to those collections, and an empty list matches nothing (no neighbors are pulled in). ## Options * `:repo` - Ecto repo (required) # `find_entities` Finds entities in the graph by name. ## Options - `:fuzzy` - Enable fuzzy matching (default: false) # `fusion_search` Combines vector search and graph search using Reciprocal Rank Fusion. This is the primary retrieval method for GraphRAG, merging results from both vector similarity and knowledge graph traversal. ## Options - `:depth` - Graph traversal depth (default: 1) - `:limit` - Maximum results to return (default: 10) - `:k` - RRF constant (default: 60) ## Example # Run vector search separately {:ok, vector_results} = Arcana.search(query, repo: MyApp.Repo) # Extract entities from query {:ok, entities} = Arcana.Graph.EntityExtractor.NER.extract(query, []) # Combine with graph search results = Arcana.Graph.fusion_search(graph, entities, vector_results) # `installed?` Returns whether the GraphRAG schema is installed in the database. Separate from `enabled?/0`, which only reads config. The graph tables ship on their own migration version, so an app can legitimately run `Arcana.Migration.up/1` and skip `Arcana.Graph.Migration.up/1` - and then a graph query fails with `relation "arcana_graph_entities" does not exist` rather than returning nothing. Config being on says the operator wants graph features; this says the database can actually answer. if Arcana.Graph.installed?(MyApp.Repo) do # safe to query entities, relationships, communities end Checks the table rather than the `arcana_graph:` version marker on purpose: the marker can be missing on an install that predates versioning, or that had its table comment clobbered, and those databases can still be queried. ## Options * `:prefix` - the Postgres schema to look in. Without it the question is whether an unqualified query would find the table, which is what the callers actually do # `normalize_levels` Normalizes a level selection into `:all` or a list of levels. # `query_depth` Resolves the query-time graph traversal depth from per-call opts and global config. Reads `:graph_depth` from `opts` first, then falls back to `config :arcana, graph: [query_depth: n]` (default 0). Raises `ArgumentError` on anything other than a non-negative integer. # `resolve_entity_extractor` Resolves the entity extractor from options and config. # `search` Searches the knowledge graph for relevant chunks. Finds entities matching the query, traverses relationships, and returns connected chunks. ## Options - `:depth` - How many hops to traverse (default: 1) ## Example entities = [%{name: "OpenAI", type: :organization}] results = Arcana.Graph.search(graph, entities, depth: 2) # `summary_levels` Returns the community hierarchy levels that queries read summaries from. Configured with `community_summary_level`, which accepts an integer, a list of integers, a range, or `:all`. Summarization consumes the same key, so the levels that get summarized and the levels `ask/2` reads can't drift apart. Returns `:all` or a list of levels. ## Examples Arcana.Graph.summary_levels() # => [0] # `to_query_graph` Converts builder output to queryable graph format. Delegates to `Arcana.Graph.GraphBuilder.to_query_graph/2`. # `traverse` Traverses the graph from a starting entity. ## Options - `:depth` - Maximum traversal depth (default: 1) --- *Consult [api-reference.md](api-reference.md) for complete listing*