Trace an agent run

Record OpenTelemetry spans to see which tools and model calls an agent used. Start with the in-memory example to inspect a run locally, or send spans to an OTLP collector. Traces include full question, tool-input, and evidence text. To build the agent first, see Retrieve and answer.

Prerequisites

Install the agent and observability extras:

uv pip install "agentic-graphrag[neo4j,agents,observability]"

Without the observability extra, build_agent(tracer=...) raises AgentMissingExtraError that names agentic-graphrag[observability].

Record spans in memory

The example stores spans in memory, runs the agent with that tracer, and prints tool-span names with their parent spans. It does not send data to a collector.

import asyncio
import hashlib

from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter

from agrag.agents.build import build_agent
from agrag.agents.settings import AgentLLMSettings
from agrag.common.data_models import DEFAULT_SCHEMA
from agrag.common.data_models.document import Document, DocumentFamily, SourceFormat
from agrag.embedding import build_embedder
from agrag.graphdb import build_graph_store
from agrag.ingestion import Graph
from agrag.ingestion.extract import GlinerExtractor

TEXT = "Aspirin treats headaches. Ibuprofen also treats headaches."
DOCUMENT_KEY = "aspirin-notes"

exporter = InMemorySpanExporter()
provider = TracerProvider()
provider.add_span_processor(SimpleSpanProcessor(exporter))
tracer = provider.get_tracer("my-app")


async def main() -> None:
    graph_store = build_graph_store("neo4j", tracer=tracer)
    embedder = build_embedder(
        "ibm-granite/granite-embedding-small-english-r2", tracer=tracer
    )
    graph = await Graph.open(
        schema=DEFAULT_SCHEMA,
        graph_store=graph_store,
        embedder=embedder,
        extractor=GlinerExtractor(tracer=tracer),
        tracer=tracer,
    )
    try:
        await graph.add(
            documents=[
                Document(
                    text=TEXT,
                    title="Aspirin notes",
                    uri="notes://aspirin",
                    document_key=DOCUMENT_KEY,
                    source_format=SourceFormat.TXT,
                    family=DocumentFamily.PROSE,
                    content_hash=hashlib.sha256(TEXT.encode()).hexdigest(),
                    loader_name="manual",
                    char_count=len(TEXT),
                )
            ]
        )
        agent = build_agent(
            graph_store=graph_store,
            embedder=embedder,
            scope=frozenset({DOCUMENT_KEY}),
            project_id="headache-research",
            agent_id="research-agent",
            llm_settings=AgentLLMSettings.from_openai_compatible_env(),
            tracer=tracer,
        )

        await agent.ainvoke(
            {"task_name": "headache-treatment", "question": "What treats headaches?"}
        )
    finally:
        await graph_store.close()


asyncio.run(main())

spans = exporter.get_finished_spans()
names = {span.context.span_id: span.name for span in spans}

for span in spans:
    if span.attributes.get("openinference.span.kind") == "TOOL":
        parent = names.get(span.parent.span_id) if span.parent else None
        print(f"{span.name} (parent={parent})")

The tracer argument is keyword-only. The default is None, which turns tracing off. build_agent does not install global instrumentation.

Each run produces these spans:

• TOOL spans for each evidence tool the agent calls, such as discover_evidence and fetch_evidence. Retrieval spans are descendants of these TOOL spans.
• LLM spans with the llm.model_name attribute and token counts.
• CHAIN spans for graph nodes. Filter on openinference.span.kind to skip them.

The agent spans are children of the span that is active when you call ainvoke. To group a run under your own span, call ainvoke inside tracer.start_as_current_span("my-request").

Send spans to an OTLP endpoint

Replace the in-memory exporter with an OTLP exporter and a batch processor. Use the endpoint of your collector, or of Phoenix.

from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

provider = TracerProvider()
provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4317"))
)
tracer = provider.get_tracer("my-app")

Pass this tracer to build_agent as in the previous section. Call provider.shutdown() before your program exits, so the processor sends the last spans.

Spans always hold full text

Spans hold the question, the tool inputs, and the evidence text. The agent path has no option to omit that text. If the trace must contain less data, add a SpanProcessor that scrubs span attributes before export, or run without a tracer.

Trace the rest of the library with the same tracer

Components record spans only when they receive a tracer. SearchEngine does not pass its tracer to the graph store or embedder, so give the same tracer to all three when you want their spans connected. Also pass it to build_agent for agent and model spans. Pass it to Graph.open for ingestion and resolution spans. When you supply a custom extractor, give it the tracer too; Graph.open does not replace an injected extractor. A default extractor built by Graph receives the graph tracer.

The next block is a reference. tracer, chunks_by_id, graph_store, schema, and chat_model stand for objects that your application already builds.

from agrag.agents.model import build_chat_model
from agrag.eval.judge import ChatModelJudge
from agrag.ingestion.extract import (
    BAMLExtractor,
    ExtractionLLMSettings,
    GlinerExtractor,
)

# LLM extractor. Pass it to Graph as llm_extractor to escalate weak chunks.
extractor = BAMLExtractor(
    settings=ExtractionLLMSettings.from_openai_compatible_env(), tracer=tracer
)

# Local extractor, traced
primary = GlinerExtractor(tracer=tracer)

# Resolution, when you build the comparator yourself
from agrag.ingestion.resolve.resolver import LLMVerify

verifier = LLMVerify(chunks_by_id=chunks_by_id, tracer=tracer)

# The eval judge
judge = ChatModelJudge(chat_model, "gpt-4o-mini", tracer=tracer)

Graph passes its tracer to its resolution and merge steps, and community report generation uses it. A custom extractor must receive the tracer separately. A tracer on Graph.open does not change the tracer already set on the graph store or embedder.

Inside an agent run, every retrieval span nests under the TOOL span of the tool call that made it. A trace answers which tool call produced a given graph query or embedding call. Retrieval spans hold the full result text. The arrays agrag.result_texts and agrag.result_ids record every result, uncapped.

The span names and attributes these components emit are listed in Trace Spans.