OpenTelemetry

Definition

OpenTelemetry is an open-source, vendor-neutral framework for instrumenting software and generating, collecting, and exporting telemetry such as traces, metrics, and logs. It supplies APIs, SDKs, conventions, a protocol and an optional Collector; storage, dashboards and alerting come from other systems.

A trace is a set of related spans: each span represents an operation with timing and context. Metrics answer aggregate questions over time, while logs describe individual events. OpenTelemetry standardizes how applications emit and move these signals; a backend is still needed to store and inspect them.

An AI agent may cross several services during one task. Propagating trace context lets a downstream span remain part of the same request path. A separate task identifier can help connect business-level work to that technical trace, but it is an application design choice.

What it covers

Instrumentation

Libraries or application code create spans, metrics and log records at meaningful operations.

Trace context

Trace and span identifiers connect work across services when context is propagated.

Collector

An optional service receives, processes and exports telemetry to one or more backends.

Semantic conventions

Shared attribute names make common operations easier to compare across implementations.

Identify the question first: which tool failed, how often do retries occur, or where did latency accumulate? Then choose the span boundaries, metric instruments and log fields that answer it. Instrumentation can be automatic for supported libraries and manual for agent-specific actions.

The Collector is optional. Its pipeline can receive telemetry, process it and export it to a backend. Decide where filtering, redaction and sampling occur, and document what evidence those choices may omit.

In practice

Practical design patterns by Min.

• Start with an operating question, then instrument the service boundaries needed to answer it.
• Carry trace context across agent, model, policy and tool services when those boundaries are under your control.
• Select only useful attributes; set sampling, redaction, access and retention policies before collecting sensitive workflow data.

Give a task or run a stable application identifier and propagate trace context across services where supported. Record meaningful operation names, outcome and duration at model, policy and tool boundaries.

Use metrics for trends such as errors and latency, a trace for one execution path and correlated logs for a particular event. Keep attribute meanings consistent across services.

Treat prompt, response and tool payload capture as a separate decision with access and retention controls. Avoid placing secrets or personal data in attributes or baggage.

Research notes

Source-grounded details and editorial applications. Follow each link for the original document.

Working example

Why the controls matter

An agent run spans the application, an authorization service and a document API. The trace shows that the model call completed, the policy check permitted the requested read, and the document API timed out twice. A retry metric shows whether the pattern is widespread; correlated logs identify the failed request without exposing the document body.

This evidence helps an operator fix a tool integration. It does not by itself prove that every action was recorded or satisfy a separate audit requirement.

Common pitfalls

• Treating OpenTelemetry as the dashboard or storage backend itself.
• Assuming sampled traces or routine logs form a complete audit trail.
• Putting prompt text, credentials or personal data into span attributes or propagated baggage by default.

A trace can be absent because of sampling, an uninstrumented path or delivery failure. Do not treat its absence as proof that an action did not happen.

Propagated context can cross trust boundaries. Sanitize or restrict incoming context from untrusted services, and avoid sending internal or sensitive baggage to external services.

Questions for operational review

Use these questions to decide what evidence your team needs. They are editorial prompts, not a compliance checklist.

• Which boundaries need a shared trace?
• What does the Collector remove or sample?
• Where is operational telemetry stored, and who may inspect it?