telemetry.dev: Turning Every AI Agent Run into an Observable Trace

Debugging AI Agents is often a headache: a single run mixes multiple model calls, tool calls, and error handling, making it hard to pinpoint which step slowed things down or who's quietly burning money. telemetry.dev offers an answer on Product Hunt — treat every Agent run as a complete Trace, where each model call and tool call is a Span within that Trace, carrying latency, token usage, cost, and error information.

From "Black-Box Runs" to "White-Box Traces"

Traditional observability tools excel at handling HTTP requests and database queries, but AI Agent workflows are a different matter: they may combine multi-turn reasoning, parallel tool calls, and conditional branches. telemetry.dev's core design maps an entire Agent run to a single Trace, then breaks it down into multiple Spans. Developers can drill down layer by layer, much like inspecting distributed traces, to view the details of each model call (latency, tokens, cost, errors) and tool call (latency, errors).

The value of this granularity lies in troubleshooting. In the past, when an Agent produced anomalous output, you often had to guess from logs; now you can follow the Trace to find exactly which Span went wrong — whether it was a model timeout, a tool returning an error, or a looping invocation.

Distributed Tracing and the Trace/Span Model

The concepts of Trace and Span originate from distributed tracing systems (such as Jaeger and Zipkin). A complete request chain is modeled as a Trace, while each unit of operation within the chain is a Span. Spans form a directed acyclic graph through parent-child relationships, expressing call hierarchy and parallelism. Each Span typically records start time, duration, status, and custom attributes. The key to migrating this model to the AI Agent context is that model calls and tool calls naturally have the characteristics of "having a start, having an end, having inputs and outputs, and possibly failing" — exactly matching the semantics of a Span. Therefore, telemetry.dev doesn't need to invent a new data model; it reuses a mature observability abstraction, letting developers use their existing troubleshooting mindset (drilling down layer by layer, locating latency bottlenecks) to understand Agent behavior.

OpenTelemetry (OTel)

OpenTelemetry is an open-source observability standard led by the Cloud Native Computing Foundation (CNCF). It unifies the collection protocols and SDKs for three types of telemetry data — Traces, Metrics, and Logs — and provides Exporters to send data to any backend. Its value lies in decoupling "collection" from "storage/display": applications only need to instrument according to the OTel specification, and the backend can be freely swapped. telemetry.dev supports OTel exporter integration, meaning teams already using OTel to monitor their infrastructure don't need to maintain a separate instrumentation system — LLM calls can flow directly into existing pipelines as Spans, significantly reducing adoption cost.

Prompts and Evals Share the Same Set of Traces

Another differentiating capability of telemetry.dev is unifying Prompt management and the Evals evaluation system on top of Traces. Modifying a Prompt doesn't require redeploying the application — after changes, you can run it directly on a dataset and then compare against production scores.

This means Prompt iteration shifts from "change code, redeploy, judge by feel" to "change Prompt, run evals, compare with data." For teams that need to continuously optimize Agent performance but don't want to release frequently, this loop can significantly shorten iteration cycles.

LLM Evals

Evals refers to a collection of methods for systematically assessing the quality of large model outputs, typically including rule-based checks, comparison against reference answers, and scoring using another model as a judge (LLM-as-a-Judge). In the Agent context, Evals need to build datasets from real runtime data to reflect performance under the production distribution. telemetry.dev's design of having Evals and Traces share the same source solves the traditional evaluation problem of "offline datasets being disconnected from online traffic": production Traces serve both as a basis for troubleshooting and as a source of evaluation samples. Scores after Prompt changes can also be directly compared against historical online scores, forming a quantifiable regression judgment.

Alerts: Errors, Cost Spikes, and Agent Loops

telemetry.dev provides built-in alerting mechanisms covering three types of issues: errors, cost spikes, and agent loops. Agent loops are a risk unique to the AI Agent context — the model may repeatedly call tools on some logic without converging, and if no one is watching, the bill will rise rapidly. Including loop detection in alerts shows the team has a clear understanding of the runtime risks of Agents.

Cost alerts are equally critical: LLM's token-based billing model means a single anomalous call can generate unexpected bills, and real-time alerts let teams intervene before costs spiral out of control.

Integration: SDK and OpenTelemetry

telemetry.dev provides TypeScript and Python SDKs, and also supports integration via OpenTelemetry exporter. The latter is particularly noteworthy: if a team is already using the OpenTelemetry ecosystem for infrastructure monitoring, they can directly reuse their existing OTel pipeline for LLM observability, reducing the migration cost of introducing a new tool. For engineering teams that have already embraced OTel, this is a pragmatic compatibility strategy.

The product received 89 votes on Product Hunt, ranking 4th, categorized under Analytics, Developer Tools, and Artificial Intelligence. Its target users are clearly developer teams building AI Agents.

Who Should Use It

telemetry.dev is suited for teams already running AI Agents in production that have encountered the following problems: difficulty pinpointing the specific step when an Agent run fails; lack of data support for Prompt iteration; difficulty attributing cost fluctuations; and a desire to incorporate LLM observability into their existing OTel system. For projects still in the prototype stage with very low call volumes, the necessity of such a tool is lower.

Overall, telemetry.dev's approach is to upgrade LLM observability from "reading logs" to a combined capability of "viewing Traces + running Evals + setting alerts." This approach of migrating operational observability experience to the AI Agent context aligns with the real needs for stability and cost control as Agent applications move from demo to production.

One-line Summary

telemetry.dev uses the Trace+Span model to provide observability, Evals, and cost alerts for AI Agents.

Paragraph Summary

telemetry.dev is an observability tool for AI Agent developers. Its core idea is to map each Agent run to a Trace, modeling each model call and tool call as a Span carrying latency, token usage, cost, and error information, turning what was once a black-box run into something that can be drilled down layer by layer for troubleshooting. It also unifies Prompt management and Evals evaluation on the same set of Traces, so changing a Prompt doesn't require redeployment — you can run evals on a dataset and compare against production scores. Built-in alerts cover three types of runtime risks: errors, cost spikes, and Agent loops. For integration, it provides TypeScript/Python SDKs and supports OpenTelemetry exporter to reuse existing monitoring pipelines. The product is suited for teams already running Agents in production that face troubleshooting and cost attribution pain points.