Attest: Zero-Model-Call Replay for Regression Testing LangGraph Agents

Attest records LangGraph runs via OTel and replays them in CI, enabling zero-model-call assertions on tool-calling behavior.
Attest is an Apache-2.0 open-source project that tackles the difficulty of asserting on LLM agent regression tests. It records a real LangGraph run via OpenTelemetry spans, then replays it in CI, answering model calls with recorded content so no real model requests are made. Assertions focus on tool-calling behavior, such as call order and per-parameter call-count limits, turning agent behavior into versionable test fixtures. The author shares the LangGraph setup and candidly notes that LangGraph.js capture is not yet available, some adapters are in beta, and most of the code was generated by a coding agent. The approach suits Python teams wanting low-cost CI integration, but is not meant to validate the semantic quality of responses.
Why Agent Testing Always Feels "Unclear"
Writing regression tests for LLM agents has always been awkward. You can't assert on return values the way you would for an ordinary function, because the same input can produce differently worded outputs from the model. You also don't want to actually call the model in CI, because it's expensive, slow, and introduces non-determinism. The common approach in the community is to write a pile of integration tests that "look right," which often ends up degrading into clicking through things manually.
The open-source project Attest (Apache-2.0) tries a different angle: record a real run, replay it in CI without making any model calls, and then assert on tool-calling behavior. Its author recently wired the capture capability into LangGraph and publicly shared the configuration approach and current shortcomings on Reddit.
Recording a LangGraph Run
Attest's capture reads OpenTelemetry spans, and LangSmith itself supports an OTel mode. The key step is pointing the tracer provider at Attest's own local receiver—the LangSmith API key is never used at all, just a placeholder.
The command the author gives is roughly this: set LANGSMITH_TRACING=true, LANGSMITH_TRACING_MODE=otel, LANGSMITH_API_KEY=unused, then run attest capture --name refund-bug -- python run_graph.py.
The recording process keeps two kinds of spans, chat and execute_tool (runnable/chain spans are ignored), and records every model interaction through a local proxy while redacting secrets.
One flaw the author explicitly calls out: the message ids and lc_run-- ids that LangChain generates inside the LangSmith prompt string are not templated, so they show up when you diff two captures. However, the replay matching stage ignores these ids, so it doesn't affect assertion results.

OpenTelemetry (OTel) is an open-source, vendor-neutral observability framework that defines a unified data model and API/SDK specification for three signal types: traces, metrics, and logs. A trace consists of a set of spans, where each span represents an operation (such as a model call or tool execution) and carries a name, attributes, timestamps, and parent-child relationships. LangSmith's OTel mode allows spans generated inside LangChain/LangGraph to be exported via the standard OTLP protocol instead of being sent to the LangSmith cloud. Attest leverages exactly this by pointing the tracer provider at a local receiver to intercept spans, completing the recording without depending on the LangSmith service. Understanding the structure of OTel spans helps clarify why Attest keeps only chat and execute_tool spans—they correspond to model interactions and tool executions respectively, and are the core objects for behavioral assertions.
LangGraph is a framework from the LangChain team for building stateful, multi-step agent applications. It organizes nodes and edges as a graph, supporting loops, conditional branching, and persistent state. Nodes in the graph can be model calls, tool executions, or custom logic, while edges define control flow. LangGraph's Python and TypeScript versions differ in instrumentation support: the Python version can already produce GenAI spans through LangSmith's OTel mode, while the TypeScript version does not yet offer equivalent instrumentation, so Attest's capture path is temporarily unavailable for LangGraph.js. Here, instrumentation refers to the framework's built-in tracing capability, which automatically generates spans that follow semantic conventions for model calls and tool executions—the foundation for record-and-replay.
Replay: No Model Touched in CI
When tests run, you execute the current version of the graph, while the proxy answers model calls using the recorded content—so CI makes zero model calls throughout. Assertions are written at the tool-call level, for example:
chain = (
expect(trace)
.tools_called_in_order(["lookup_order", "issue_refund"])
.tool_cardinality("issue_refund", per="order_id", at_most=1)
)
attest.evaluate(chain)
The meaning of this assertion is: the tools must be called in order, and for the same order_id, issue_refund may appear at most once. This maps exactly to the "duplicate refund" incident the author demonstrates—first record the bug, then run the fixed code in CI to confirm that the duplicate refund is blocked.
If you only want callbacks and don't need full capture, Attest also provides a LangChainAdapter: inside the adapter.capture() context, attach the handler to the callbacks of graph.invoke, then retrieve the trace object from adapter.trace.

Record-and-replay testing is a classic testing pattern: first record the inputs and outputs of a real interaction in production or development, then substitute the recorded data for the real dependency during testing. In the agent scenario, model calls are replaced by recorded content, but the current version's graph logic, tool implementations, and routing decisions still execute for real. This validates the impact of code changes on tool-calling behavior while avoiding the cost, latency, and non-determinism of model calls. The replay matching stage usually needs to handle non-deterministic factors (such as random ids and timestamps); Attest's strategy is to ignore differences in these ids and focus only on behavioral characteristics such as tool call order, arguments, and counts. This pattern requires that recording covers enough branches; otherwise, replay can only verify the paths that were recorded.
Clear Boundaries and Shortcomings
The author states the limitations plainly, which deserves credit:
- The capture path has been tested on Python, with langgraph 1.2.19 and LangSmith 0.14.17.
- The TypeScript version of LangGraph currently has no instrumentation that produces GenAI spans, so capture for LangGraph.js is unavailable; the LangChain.js callback adapter works but is in beta.
- The Python LangChain adapter is likewise in beta.
- The author candidly notes that most of Attest's code was generated by a coding agent based on plans he wrote and reviewed. This is important context for anyone evaluating the project's maturity.
Who This Approach Is For
The value of record-and-replay testing lies in turning "agent behavior" into a versionable asset: a recorded trace is a test fixture, and assertions are an explicit expression of business rules—for example, refunds may happen only once, and a lookup must precede an action. It is not suited to validating the semantic quality of model responses, nor does it solve the question of whether the recording itself covers enough branches. For LangGraph Python users, this is a low-barrier option that can be plugged into CI immediately; TypeScript users will need to wait for instrumentation to catch up.
For teams moving agents from demo to production, this combination of "zero-model-call replay + tool-call assertions" may be one of the most cost-effective forms of regression testing available today.
Related articles

Baseten Releases Bison-1 and Groundhog-1, Topping the Decision Model Leaderboard
Baseten releases open-weight decision models bison-1 and groundhog-1 built on Gemma 4; bison-1 scores 76 on the AA Decision Index v2.1 to top the leaderboard. Both support 128k context and image input—here we analyze their capability combination, benchmark significance, and selection advice for developers.

Cline CLI v3.0.75 Released: Yolo Mode Now Default for One-Shot Runs
Cline CLI v3.0.75: one-shot runs now default to yolo mode, --timeout and Ctrl+C can interrupt the first round, read_files is no longer truncated to 8k chars, plus reasoning parameter fixes and 94 new providers.

Cline Desktop v0.0.50 Update: Model Catalog Overhaul and Reasoning Control Fixes
Cline Desktop v0.0.50 fixes Reflection Beam 400 errors, misdelivered reasoning parameters, and stuck compaction; read_files no longer truncates to 8k characters; adds 94 providers and changes 67 default models.