Tiltmeter: Open-Source Tool Monitors Third-Party API Changes for Agents

When a third-party API's response format changes, typed code immediately crashes—but an Agent might "cleverly" fill in the gaps and keep running, which is actually more dangerous. The open-source tool Tiltmeter aims to solve this problem: it learns JSON structures by monitoring real API responses, notifies Agents when inconsistencies are detected, and can even automatically fix code.

Why Monitor API Changes?

When a third-party API's response structure changes, code in typed languages (like TypeScript) throws errors, and developers can immediately spot the problem. But AI Agents behave differently: they typically read the new response, try to fill in missing fields or ignore type errors, and continue executing the task. This "resilience" seems like an advantage, but it actually masks the breakdown of the underlying data contract, potentially leading to downstream logic errors, data corruption, or even business losses. As Tiltmeter's developer points out: "Agents typically read the new response, fill in the gaps, and keep running—which is worse." Therefore, a dedicated tool is needed to catch changes in the API's "shape," so Agents can check before trusting a response.

How Does Tiltmeter Work?

Tiltmeter learns the structure of the JSON APIs you depend on from real responses—no need to write specifications in advance. It builds a baseline from the most recent 20 checks and only reports contradictions: fields removed, types changed, new null values appearing, JSON bodies turning into HTML error pages, etc. This baseline-comparison approach effectively reduces false positives and focuses on genuinely breaking changes.

The tool's core is a small LangGraph graph with the flow: observe → compare → explain → persist → notify. The diff comparison part is entirely deterministic Python code with no LLM involved, ensuring reliability and reproducibility.

Integration with Agent Workflows

MCP Server Support

Tiltmeter runs an MCP (Model Context Protocol) server, enabling LangChain or LangGraph Agents to call the has_drifted or get_schema methods via langchain-mcp-adapters, checking for drift before trusting a response. This gives Agents real-time awareness of API changes, preventing them from acting on outdated assumptions.

MCP (Model Context Protocol) is an open protocol proposed by Anthropic to standardize how AI models connect to external tools and data sources. It defines a unified interface that allows Agents to discover and invoke external capabilities consistently, without writing a separate adapter for each tool. langchain-mcp-adapters is an adapter library in the LangChain ecosystem for connecting to MCP servers; it converts the tools exposed by an MCP server into tools that LangChain/LangGraph Agents can call directly. In Tiltmeter's case, the MCP server encapsulates the two checking capabilities has_drifted and get_schema as standard tools—the Agent doesn't need to understand the underlying implementation, it just calls them per the protocol to complete a drift check before trusting the API response.

Automatic Fixes and Human Review

The tiltmeter fix command hands drift issues to a Deep Agent. This Agent edits code in a temporary git worktree and opens a Pull Request. For things it can't determine (e.g., whether "1200" means cents or dollars), it leaves the decision to humans. This design balances automation efficiency with safety, ensuring critical decisions are made by people.

git worktree is a Git mechanism that allows multiple working directories to be checked out simultaneously from the same repository, each corresponding to a different branch or commit. Unlike switching branches directly, worktrees don't interfere with the current workspace state, making them ideal for automated tools to modify code in isolated environments. tiltmeter fix uses a temporary worktree to let the Deep Agent edit code in an independent, clean working directory, avoiding contamination of the developer's ongoing work. After the edits are complete, the Agent creates a Pull Request based on that worktree, so changes go through the normal code review process. This isolation design reduces the risk of automated modifications and makes rollback and auditing easier.

Installation and Usage

Tiltmeter is MIT-licensed and runs entirely locally. Install it with pip install tiltmeter. The project website is https://tiltmeter.dev. The developer encourages feedback: "If you connect Agents to third-party APIs, what problems do you encounter most often?"

Summary

Tiltmeter offers an API change monitoring solution for the Agent era, letting Agents proactively check data contracts through MCP integration, combined with automatic fixes and human review. For AI applications that depend on third-party APIs, this may reduce hidden failures and improve system reliability.

One-line Summary

Tiltmeter learns JSON structures by monitoring real API responses, detects changes and notifies Agents, and supports automatic fixes.

Paragraph Summary

Tiltmeter is an open-source tool designed to address the risk of AI Agents masking problems and continuing to run when third-party API response structures change. It learns JSON structures from real responses, builds a baseline from the most recent 20 checks, and only reports breaking changes such as removed fields, changed types, new null values, or JSON turning into HTML error pages. The tool's core is a LangGraph graph with the flow observe, compare, explain, persist, notify—where the diff comparison is deterministic Python code with no LLM involved. Tiltmeter provides an MCP server that lets LangChain or LangGraph Agents check for drift via has_drifted or get_schema before trusting a response; the tiltmeter fix command uses a Deep Agent to automatically fix code in a temporary git worktree and create a PR, leaving uncertain cases to human judgment. The tool is MIT-licensed, runs entirely locally, and can be installed via pip.