Three Hermes Plugins Audit AI Agents' Self-Written Memory

AI agents write their own memory and skill files during long-running sessions, but this process lacks oversight and can lead to repetitive loops, fabricated user decisions, and skills of unknown provenance. A developer built three Hermes plugins to audit these failure modes; two have been merged into the upstream directory and one is under review.

Background: Three Risks of Agent Self-Writing

When running an AI agent, it continuously writes its run loops, decisions, and skills to files. Local models or stuck streams easily fall into repetition. Hermes itself truncates runaway text, but by design it allows vendor-confirmed tool calls to pass through even when the arguments are repetitive. Tool calls are also the only operation that can change external process state—writing files, posting, executing shell commands. As a result, repeated tool calls can cause real damage.

Another risk is the agent recording decisions the user never made. "The user decided X" is the most dangerous sentence an agent can write, because the next session will believe it. In addition, self-rewriting skills need traceable text provenance, otherwise they cannot be trusted.

Hermes framework and plugin mechanism: Hermes is an open-source framework for running AI agents, supporting agents that autonomously call tools and write memory and skill files across long-running sessions. Its plugin system lets developers mount hooks at key lifecycle points (such as pre_tool_call and before/after memory writes) to audit or intervene in agent behavior. Plugins are validated via hermes plugins validate and can be submitted to the upstream directory for others to install. This design allows third parties to add safety guardrails to agents without modifying core code, which is the technical foundation that made these three plugins independently developable and distributable.

Plugin 1: degeneracy-guard—Blocking Repetitive Tool Calls

hermes-degeneracy-guard hooks into the pre_tool_call stage and scores arguments before tool execution. When arguments look like a loop rather than data, it hands the call to Hermes's own approval gate instead of guessing on its own. Rejection, approval-gate errors, or timeouts all reject the call; cron defaults to rejection, so unattended runs fail closed rather than writing corrupted files.

Evidence: five hooks, none of which edit Hermes's own files; hermes plugins validate . passes on a pinned commit. Directory PR #141341 was merged on 2026-10-20.

pre_tool_call hook and approval gate: pre_tool_call is the plugin hook Hermes triggers before tool execution, allowing plugins to inspect or even modify call arguments. The approval gate is Hermes's built-in safety mechanism, typically used for high-risk operations that require human confirmation. degeneracy-guard hands suspicious repetitive calls to the approval gate rather than passing or rejecting them outright, leveraging the existing human oversight process while avoiding misjudgment by the plugin itself. In cron scenarios it rejects by default because no human confirmation is possible when unattended; failing closed prevents corrupted files from being silently written.

Plugin 2: provenance-gate—Detecting Fabricated User Decisions

hermes-provenance-gate checks every memory and skill write for constructs like "the user decided X" and verifies them against the user's actual words: the quote must appear verbatim in this session's user input, share keywords with the claim, and not drop negations. By default it only logs writes that fail the check; block mode rejects them.

Evidence: on 133 lines written and labeled by another model session that had never seen the plugin code (49 claims, 84 non-claims), precision was 1.000 (34/34) and recall was 0.694 (34/49). False negatives are published alongside hits, and reviews from two independent model sessions have been turned into regression tests. Directory PR #141935 was merged on 2026-10-20.

Precision and recall: In binary detection, precision is the proportion of samples flagged as "fabricated decisions" that are truly fabricated, and recall is the proportion of all real fabricated decisions that are successfully detected. provenance-gate's precision of 1.000 means all 34 claims it flagged were indeed fabricated, with no false positives; recall of 0.694 means it caught 34 of 49 real fabricated claims, missing 15. High precision ensures legitimate memory writes are not falsely penalized, while limited recall indicates some fabricated decisions may still slip through, requiring other audit measures.

Plugin 3: skill-lineage—Tracing Skill Text Provenance

hermes-skill-lineage records the content fingerprint before and after each skill write, sources that appeared earlier in the same session, features linking sources to added lines, and a seq → prev_hash → hash chain on entries. trace answers where a given version came from; descendants searches in reverse—given a page or skill, it returns all skills whose content ultimately embeds it; report lists versions containing lines with untraceable provenance; verify re-walks the chain, and head prints a value to be recorded elsewhere so later runs can distinguish a truncated chain from a complete one.

Evidence: 48 tests pass, one module is skipped on Python 3.11.16 and 3.14.7 (requires Hermes itself to be importable); hermes plugins validate . passes without warnings on Hermes 0.22.0. A manual end-to-end run in an isolated data root created a skill via skill_manage, then ran status/trace/descendants/report/verify/head—the raw logs are committed to the repository. Directory PR #142381 is open.

Uncovered Streaming Paths and Core Fixes

Hermes added a guard that cuts off a stream as soon as it turns into a repetitive loop. It is wired into chat_completions and Anthropic Messages streams; Bedrock Converse and Codex Responses streams do not yet use it, so on those streams an uncapped endpoint can stream a loop up to the output limit. #141477 is the report. #141479 wires the existing guard into both, reusing the same detection logic and end-of-turn handoff without adding new detection logic. Both are open.

Unsolved Problems the Developer Acknowledges

Source-to-line links are computed from shared shingles, anchors, and session order—not causal proof. The developer wants a cheaper way to distinguish "this line came from that page" from "they just happen to share words."

Keyed hashing means third parties can verify internal consistency but cannot re-derive links from public artifacts. The obvious alternative of publishing unkeyed digests would leak content through short-string dictionary attacks. The developer chose privacy over third-party reproducibility but is unsure whether that trade-off is right.

The three plugins have inconsistent default behavior: degeneracy-guard takes action, handing suspicious calls to the approval gate; provenance-gate observes, recording what it would have rejected as would_block until PROVENANCE_GATE_MODE=block is set; skill-lineage only records and never blocks. The developer set these based on differing costs of false blocking—corrupted files are hard to notice, rejecting a memory write is cheap to retry, and losing a fingerprint costs nothing—but three defaults across three sibling tools is a smell, and he would rather hear why it's wrong than defend it.

All three plugins are Apache-2.0 licensed, with no paid tier and no account required.

Disclosure: The developer uses AI agents in his own workflow; some text was drafted with AI assistance and edited by him. All measurements above are identifiable runs; failures, skips, and unmeasured areas are stated as such.

Short-string dictionary attacks and keyed hashing: A keyed hash computes a hash value using a secret key, so third parties cannot verify or re-derive content without the key. If the developer published unkeyed digests to allow public verification, an attacker could enumerate common short strings (such as "the user decided to delete the file"), compute their hashes, and compare them against the public digest to reverse-engineer the original content—this is a short-string dictionary attack. The developer chose keyed hashing, sacrificing third-party reproducibility in exchange for content privacy, a trade-off that is especially thorny in scenarios requiring public audit while involving sensitive memory.

One-line Summary

Developer builds three Hermes plugins to audit AI agents' self-writing risks; two merged upstream.

Paragraph Summary

AI agents write their own memory and skill files during long-running sessions, and lack of oversight can lead to repetitive tool calls, fabricated user decisions, and skills of unknown provenance. The developer built three Hermes plugins: degeneracy-guard intercepts repetitive-argument tool calls before execution and hands them to the approval gate, failing closed by default; provenance-gate detects "the user decided X"-style fabricated decisions with precision 1.000 and recall 0.694; skill-lineage records skill text provenance with a hash chain, supporting tracing and verification. The first two are merged upstream; the third PR is open. Additionally, Bedrock Converse and Codex Responses streams do not yet use the repetitive-loop guard, and fix PRs are open. The developer also disclosed unsolved problems, including the non-causal nature of provenance links, the privacy trade-off of keyed hashing, and inconsistent defaults across the three plugins.