Validator
A component in long-running agent systems that objectively verifies whether execution results satisfy mechanically checkable constraints, producing binary objective judgments. It complements the evaluator's semantic assessments and serves as a reliability anchor for agent behavior.
Core Facts
A verifier is a standalone judgment program that accepts model-generated code and test results, and outputs a deterministic reward score.
The vast majority of AI risk disclosures lack substantiveness and are more boilerplate or compliance-oriented in nature.
Timeline (last 90 days)
When an attacker can implant matching sources for 30% of lies, the trust-source routing system actually performs worse than using the verifier alone.
The model's own prior signal can save 43% of verifier calls.
A validator that only says "looks good" is not much different from having no validator at all.
Over the past two weeks, five readers sent corrections to the author, all about the validator rather than the validated object; this was the sixth issue of the same kind but no one sent it, and the author found it by re-running the measurement before publishing an article that depends on it.
Output divergence itself is not necessarily a defect; it is only an incident when the divergence changes a downstream decision or causes validation to fail.
Mitigating the risk of the evidence system's "ceremonial burden" requires institutional design: evidence templates should be fixed in advance by humans rather than generated by the agent; verifier invocations should record the command, version, and input hash to ensure third parties can independently replay them; and key evidence should come from tools or models different from the generating agent to form cross-checks.
The validator allows specific binaries and subcommands, outright rejects write operation verbs, rejects pipes to a shell, and fails closed on anything it does not recognize.
The validator is implemented in about 120 lines of Python.
The validator outright rejects | sh, | bash, xargs, eval, and redirections like > / and >>, excluding executing pipes entirely from the first commands in runbooks.
kubectl exec does not directly modify objects itself, but it can execute arbitrary processes inside a container, so the validator lists it together with delete and scale as deny words.
17 more timeline events
All Facts (20)
A verifier is a standalone judgment program that accepts model-generated code and test results, and outputs a deterministic reward score.
80%VerifiedThe vast majority of AI risk disclosures lack substantiveness and are more boilerplate or compliance-oriented in nature.
75%UnverifiedThe model's own prior signal can save 43% of verifier calls.
50%UnverifiedWhen an attacker can implant matching sources for 30% of lies, the trust-source routing system actually performs worse than using the verifier alone.
50%UnverifiedA validator that only says "looks good" is not much different from having no validator at all.
50%UnverifiedOutput divergence itself is not necessarily a defect; it is only an incident when the divergence changes a downstream decision or causes validation to fail.
50%UnverifiedMitigating the risk of the evidence system's "ceremonial burden" requires institutional design: evidence templates should be fixed in advance by humans rather than generated by the agent; verifier invocations should record the command, version, and input hash to ensure third parties can independently replay them; and key evidence should come from tools or models different from the generating agent to form cross-checks.
50%UnverifiedThe validator allows specific binaries and subcommands, outright rejects write operation verbs, rejects pipes to a shell, and fails closed on anything it does not recognize.
50%UnverifiedThe validator is implemented in about 120 lines of Python.
50%UnverifiedThe validator outright rejects | sh, | bash, xargs, eval, and redirections like > / and >>, excluding executing pipes entirely from the first commands in runbooks.
50%Unverifiedkubectl exec does not directly modify objects itself, but it can execute arbitrary processes inside a container, so the validator lists it together with delete and scale as deny words.
50%UnverifiedThe validator should check the sentinel marker (such as # ---END---) before calling compile(); reversing the order lets syntactically legal truncated prefixes through.
50%UnverifiedTask-specific verifiers typically perform well within their training distribution but quickly fail when switched to a different task category, showing weak generalization ability.
50%UnverifiedFeedback from code execution is usually just a pass/fail boolean value, which is too coarse-grained; code similarity cannot be mapped to the final visual output; and verifiers trained for specific tasks lack generalization ability.
50%UnverifiedThe new paradigm proposed by the study can be summarized as: humans design the constraint harness and the verifier, and the agent is responsible for building the model.
50%UnverifiedA study proposes converting any verifiable EO task into a constraint harness and a verifier, allowing general programming agents to recursively build and improve remote sensing models, free from reliance on human experts.
50%UnverifiedThe key insight of the study is that any Earth observation task can be converted into two components: a constraint harness and a verifier; the constraint harness defines the resources, tools, and operations available to the agent; the verifier defines the success criteria.
50%UnverifiedThe programming agent decides its next action by reading the feedback signals returned by the verifier (such as classification accuracy, detection mAP, etc.), forming a 'generate-execute-evaluate' loop that achieves recursive self-improvement.
50%UnverifiedUnder this framework, few-shot learning corresponds to the constraint harness restricting the use of only a very small number of labeled samples, while the verifier still uses the standard test set.
50%UnverifiedDomain adaptation corresponds to the verifier requiring the model to maintain performance on independent data from different sensors or different geographic regions.
50%