LLM judge
A paradigm in which a large language model is used as a judge to automatically score model outputs. It is typically used together with scoring rubrics.
Timeline (last 90 days)
When switching to an LLM judge from another vendor, the official agent rankings were preserved, but the number of accepted papers doubled.
The evaluation criteria of an LLM judge drift as model versions are updated, business rules are adjusted, and user behavior changes, and the drift itself does not proactively raise errors.
LLM judges perform close to coin-flipping when detecting omission-type errors.
The LLM judge mechanism verifies the presence of information rather than its absence, so there is no clear anchor to trigger a check for omissions.
The solution to missed detections is to restructure the task rather than upgrade the judge model.
The research team used an LLM as a judge to identify the most and least strongly activated samples for each principal component, evaluate whether they suggest a transferable deception direction and assign scores, and select the highest-scoring subset of principal components to train the probe.
All Facts (6)
When switching to an LLM judge from another vendor, the official agent rankings were preserved, but the number of accepted papers doubled.
50%UnverifiedThe evaluation criteria of an LLM judge drift as model versions are updated, business rules are adjusted, and user behavior changes, and the drift itself does not proactively raise errors.
50%UnverifiedLLM judges perform close to coin-flipping when detecting omission-type errors.
50%UnverifiedThe LLM judge mechanism verifies the presence of information rather than its absence, so there is no clear anchor to trigger a check for omissions.
50%UnverifiedThe solution to missed detections is to restructure the task rather than upgrade the judge model.
50%UnverifiedThe research team used an LLM as a judge to identify the most and least strongly activated samples for each principal component, evaluate whether they suggest a transferable deception direction and assign scores, and select the highest-scoring subset of principal components to train the probe.
50%