RubricAtlas
An evaluation framework that automatically mines type-level analytical requirements from expert reports and organizes checks around recurring analytical questions. It is used to evaluate deep research agents.
Timeline (last 90 days)
RubricAtlas is an evaluation framework that automatically mines type-level analytical requirements from expert reports and organizes checks around recurring analytical questions.
RubricAtlas was built on 3,079 Chinese financial research reports, covering 99 research types.
RubricAtlas links binary checkpoints to specific passages in the reports and evaluates completeness and argumentative validity separately.
In tests on ten agents, the two review pipelines achieved a checkpoint agreement of κ = 0.71 and an agent ranking consistency of Kendall τ = 0.87.
Selecting requirements from different reports changes evaluation scores and agent rankings.
Controlled deletion experiments show that removing the relevant analysis from a report lowers the score of the corresponding dimension.
Filler experiments show that adding irrelevant content to a report produced no detected score increase.
The rubric measures analytical completeness rather than text length or surface richness.
RubricAtlas's construction corpus is concentrated on Chinese financial research reports, and its applicability to other domains and languages has not yet been concluded.
All Facts (9)
RubricAtlas is an evaluation framework that automatically mines type-level analytical requirements from expert reports and organizes checks around recurring analytical questions.
50%UnverifiedRubricAtlas was built on 3,079 Chinese financial research reports, covering 99 research types.
50%UnverifiedRubricAtlas links binary checkpoints to specific passages in the reports and evaluates completeness and argumentative validity separately.
50%UnverifiedIn tests on ten agents, the two review pipelines achieved a checkpoint agreement of κ = 0.71 and an agent ranking consistency of Kendall τ = 0.87.
50%UnverifiedSelecting requirements from different reports changes evaluation scores and agent rankings.
50%UnverifiedControlled deletion experiments show that removing the relevant analysis from a report lowers the score of the corresponding dimension.
50%UnverifiedFiller experiments show that adding irrelevant content to a report produced no detected score increase.
50%UnverifiedThe rubric measures analytical completeness rather than text length or surface richness.
50%UnverifiedRubricAtlas's construction corpus is concentrated on Chinese financial research reports, and its applicability to other domains and languages has not yet been concluded.
50%