CriticGen
A fine-grained, generation-aware LLM evaluation framework that dynamically generates sample-level scoring dimensions and rubrics, converting evaluation results directly into actionable improvement feedback.
Timeline (last 90 days)
The core argument of the paper is that fine-grained evaluation must simultaneously be instance-specific and actionable.
CriticGen is conditioned on a dynamic scoring rubric and jointly produces four outputs: scores, rationales, actionable improvement suggestions, and improved answers.
CriticGen's dynamic scoring criteria are organized under three high-level categories: subjective constraints, objective constraints, and self-derived constraints.
The relevance and coverage of scoring rubrics generated by CriticGen improved from 3.33/4.03 to 3.97/4.24, respectively.
CriticGen's scores achieved a Pearson correlation coefficient of 0.9556 and a Spearman correlation coefficient of 0.9560 with human judgments.
CriticGen is a fine-grained, generation-aware evaluation framework that generates dedicated evaluation dimensions and scoring criteria for each sample, and jointly produces scores, rationales, actionable improvement suggestions, and improved answers.
Experiments show that CriticGen improved 73.17% of answers while maintaining a non-degradation rate of 93.28%.
The structured feedback produced by CriticGen can be used to automatically generate preference pairs in alignment training methods such as DPO or KTO, using the original answer as the rejected sample and the revised answer as the chosen sample.
Methods such as CRITIC, Self-Refine, and Reflexion have emerged in the field of self-correction, and CriticGen can be viewed as the latest development in this technical lineage.
CriticGen's F1 scores on criteria-based rationales and actionable suggestions improved from 0.6369/0.5994 to 0.7554/0.7900, respectively.
1 more timeline events
All Facts (11)
The core argument of the paper is that fine-grained evaluation must simultaneously be instance-specific and actionable.
50%UnverifiedCriticGen is conditioned on a dynamic scoring rubric and jointly produces four outputs: scores, rationales, actionable improvement suggestions, and improved answers.
50%UnverifiedThe relevance and coverage of scoring rubrics generated by CriticGen improved from 3.33/4.03 to 3.97/4.24, respectively.
50%UnverifiedCriticGen's scores achieved a Pearson correlation coefficient of 0.9556 and a Spearman correlation coefficient of 0.9560 with human judgments.
50%UnverifiedCriticGen is a fine-grained, generation-aware evaluation framework that generates dedicated evaluation dimensions and scoring criteria for each sample, and jointly produces scores, rationales, actionable improvement suggestions, and improved answers.
50%UnverifiedExperiments show that CriticGen improved 73.17% of answers while maintaining a non-degradation rate of 93.28%.
50%UnverifiedThe structured feedback produced by CriticGen can be used to automatically generate preference pairs in alignment training methods such as DPO or KTO, using the original answer as the rejected sample and the revised answer as the chosen sample.
50%UnverifiedMethods such as CRITIC, Self-Refine, and Reflexion have emerged in the field of self-correction, and CriticGen can be viewed as the latest development in this technical lineage.
50%UnverifiedCriticGen's F1 scores on criteria-based rationales and actionable suggestions improved from 0.6369/0.5994 to 0.7554/0.7900, respectively.
50%UnverifiedCriticGen's dynamic scoring criteria are organized under three high-level categories: subjective constraints, objective constraints, and self-derived constraints.
50%UnverifiedarXiv published the paper "CriticGen: Generation-Aware Evaluation as Actionable Feedback."
50%