Auto-Research Arena
A multi-objective benchmark for automated research systems that separates adoptability from improvement, and supports the evaluation of diverse research objectives within a shared experimental environment.
Timeline (last 90 days)
Auto-Research Arena is a multi-objective benchmark for evaluating automated research systems and discovering new ideas in language modeling.
The core design of Auto-Research Arena is to separate admissibility from improvement, where admissibility is determined by model quality and resource constraints, and improvement is measured against a chosen objective.
The researchers instantiated Auto-Research Arena in language model pretraining, defining nine objectives covering model, data, training, and inference efficiency.
The evaluation of Auto-Research Arena used seven research harnesses and two base models.
Evaluation results show that several complex methods were surpassed by a simple sequential search baseline.
In a new quality-driven search, reusing the findings revealed by Arena achieved lower training loss under matched data and FLOPs, and lower held-out BPB at the same model size.
The code for Auto-Research Arena has been open-sourced on GitHub at https://github.com/Auto-Research-Arena/auto-research-arena.
All Facts (7)
Auto-Research Arena is a multi-objective benchmark for evaluating automated research systems and discovering new ideas in language modeling.
50%UnverifiedThe core design of Auto-Research Arena is to separate admissibility from improvement, where admissibility is determined by model quality and resource constraints, and improvement is measured against a chosen objective.
50%UnverifiedThe researchers instantiated Auto-Research Arena in language model pretraining, defining nine objectives covering model, data, training, and inference efficiency.
50%UnverifiedThe evaluation of Auto-Research Arena used seven research harnesses and two base models.
50%UnverifiedEvaluation results show that several complex methods were surpassed by a simple sequential search baseline.
50%UnverifiedIn a new quality-driven search, reusing the findings revealed by Arena achieved lower training loss under matched data and FLOPs, and lower held-out BPB at the same model size.
50%UnverifiedThe code for Auto-Research Arena has been open-sourced on GitHub at https://github.com/Auto-Research-Arena/auto-research-arena.
50%