Reproducibility Crisis
The phenomenon in scientific research whereby other researchers find it difficult to reproduce published results; in machine learning it is often caused by issues such as selective reporting, p-hacking and benchmark overfitting.
Core Facts
Timeline (last 90 days)
The reproducibility crisis in AI often stems from issues such as opaque data preprocessing, unfixed random seeds, and incomplete reporting of the hyperparameter search space.
Multiple surveys show that the results of many published studies cannot be independently reproduced, with the absence of code and environment configuration being one of the main reasons.
Selective reporting systematically exaggerates a method's actual effectiveness and is one of the important causes of the 'reproducibility crisis' in machine learning.
The lack of rigorous replication validation for a massive volume of papers has led to a reproducibility crisis, with incorrect or irreproducible conclusions entering the literature database.
All Facts (5)
In 2015, the Open Science Collaboration attempted to replicate 100 psychology studies, and only about 36% were successfully replicated.
80%UnverifiedThe reproducibility crisis in AI often stems from issues such as opaque data preprocessing, unfixed random seeds, and incomplete reporting of the hyperparameter search space.
50%UnverifiedMultiple surveys show that the results of many published studies cannot be independently reproduced, with the absence of code and environment configuration being one of the main reasons.
50%UnverifiedSelective reporting systematically exaggerates a method's actual effectiveness and is one of the important causes of the 'reproducibility crisis' in machine learning.
50%UnverifiedThe lack of rigorous replication validation for a massive volume of papers has led to a reproducibility crisis, with incorrect or irreproducible conclusions entering the literature database.
50%