FineWeb-Edu
A high-quality web text dataset released by Hugging Face in 2024. It is filtered from CommonCrawl for educational content, retaining high-quality samples via classifier scoring, and is suitable for training small language models.
Timeline (last 90 days)
On the FineWeb-Edu dataset, recurrent UDLM reaches the same validation NLL as non-recurrent UDLM using only about 1/3.3 of the non-recurrent model's FLOPs.
The paper conducts experiments on the FineWeb-Edu dataset at four model scales, from 60M to 1B parameters.
The model is trained in two stages: first masked language model pre-training on about 20 million sentences from FineWeb-Edu, then correction fine-tuning on 12.02 million (noisy, clean) pairs, of which 30% are identity pairs.
The model trained using Edu-QuRating-filtered mixed sets achieved a higher overall observed accuracy than the FineWeb-Edu baseline model across nine benchmark tests, with performance gains concentrated on specific tasks.
HuggingFace released the FineWeb-Edu dataset in 2024, scoring the educational value of web page content on a scale of 0–5 by training a classifier.
In controlled single-run pre-training comparison experiments, the model trained on the Edu-QuRating-curated mixture achieved higher overall observed accuracy across nine benchmarks than the FineWeb-Edu baseline model, with performance gains concentrated on specific tasks.
All Facts (6)
On the FineWeb-Edu dataset, recurrent UDLM reaches the same validation NLL as non-recurrent UDLM using only about 1/3.3 of the non-recurrent model's FLOPs.
50%UnverifiedThe paper conducts experiments on the FineWeb-Edu dataset at four model scales, from 60M to 1B parameters.
50%UnverifiedThe model is trained in two stages: first masked language model pre-training on about 20 million sentences from FineWeb-Edu, then correction fine-tuning on 12.02 million (noisy, clean) pairs, of which 30% are identity pairs.
50%UnverifiedThe model trained using Edu-QuRating-filtered mixed sets achieved a higher overall observed accuracy than the FineWeb-Edu baseline model across nine benchmark tests, with performance gains concentrated on specific tasks.
50%UnverifiedHuggingFace released the FineWeb-Edu dataset in 2024, scoring the educational value of web page content on a scale of 0–5 by training a classifier.
50%UnverifiedIn controlled single-run pre-training comparison experiments, the model trained on the Edu-QuRating-curated mixture achieved higher overall observed accuracy across nine benchmarks than the FineWeb-Edu baseline model, with performance gains concentrated on specific tasks.
50%