FineWeb
A large-scale open-source web-text pretraining dataset released by Hugging Face, filtered and cleaned from Common Crawl. It totals approximately 44 TB and 15 trillion tokens, intended for use in large language model pretraining.
Timeline (last 90 days)
FineWeb is a large-scale high-quality web text dataset released by HuggingFace.
The FineWeb public dataset contains approximately 44 TB of text data, equivalent to approximately 15 trillion tokens.
FineWeb is a 15T-token high-quality dataset extracted from Common Crawl by HuggingFace, Dolma is a 3T-token dataset curated by AI2, and RedPajama is approximately 1.2T tokens
预训练阶段从堆算力堆数据的 Scaling Laws 研究逐渐转向数据质量优化，如 RedPajama、FineWeb 等高质量语料库的出现
All Facts (4)
FineWeb is a large-scale high-quality web text dataset released by HuggingFace.
50%UnverifiedThe FineWeb public dataset contains approximately 44 TB of text data, equivalent to approximately 15 trillion tokens.
50%UnverifiedFineWeb is a 15T-token high-quality dataset extracted from Common Crawl by HuggingFace, Dolma is a 3T-token dataset curated by AI2, and RedPajama is approximately 1.2T tokens
50%Unverified预训练阶段从堆算力堆数据的 Scaling Laws 研究逐渐转向数据质量优化，如 RedPajama、FineWeb 等高质量语料库的出现
50%