datasets

An open-source Python library released by Hugging Face for efficiently loading and preprocessing large-scale datasets. It provides a unified interface for accessing datasets hosted on the Hugging Face Hub as well as local data files, and supports memory-mapped storage and streaming modes, allowing datasets larger than available RAM or disk to be processed incrementally. The library integrates with common machine learning frameworks and includes utilities for mapping, filtering and formatting data prior to model training.