TinyStories
A dataset of short children's stories written in simple English, designed for training and evaluating small language models. It provides a controlled benchmark environment widely used in language model research to study emergent capabilities and performance of compact architectures.
Timeline (last 90 days)
The paper evaluated three configurations with 10.94M parameters each on the TinyStories dataset: pure Transformer, pure wave, and hybrid wave-attention, with context lengths up to 4096 tokens.
TinyStories is a synthetic story dataset released by Microsoft, designed specifically for small models, with simple grammar but complete semantics.
On language modeling tasks (Text8, TinyStories, OpenWebText), binary diffusion remains competitive but does not significantly surpass the baselines.
The publicly released VKORE-46M-NPU is a 46M-parameter, character-level model trained on the TinyStories dataset, and its purpose is to verify the feasibility of the training path rather than model quality.
The benchmark uses llama2.c locked to a specific commit, paired with TinyStories FP32 checkpoints (15M, 42M, 110M parameters); these are small models running on llama2.c rather than the full Llama 2.
The study validates on three datasets—TinyStories, WikiText-103, and CNN/DailyMail—scaling each corpus from 512 chunks to 2,048 chunks, with each chunk consisting of 64 tokens.
All Facts (6)
The paper evaluated three configurations with 10.94M parameters each on the TinyStories dataset: pure Transformer, pure wave, and hybrid wave-attention, with context lengths up to 4096 tokens.
50%UnverifiedTinyStories is a synthetic story dataset released by Microsoft, designed specifically for small models, with simple grammar but complete semantics.
50%UnverifiedOn language modeling tasks (Text8, TinyStories, OpenWebText), binary diffusion remains competitive but does not significantly surpass the baselines.
50%UnverifiedThe publicly released VKORE-46M-NPU is a 46M-parameter, character-level model trained on the TinyStories dataset, and its purpose is to verify the feasibility of the training path rather than model quality.
50%UnverifiedThe benchmark uses llama2.c locked to a specific commit, paired with TinyStories FP32 checkpoints (15M, 42M, 110M parameters); these are small models running on llama2.c rather than the full Llama 2.
50%UnverifiedThe study validates on three datasets—TinyStories, WikiText-103, and CNN/DailyMail—scaling each corpus from 512 chunks to 2,048 chunks, with each chunk consisting of 64 tokens.
50%