LIMA
A study on the scale of supervised fine-tuning data for large language models. It found that a small number of high-quality examples is sufficient to achieve good results, and that increasing the number of examples yields limited quality improvement.
Timeline (last 90 days)
SFT mainly teaches the model format and persona rather than knowledge; LIMA expanded examples from 2,000 to 32,000, with almost no change in quality.
The 2023 Microsoft Research paper 'LIMA: Less Is More for Alignment' showed that using only 1,000 carefully curated high-quality samples can enable a base model to exhibit conversational capabilities comparable to fine-tuning on tens of thousands of samples.
All Facts (2)
SFT mainly teaches the model format and persona rather than knowledge; LIMA expanded examples from 2,000 to 32,000, with almost no change in quality.
50%UnverifiedThe 2023 Microsoft Research paper 'LIMA: Less Is More for Alignment' showed that using only 1,000 carefully curated high-quality samples can enable a base model to exhibit conversational capabilities comparable to fine-tuning on tens of thousands of samples.
50%