Synthetic Data

Synthetic data refers to data artificially generated through algorithms rather than collected directly from the real world; it simulates the statistical characteristics of real data distributions but does not correspond to real individuals or events.

A common approach for generating synthetic data is to have a model rewrite, expand, or role-shift based on real user conversations, producing training samples that are fictitious in form but retain the semantic information of the original.

When language models are repeatedly trained on data generated by themselves or similar models, systematic degradation occurs, with the diversity of model outputs decreasing with each generation and the tail information of the distribution gradually being lost.

If the synthetic data generation process omits correlations between key variables, models trained on such data often perform poorly in real-world deployment environments.

合成数据在长尾场景补充和隐私敏感场景替代两个维度上价值最高，而非作为主体数据源

Research published in 2023, such as "The Curse of Recursion," shows that when models are repeatedly trained on their own generated data, the data distribution gradually degrades, diversity continuously decreases, and model capabilities systematically decline (model collapse).

Procedural synthetic data can generate on demand edge-case scenarios that are difficult to collect in reality—such as arbitrary-intensity glare, low illumination, and heavy occlusion—and provides precise annotations.

Procedural synthetic data provides robotics and autonomous driving developers with a low-cost, high-precision, and infinitely scalable data acquisition path

The industry generally views synthetic data as a supplement to, rather than a replacement for, real corpora.

The robustness of synthetic-data-driven models in real-world scenarios, their coverage of accents and dialects, and audio quality degradation over long-term use all still require more validation.

The study shows that even for languages with scarce high-resource data, advanced speech interaction systems can be developed through modular architecture and synthetic data.

The study was validated on synthetic data mirroring the real road network and has not yet entered a real public deployment stage.

Downstream fine-tuning experiments are a key step in testing the practical utility of synthetic data: synthetic data must improve model performance on real medical dialogue tasks to prove its effectiveness, not merely approach on surface-level metrics.

The synthetic data are generated based on an idealized noise model, whereas the real radio environment contains complex factors such as radio frequency interference (RFI), non-Gaussian noise, and instrumental gain fluctuations, which are difficult to fully reproduce in synthetic data.

Because financial data is highly sensitive, developers can often only use desensitized, sampled, or synthetic data, and the joint distribution of such data and production data may differ systematically.

The difficulty of real-time speech translation lies not only in the model; accents, background noise, multiple people talking over each other, and technical terminology all affect the final experience.

The key challenges facing the synthetic data approach are how to strike a balance between synthetic and real data, and how to design filtering and validation mechanisms.

The author believes one reason the 50-example fine-tune counterintuitively outperformed the larger-data fine-tune is that the AI-generated synthetic complexity labels are inconsistent, and the more data there is, the more opportunities there are for noise to be amplified.

A ten-million-row synthetic dataset with no lineage is strictly worse than a hundred-thousand-row dataset in which every sample is traceable, because only the latter is debuggable.

When reviewing samples that mix real and synthetic text, human annotators anchor on the surface fluency of synthetic text, thereby systematically favoring synthetic samples.