Synthetic Training Data and Model Collapse: Where Self-Generated Data Helps and Where It Hurts

The supply of high-quality human-written text is finite, and frontier training runs are approaching it. Epoch AI’s 2024 analysis estimated the effective stock of quality public text at around 300 trillion tokens (90% interval: 100 to 1,000 trillion) and projected that, at current trends, models would fully use it somewhere between 2026 and 2032, earlier if heavily overtrained (Villalobos et al., Epoch AI). Synthetic data, generated by models for models, is the obvious response. The obvious worry is model collapse. Both the promise and the risk are real, and the research of the past two years makes it possible to say fairly precisely which conditions produce which outcome.

What model collapse is

The canonical result is Shumailov et al., “AI models collapse when trained on recursively generated data” (Nature, 2024). The authors trained successive generations of models, each on data produced by the previous one, and found that indiscriminate training on model-generated content causes irreversible defects. They describe two phases:

• Early collapse: the model loses information about the tails of the original distribution, the rare events and minority patterns.
• Late collapse: the model converges on a distribution that bears little resemblance to the original, often with sharply reduced variance.

The mechanism is not specific to language models. The paper reproduces it in variational autoencoders and Gaussian mixture models, and attributes it to compounding errors: statistical approximation error from finite sampling (low-probability events are under-sampled and then vanish), plus errors from limited model expressivity and imperfect learning procedures. Each generation inherits the previous one’s blind spots and adds its own.

Alemohammad et al. (ICLR 2024) reached a similar conclusion for image generators, naming the effect Model Autophagy Disorder. Their central finding: without enough fresh real data in each generation of a self-consuming loop, quality or diversity progressively declines, and the decline appears within a few generations. Sampling choices that favour quality, such as low-temperature or truncated sampling, accelerate the loss of diversity.

Why the setup matters: replace versus accumulate

The collapse experiments share a design choice. Each generation’s training data replaces the previous generation’s rather than adding to it. Gerstgrasser et al. (2024) asked what happens under the more realistic assumption that synthetic data accumulates alongside the original real data.

Their result is the most important qualification in this literature. Replacement does produce collapse. Accumulation does not: across language models, diffusion models for molecule generation, and variational autoencoders, test error stayed bounded as generations increased. In a linear-model setting they prove that with accumulation, error has a finite upper bound independent of the number of iterations.

The disagreement has not fully closed. Work led by Elvis Dohmatob, including “Model Collapse Demystified: The Case of Regression” and “A Tale of Tails”, argues that synthetic data still changes how error scales, and that adding it is not free even when collapse is avoided. The defensible summary is narrower than either headline: discarding real data in favour of model output is reliably harmful; mixing model output into a retained real corpus is not automatically catastrophic, but its effect depends on proportion, quality, and how the synthetic data was filtered.

The open-web version of the problem

Labs do not deliberately run replacement loops, but the web may run one for them. As model-generated text spreads online, crawls taken later contain a growing share of it, frequently unlabelled. Shumailov et al. note that the value of data collected from genuine human interaction rises as that happens. The practical defences are data provenance (pre-generative-AI snapshots retain value), filtering for model-generated text where it can be detected, and keeping curated human corpora rather than refreshing them wholesale.

Where synthetic data helps

The productive uses of synthetic data share one property: something other than the generating model decides what is kept.

Verified outputs

The Llama 3 team reported that training their 405B model on its own unfiltered generations for code was not helpful and could degrade performance. What worked was using execution as a source of truth: roughly a million synthetic coding dialogues were passed through static analysis, linters, and model-written unit tests run in containers, with failing solutions revised and only fully passing examples kept for supervised fine-tuning (The Llama 3 Herd of Models). The verifier injects information the model did not have, which is exactly what a collapse loop lacks.

The same logic applies wherever an external checker exists: mathematical answers with known solutions, formal proofs checked by a proof assistant, code with tests, and structured outputs that can be validated against a schema.

Distillation and targeted coverage

A stronger model generating data for a smaller one is not a recursive loop; the student learns from a better distribution than its own. Microsoft’s phi-1 work showed that a small code model trained largely on curated and synthetic “textbook-quality” data could perform well beyond its size on coding benchmarks (Gunasekar et al., 2023). Synthetic data is also effective for filling known gaps: rare formats, long-context tasks, low-resource languages, or instruction styles thinly represented in natural data.

Preference and alignment data

Model-generated responses ranked by humans, reward models, or rule-based checks are now standard in post-training, including the preference pipelines described in our RLHF explainer. The human or reward signal is the external information.

Where it hurts

• Unverified self-training at scale. Training a model on large volumes of its own unfiltered output, with real data removed or diluted, is the documented collapse condition.
• Tail-dependent tasks. Collapse starts in the tails. Applications that depend on rare cases, such as fraud, medical edge cases, minority dialects, or unusual code paths, lose the most when synthetic data crowds out real examples.
• Diversity-reducing generation. Low-temperature sampling and aggressive quality filtering produce clean but homogeneous data. Without deliberate diversity controls, synthetic corpora narrow the output distribution.
• Evaluation contamination. Synthetic data generated by a model that has seen a benchmark can leak that benchmark into training, inflating scores in ways covered in our piece on benchmark contamination.

A working policy for teams

Keep the original real corpus and add synthetic data to it rather than substituting for it. Require an external signal (execution, a proof checker, a reward model, or human review) for any synthetic example that teaches a capability rather than a format. Track provenance so synthetic share is known per dataset and per training run. Measure diversity and tail performance directly, not only aggregate benchmark scores, because collapse shows up there first. And treat a rising synthetic fraction as a parameter to be validated with ablations, not a free source of tokens.