Why do diffusion models stop copying and start inventing? A study finds a middle phase
Across four image datasets, researchers found recurring outputs that copied neither the training set nor ordinary one-off generations—but the controlled benchmark does not establish how often commercial systems reproduce creative work.
The short version
- What happened
- Researchers trained grids of diffusion models on MNIST, Fashion-MNIST, CIFAR-10 and LSUN-Church while increasing the training-set size, then classified outputs as memorised, intermediate or generalised.
- Who it affects
- Artists, publishers and model users need audits that distinguish direct training-data copies from repeatable model-made patterns; this study offers a testable framework, not an estimate of how often either occurs in deployed products.
- What remains unproven
- The dense experiments used small, controlled image benchmarks and relatively compact diffusion models, not present-day commercial systems.
Keep in mind
The dense experiments used small, controlled image benchmarks and relatively compact diffusion models, not present-day commercial systems.
Nearest-neighbour thresholds were selected partly through manual inspection and depend on the pixel or perceptual distance metric.
The Stable Diffusion 1.4 analysis was qualitative and used candidate prompts drawn from prior studies.
The paper does not measure copyright infringement, harmful content, user outcomes or the frequency of copying in a deployed service.
Source analysis · 2 linked sources. Primary analysed source date: 25 October 2026. Earliest linked supporting source: 27 May 2025.
Read the full report · 4 min The evidence, methods, implications and limitations
Between a copy and a novel image, the model formed repeatable attractors
The peer-reviewed study treats a diffusion model as a form of associative memory and asks what changes as it sees more training examples. On four image datasets—MNIST, Fashion-MNIST, CIFAR-10 and LSUN-Church—the team trained separate models across roughly 38 increasing dataset sizes, from very small subsets towards totals of 50,000 to 126,227 images. Generated pools were usually four times the corresponding training set; a CIFAR-10 repeat used eight times as many generated images.
Outputs were labelled memorised when they closely matched a training image, generalised when they were unique in both the training and generated pools, and 'spurious' when they recurred in the generated pool but were absent from training. As dataset size grew, copying fell and generalisation rose. The recurring middle category peaked near that boundary, and moving the model's width shifted where the transition happened. In this paper, 'spurious' describes a stable generated pattern; it does not mean a factual error or unsafe output.[1][2]
The finding may sharpen audits, but it is not a product risk score
The researchers measured nearest-neighbour distances using pixel-space distance for the simpler datasets and a learned perceptual metric for CIFAR-10 and LSUN-Church. They also compared the curvature of the model's energy landscape, finding sharper basins for memorised outputs, intermediate curvature for the recurring middle states and flatter geometry for generalised outputs. Up to 512 examples per category were used in the reported curvature calculations. A qualitative Stable Diffusion 1.4 check was much smaller and based on candidate prompts from earlier work.
For model builders, the practical message is that 'not a training-set duplicate' does not necessarily mean a model has learned a broad, smooth distribution: it may have created another narrow pattern that repeats. Audits could therefore compare outputs with both training data and other generations. But the thresholds were manually tuned with histogram and image inspection, the benchmarks are small, and no creator, consumer or deployment outcome was measured. Larger blinded audits on current production models, with pre-registered thresholds and independent data, would show whether the mechanism predicts real copying, diversity or safety failures.[1][2]
The distinction that matters
Three output states in the study
The classification depends on comparisons with both the training set and a larger pool of generated images.
Memorised
Repeated among generations and close to a training image
Evidence of a copied training pattern
Intermediate
Repeated among generations but absent from training
A stable new attractor, termed spurious by the authors
Generalised
No close duplicate in either comparison pool
A unique sample under the study's distance threshold