Can thermodynamics show what an AI learns about event order? A new measure found a signal—but not causal understanding
A peer-reviewed framework estimates time-direction asymmetry in autoregressive generators without enumerating every history. GPT-2 and Qwen3-4B-Base assigned larger sentence-level irreversibility to causally ordered text in limited tests, while token-level results were dominated by syntax.
The short version
- What happened
- The paper defines entropy production for autoregressive generators by comparing forward sequences with a backward process built from the same architectural components.
- Who it affects
- Researchers gain a potentially efficient probe for sequence directionality, but should report tokenization and syntax controls before interpreting it as evidence about a model's world representation.
- What remains unproven
- The language-model evidence is a proof of concept using GPT-2 and Qwen3-4B-Base, not a broad model survey or deployed evaluation.
Keep in mind
The language-model evidence is a proof of concept using GPT-2 and Qwen3-4B-Base, not a broad model survey or deployed evaluation.
Token-level entropy production was dominated by a syntactic artifact, and sentence-level controls cannot rule out all textual cues.
The method requires access to model probabilities and a backward construction, limiting direct use with black-box commercial chat systems.
No human participants, real-world interventions, safety outcomes, planning tasks or causal counterfactual tests were included.
The theoretical and empirical work is from one author and has not yet been independently replicated.
Source analysis · 3 linked sources. Primary analysed source date: 5 November 2026. Latest linked supporting source: 6 November 2026.
Read the full report · 9 min The evidence, methods, implications and limitations
The answer: the method detects direction in generated sequences, not an AI's understanding of cause
A peer-reviewed paper published in PRX Intelligence proposes a way to measure how strongly an autoregressive generator distinguishes the forward direction of a sequence from a constructed backward direction. The quantity is called entropy production, borrowing a formal idea from stochastic thermodynamics. In the paper's proof-of-concept language tests, sentence-level entropy production tended to be larger for text sets whose events had a causal order than for sets described as non-causal. The pattern appeared with GPT-2 and was checked again with the larger Qwen3-4B-Base model.
That result is narrower than saying a model understands cause and effect. The measure is computed from probabilities assigned to sequences and from a reverse-time construction. It can reveal statistical asymmetry: a model may judge one ordering more compatible with patterns learned from text than the reverse ordering. Training text contains many regularities about what usually happens first, which can produce that asymmetry without the model representing mechanisms, interventions or counterfactual consequences in the way causal inference requires.
The authors themselves identify an important warning. At token level, much of the measured entropy production was dominated by a syntactic artifact. Punctuation, capitalization and the conventions of beginnings and endings can make a reversed token sequence obviously unlike ordinary language. Moving to sentence-level comparisons reduced that particular problem, but it did not turn the experiment into a general test of a world model. The paper establishes a mathematical framework and an illustrative signal, not a certificate of reasoning.[1][2]
What entropy production means in this setting
Autoregressive systems generate one output after another. A Transformer assigns a distribution to the next token from a deterministic summary of the preceding tokens; a recurrent network updates a latent state; a Kalman filter produces innovations from past measurements. The observed sequence can depend on the full history even when the hidden computation is deterministic. That makes the observed process non-Markovian: the next visible element cannot generally be described using only the immediately preceding visible element.
Stochastic thermodynamics often measures irreversibility by comparing the probability of a forward trajectory with the probability of an associated backward trajectory. A large difference means the process has a stronger arrow of time. Applying that idea to a language model is not a claim that tokens dissipate heat. It is a mathematical comparison between path probabilities. The paper constructs a backward generator using the same class of architectural components in reverse temporal order, then defines entropy production from the log ratio between the forward and backward path measures.
The framework also decomposes the total into non-negative contributions for individual steps. Those terms are interpreted through retrospective inference and split further into compression loss and model mismatch. This decomposition matters because one large total can hide different sources of asymmetry. A researcher can ask whether the model's summary of the past discarded information relevant to looking backward, or whether the backward model's probabilities simply failed to match the reverse-time distribution.[1]
The distinction that matters
What the new measure compares
Each step adds evidence about whether the generated path has a preferred direction, but none of the steps alone establishes causal understanding.
Forward path
Score the generated sequence
Probability under the autoregressive model
Backward path
Apply the reverse-time construction
A comparison probability
Log ratio
Estimate entropy production
Sequence irreversibility
Controls
Remove syntax and test causal ordering
Interpretability remains conditional
The computation avoids enumerating every history, but still needs access to probabilities
A direct calculation over every possible history would become intractable as sequence length and vocabulary grow. The paper derives an estimator based on sampled trajectories and the output probabilities produced along them. In the authors' analysis, this avoids an exponential sampling cost associated with listing all histories. The contribution is therefore partly practical: it turns an abstract definition into something that can be estimated for modern sequence models when researchers can inspect their probability distributions.
The access condition is consequential. Many commercial chat interfaces return generated text but not complete next-token probabilities, internal states or a separately trainable backward process. The method cannot automatically be applied to those black-box services. Even for an open model, sampling, scoring and fitting the reverse construction still require compute. Avoiding exponential enumeration does not mean the analysis is free or that one score can be compared fairly across every tokenizer, model size and prompt format.
The paper checks the theory in a linear Gaussian case, where the autoregressive system reduces to the Kalman innovation representation and the entropy production has an analytical expression. Agreement there supports the mathematical construction. The language-model experiment answers a different question: whether a measurable asymmetry appears in generated or scored text. It does not validate a downstream diagnostic for hallucination, planning reliability, safety or truthfulness.[1]
How the language test should be read
The first language experiment used GPT-2, a roughly 124-million-parameter pretrained Transformer in the implementation described by the authors. They sampled and evaluated sequences with the Hugging Face implementation on NVIDIA T4 or V100 hardware. Exact token-by-token reproduction is not guaranteed across hardware and software even with a fixed seed. The paper then reports a re-evaluation with Qwen3-4B-Base, a substantially larger base model, to check whether the sentence-level tendency was peculiar to GPT-2.
For sentence-level comparisons, the analysis contrasts text sets with an intended causal ordering against non-causal sets and asks whether forward-versus-backward asymmetry differs. A higher value for the causally ordered group is compatible with the model encoding regularities about event order. But the denominator is a designed collection of texts rather than people, physical interventions or an independently measured causal system. The study does not ask the model to predict what would happen if an event were prevented, or to distinguish correlation from intervention.
The token-level artifact illustrates why such diagnostics need controls. Ordinary text has strong directional conventions: sentences begin with capitals, punctuation ends clauses, and word fragments produced by a tokenizer are not interchangeable under reversal. A metric can detect those conventions and look impressive while missing the scientific property of interest. The authors' move to sentence-level analysis and larger-model check helps, but further adversarial controls and independent replication are needed before the measure can be interpreted as evidence about learned world structure.[1]
What researchers and practitioners can use now
For model researchers, the work provides a new probe that can sit beside behavioural benchmarks, representation analysis and causal tests. It may help compare how architectural choices or training data change sequence irreversibility, provided the same tokenizer, sampling protocol and backward construction are used. The per-step decomposition could help locate where an asymmetry enters a sequence rather than collapsing an entire answer into one number.
For practitioners, the immediate implication is methodological restraint. A product team should not label a model as having a world model because it scores highly on this measure. Nor should a low value be interpreted as absence of useful temporal knowledge. Before a diagnostic informs deployment, it would need demonstrated links to operational outcomes such as detecting reordered procedures, anticipating consequences, maintaining plans or avoiding unsafe action sequences. Those links were not tested.
The research was supported by the JST ERATO programme and the University of Tokyo's Beyond AI initiative, according to the university release. It is a single-author theoretical and computational paper. Peer review strengthens confidence that the framework has passed journal scrutiny, but it does not replace independent reproduction. The accompanying Japanese institutional releases accurately identify the paper and intended contribution; they are not separate evidence of performance.[1][2][3]
What would change the assessment
Confidence would rise if independent teams reproduce the estimator across model families, tokenizers, languages and non-language autoregressive systems, with preregistered controls that remove punctuation and formatting cues. A useful next step would compare entropy production with established tests of temporal order, causal intervention and planning, asking whether the new score predicts any capability that matters outside the constructed text sets.
The assessment would become more consequential if the measure identified failures before they appeared in real tasks—for example, warning that a system will reverse clinical steps, break a manufacturing sequence or lose the dependency structure of a long plan. That would require prospective evaluation and clear false-positive and false-negative rates. A score that merely tracks text style would not justify operational use.
For now, the durable result is the framework: a way to bring non-Markovian autoregressive generators into a stochastic-thermodynamic description and estimate an arrow-of-time quantity from sampled paths. The language result is a proof of concept with an honest caveat. It suggests that learned text statistics contain detectable event-order asymmetry, while leaving open the much harder question of whether a model understands the world that produced those statistics.[1][2][3]