Can AI teams estimate the energy cost of every generated token?
Across 374 held-out executions on NVIDIA GH200 and GB300 systems, a memory-traffic model stayed within 3.73% on average—but only for a narrow set of dense-model decoding conditions.
The short version
- What happened
- The researchers tested Qwen2.5-7B, Qwen2.5-14B and Mistral-7B on NVIDIA GH200 and GB300 hardware, including one two-GPU configuration.
- Who it affects
- More transparent serving estimates could help organizations choose lower-energy configurations, but incomplete accounting can still understate the electricity and emissions associated with a service.
- What remains unproven
- The unreviewed work covers a small set of dense models and NVIDIA accelerators under controlled decoding conditions; it excludes several major system and lifecycle energy sources.
Keep in mind
The unreviewed work covers a small set of dense models and NVIDIA accelerators under controlled decoding conditions; it excludes several major system and lifecycle energy sources.
Source analysis · 1 linked source. Primary analysed source date: 20 October 2026.
Read the full report · 3 min The evidence, methods, implications and limitations
Memory traffic predicted decode energy better than shortcuts
Researchers from Texas Tech, the Texas Advanced Computing Center and NVIDIA propose a falsifiable way to estimate the GPU energy used for each generated token. Their model links energy to high-bandwidth-memory traffic, model size, sequence length and fixed overheads. It was fitted and tested across Qwen2.5 models with 7 and 14 billion parameters and Mistral-7B on GH200 and GB300 hardware, with five executions for each measured cell.
The complete model was tested on 374 held-out energy executions and 177 held-out memory profiles. Across held-out condition averages, predictions stayed within 3.73%; on the GB300 tests highlighted by the authors, the largest mean cell error was 2.14%. Models using only elapsed time or only bytes transferred exceeded 10% error in those comparisons, suggesting that both memory movement and fixed costs matter.[1]
The result is a component estimate, not a data-centre footprint
For developers and infrastructure teams, a validated per-token estimate could help compare serving configurations before running a full energy audit. It might also make energy reporting more specific than a single model-wide average. But the paper measures GPU-board energy during autoregressive decoding; it excludes host processors, networking, storage, cooling, training and embodied emissions.
The tested workload was dense BF16 inference with full-history decoding. Multi-node systems, pipeline parallelism, mixture-of-experts routing and other model families were not covered. One author lists NVIDIA, the hardware supplier, as an affiliation, and the preprint gives no separate conflict statement. Independent replication on additional hardware and end-to-end facility measurements would change the assessment.[1]
The distinction that matters
Where a per-token energy estimate begins and ends
The proposed law models one part of the serving stack.
Weights and cache
Estimate memory bytes moved
Captures sequence and model-size effects
GPU board
Combine traffic with fixed overhead
Predicts decode energy per token
Whole system
Add CPU, network and cooling
Not measured in this study