Eight vision-language models read charts differently from people—why does that matter?
Across six assessments, models were generally less accurate than people and made different patterns of mistakes. The study is broad and peer reviewed, but its model snapshot dates from 2024.
The short version
- What happened
- Researchers tested eight vision-language models on six chart assessments containing 851 items; every model received every item ten times, producing 8,510 responses per model.
- Who it affects
- Workers using AI to interpret dashboards should verify source values and calculations rather than assuming a fluent explanation reflects human-like visual understanding.
- What remains unproven
- The evaluated model snapshot is from April–May 2024 and excludes current systems.
Keep in mind
The evaluated model snapshot is from April–May 2024 and excludes current systems.
Six assessments improve breadth but do not represent every workplace chart, language, accessibility need or decision consequence.
The study measures task performance, not whether model assistance improves or harms real human decisions.
Source analysis · 2 linked sources. Primary analysed source date: 19 October 2026. Earliest linked supporting source: 22 May 2025.
Read the full report · 3 min The evidence, methods, implications and limitations
A broad comparison with human answers
A peer-reviewed study has found that eight vision-language models not only trailed people on chart questions overall but also made errors unlike the errors people made. The researchers combined six assessments covering basic graph reading, literacy, misleading charts, higher-order reasoning and natural-language chart questions. The suite contained 851 items; each model answered every item ten times, producing 8,510 responses per model.
The human comparison drew on existing assessment samples—1,135 people for two tests, 497 for CALVI, 531 for HOLF and 1,743 for HOLF-Multi—plus 50 recruited participants who each answered 25 ChartQA items. GPT-4V was the best model on five of six assessments when valid responses were compared, but people still led across value identification, arithmetic and statistical inference. No model approached the paper's human noise ceiling for matching human error patterns.[1][2]
Invalid answers and an old model snapshot matter
Formatting failures changed the picture. One LLaVA model produced valid answers for only 2,735 of 8,510 prompts; MatCha produced 7,082. Counting only valid answers raised reported accuracy, especially on multiple-choice tasks. That distinction matters wherever an automated system must deliver a usable answer, not merely demonstrate latent knowledge.
The main limit is time: BLIP2, three LLaVA variants, Pix2Struct, MatCha and GPT-4V were evaluated in April and May 2024. The findings cannot rank current systems. They instead provide a reusable human-centred method. New evidence should rerun the same assessments on current models, pre-register scoring, report accessibility and real-workflow outcomes, and test whether review tools help people catch the distinctive mistakes.[1][2]
The distinction that matters
What the evaluation separates
Accuracy alone does not show whether a model understands a chart like a person.
Accuracy
Score 851 chart questions
Compare model and human correctness
Validity
Count unusable or malformed answers
Avoid flattering valid-only scores
Error pattern
Compare which items are missed
Test whether mistakes resemble human mistakes