Did ChatGPT teach consultants data science? It lifted task scores, but the knowledge did not stick
In a randomized experiment, GPT-4 helped generalist consultants approach expert performance on three technical tasks. A test immediately afterwards found no clear gain in unaided technical knowledge.
The short version
- What happened
- A randomized experiment analysed 487 BCG consultants after 986 were invited; participants completed two of three 90-minute data-science tasks, with 44 BCG data scientists providing an expert benchmark.
- Who it affects
- Workers may complete unfamiliar technical tasks more effectively with AI, but employers should not relabel temporary tool-assisted output as a learned professional skill.
- What remains unproven
- 487-person analysis sample drawn from one highly educated consulting workforce
Keep in mind
487-person analysis sample drawn from one highly educated consulting workforce
Short tasks and an immediate post-test cannot establish durable learning
No live client work, job-transition outcome or end-to-end data-science assessment
The study ran in March and April 2024 using GPT-4-era tooling
BCG and OpenAI affiliations and direct study involvement require consideration
Source analysis · 1 linked source. Primary analysed source date: 26 October 2026.
Read the full report · 3 min The evidence, methods, implications and limitations
GPT-4 closed much of the task-performance gap
A peer-reviewed randomized experiment found that GPT-4 acted as a strong temporary performance aid for generalist consultants tackling work normally outside their expertise. Boston Consulting Group invited 986 consultants; 573 started and 487 formed the analysis sample. Each completed two of three 90-minute tasks—coding, statistical understanding and prediction—while 44 BCG data scientists supplied an expert benchmark.
The treatment group received 20 minutes of training plus ChatGPT; the control group received matched training and could use public online resources. ChatGPT access increased coding scores by 49 percentage points versus control (95% confidence interval 42–56), statistics scores by 20 points (15–25), and prediction similarity to expert answers by 17 points (9–25). Treated consultants still remained below the data-scientist average on coding and statistics.[1]
Better output was not the same as learning
The important limit came immediately afterwards: researchers found no evidence that treated participants had gained more technical knowledge when answering without ChatGPT. In an exploratory measure, they were also worse at judging which problems the system could solve. For workers, this separates assisted performance from a durable new skill; for employers, it cautions against treating higher short-term output as proof that training has succeeded.
The participants were highly educated consultants from one firm, the tasks were brief, and the experiment ran in March and April 2024. It did not test delayed retention, messy end-to-end projects, job mobility or whether errors would be caught in live work. Authors include BCG and OpenAI researchers, and the two organizations helped supply participants, benchmarks or task design. Independent replication with diverse workers and delayed unaided tests would change the assessment.[1]
The distinction that matters
Where ChatGPT changed performance
Effects compare GPT-4 access with matched training plus ordinary public online resources; they are not gains over doing nothing.
Coding
Consultants completed a 90-minute technical task
+49 percentage points (95% CI 42–56)
Statistics
Consultants interpreted statistical evidence
+20 points (95% CI 15–25)
Prediction
Answers were compared with expert response vectors
+17 points closer (95% CI 9–25)