AI tutors often redirected a valid student strategy in simulation
New analysis today of an unreviewed 26 October preprint: four models were tested on 150 programming problems. Targeted fine-tuning improved responsiveness, but no learners or classrooms were involved.
The short version
- What happened
- Researchers built simulated conversations around 150 NeetCode programming problems, each represented by brute-force, suboptimal and optimal solutions, and tested four language models as tutors.
- Who it affects
- A tutor that ignores a learner's workable idea can make help feel confusing or dismissive. This study identifies that behaviour in simulation, but it cannot say whether students learned more, persisted longer or trusted the system less.
- What remains unproven
- Unreviewed preprint
Keep in mind
Unreviewed preprint
No real students
150 curated programming problems
Synthetic intuitions and conversations
GPT-OSS used as generator and judge
Three fine-tuning splits
No learning or transfer outcome
No explicit funding or interests statement
Source analysis · 1 linked source. Primary analysed source date: 26 October 2026.
Read the full report · 4 min The evidence, methods, implications and limitations
The benchmark tests whether a tutor follows the student's thinking
Four language models frequently favoured their own solution path when a simulated programming student proposed another valid approach, according to an unreviewed preprint submitted on 26 October. The researchers assembled 150 problems from the NeetCode 150 collection and represented each with brute-force, suboptimal and optimal solutions. They generated one conversation per problem and condition for GPT-OSS 120B, Qwen3.8-27B, Llama 3.3 70B and Gemma 4 12B.
The experiment crossed a direct or question-first opening with a correct or plausible-but-false student intuition. Among solution-specific opening responses, brute-force solutions were the most common choice across models, at 22.2% to 32.1%; optimal solutions accounted for 13.0% to 19.4%. That distribution does not by itself prove poor teaching—simple methods can be pedagogically useful—but it established a preferred path against which the student's path could differ.[1]
Asking first helped some models, but did not solve the problem
The paper's Solution Support Rate measured whether a tutor continued with the student's proposed solution. A question-first opening raised that rate for several models when the simulated student's intuition was correct: GPT-OSS increased from 0.336 to 0.534, Qwen from 0.472 to 0.589 and Gemma from 0.380 to 0.507. Llama was the exception, changing from 0.465 to 0.459. With false intuitions, results were mixed, showing that eliciting an approach did not guarantee useful adaptation.
After a solution was completed, every base model sometimes moved the student toward a less efficient method—between 6.1% and 7.1% of conversations in the reported comparison. Supervised fine-tuning on synthetic responsive conversations improved Gemma's scores. The strongest post-solution score reached 0.936 for correct intuitions when candidate solutions were available during training and inference, compared with 0.343 for the base model in the question-first condition.[1]
The distinction that matters
Responsive tutoring depends on timing
The benchmark separated three moments when a tutor can either follow or override a learner.
Opening
Ask first or prescribe a route
Question-first improved support for three of four models
During work
Build on the proposed strategy
Support rates remained model- and condition-dependent
After completion
Introduce a better alternative
Base models sometimes moved to a worse method
This is a model-behaviour test, not evidence of learning
No student read the replies, wrote code or learned from the tutor. GPT-OSS generated the correct and false intuitions and also acted as the response judge, creating a shared-model dependency that human validation should test. The problems were curated programming interview tasks; the authors say they may not represent the diversity of multiple-solution learning. Fine-tuning used an 80/20 problem split repeated with three seeds, which helps guard against one fortunate split but does not provide classroom replication.
Teachers and product teams should therefore treat responsiveness as one evaluation layer, alongside factual accuracy, pedagogy, accessibility and actual learning. A stronger trial would preregister tutor behaviours, use independent human ratings, recruit learners with different experience levels, compare against human tutoring and measure whether students can explain and transfer a solution later. The paper does not include an explicit funding or competing-interests statement, so those interests cannot be fully assessed from the source.[1]