Can automated feedback help peer tutors draw out learner reasoning? A randomized trial found modest gains
Across two SAT mathematics bootcamps, transcript-based feedback increased three measured talk moves by 8–15%. It did not reduce tutors’ 91.5% share of speaking time or improve attendance, practice-test performance or satisfaction after correction.
The short version
- What happened
- The trial randomized 1,199 tutor assignments across four conditions in two online SAT mathematics bootcamps; the pooled complete-case analysis covered 5,653 sessions from 899 tutors.
- Who it affects
- Volunteer tutors may receive specific prompts for drawing out learner thinking when human coaching is unavailable, but the tool did not reduce their dominance of speaking time.
- What remains unproven
- Unreviewed preprint from one online SAT platform; most tutors were high-school volunteers and generalisability to schools or professional tutoring is unknown.
Keep in mind
Unreviewed preprint from one online SAT platform; most tutors were high-school volunteers and generalisability to schools or professional tutoring is unknown.
The four-week trial had incomplete delivery, prior exposure among some controls and no log showing whether or when tutors opened feedback.
Discourse outcomes came from automated transcripts and classifiers that detect the presence, not the quality or appropriateness, of a talk move.
No subsequent SAT scores were available; pooled attendance, practice-test and satisfaction outcomes were null.
Interviews were small, coded by one researcher and conducted by platform staff; several authors were affiliated with Schoolhouse.world.
Source analysis · 1 linked source. Primary analysed source date: 4 November 2026.
Read the full report · 9 min The evidence, methods, implications and limitations
The direct answer: feedback changed selected moves, not who held the floor
Automated feedback can nudge volunteer peer tutors to invite and build on learner thinking, but it did not make the sessions broadly learner-led. In an unreviewed preregistered randomized trial on Schoolhouse.world, tutors assigned to receive transcript-based feedback elicited learner ideas about 8% more often, took up learner contributions about 9% more often and were followed by about 15% more coded learner reasoning than tutors assigned no feedback. These are modest, statistically supported changes in discourse markers—not evidence of higher examination scores or durable learning.
The strongest null result is just as important. Tutors in the control group occupied 91.5% of speaking time, and the estimated treatment effect on that share was effectively zero. Adding a participation message for learners did not further change discourse. Attendance, practice-test performance and satisfaction measures also showed no significant pooled effect from tutor feedback. The trial therefore separates two claims that are often blurred: a system can change particular conversational techniques without redistributing participation or improving measured educational outcomes over four weeks.[1]
Two bootcamps, four conditions and one tutor-level randomization
The setting was two consecutive four-week SAT mathematics bootcamps in spring 2024. Each offered eight 75-minute Zoom sessions to groups of up to ten learners. Schoolhouse.world tutors are volunteers, mostly high-school students, who qualify through SAT mathematics performance and platform training rather than professional teaching credentials. Across both bootcamps the study involved 1,030 distinct tutors, 7,443 learners and 6,508 eligible recorded sessions before complete-case restrictions. The platform serves learners across more than 180 countries, although the paper does not report a country denominator for this trial.
Before sessions began, 1,199 tutor assignments were randomized at tutor level. Control tutors received no feedback; a tutor-feedback group received automated analysis of its own discourse; and two joint groups received the same tutor feedback while learners saw a participation message framed either around personal goals or benefits to the group. Randomization was stratified by SAT score band and section size. A tutor running several sections received the same assignment within a bootcamp, while 169 tutors participating in both bootcamps were randomized again for the second.[1]
The distinction that matters
What changed in the randomized trial
The intervention improved three coded discourse measures but did not alter talk balance or measured short-term learning.
Eliciting
+1.41 per hour
8% above the control mean
Uptake
+0.52 per hour
9% above the control mean
Reasoning
+0.24 per hour
15% above the control mean
Talk share
No detectable change
Tutors still spoke about 91.5% of the time
Learning
No pooled effect
Practice tests and attendance did not improve
What the intervention actually showed tutors and learners
Tutor feedback arrived after a recorded session and usually before the next one. It compared the tutor’s talk share and proportion of participating learners with earlier sessions and the bootcamp average. Classifiers highlighted examples of eliciting ideas, taking up contributions and asking for reasoning. GPT-4 Turbo generated one to three suggestions grounded in the transcript roughly every other session. The sequence followed a fixed curriculum rather than choosing whichever metric looked weakest. Tutors could revisit earlier feedback and received email reminders when they had not opened it.
Learner feedback was thinner. In the joint conditions, learners saw a modal when entering their next session showing the group’s share of talk and a message encouraging participation. They could not revisit it and received no reminder. One framing linked speaking to their own score goals; the other argued that questions could help peers. The display reported a group statistic rather than each learner’s contribution. That design difference matters when interpreting the null result: tutor and learner arms did not receive equivalent information, timing, persistence or opportunity to act.[1]
The analysis used intention to treat—and delivery was incomplete
The preregistered primary analysis compared groups as assigned, whether or not a particular tutor viewed feedback. It excluded the baseline first session, the practice-test fifth session, recordings shorter than 30 minutes and sessions missing required covariates. The pooled complete-case sample contained 5,653 sessions from 899 tutors; learner-level models used up to 14,839 learner-sessions. Ordinary least-squares models included baseline discourse measures, tutor experience and gender, cohort composition, SAT band, session-number and bootcamp effects, with standard errors clustered by tutor and Benjamini–Hochberg correction within outcome families.
Delivery was not complete. Feedback reached 77.9% of treated learner-sessions in the first bootcamp and 56.0% in the second, where an error withheld much early feedback. Some second-bootcamp control tutors had also received feedback in the first bootcamp or a pilot. Both issues could reduce the difference between assigned groups. Instrumental-variable and sensitivity analyses produced somewhat larger estimates, but they require additional assumptions. The intention-to-treat result remains the most policy-relevant estimate of offering this imperfectly delivered system under the tested conditions.[1]
What increased—and what did not
Across bootcamps, treated tutors elicited learner ideas 1.408 more times per hour than controls, whose mean was 17.161; uptake increased by 0.519 over a control mean of 5.914; and coded learner reasoning increased by 0.238 over 1.637. The associated p-values were .005, .004 and .001, and all three survived the family correction. Effect sizes were about 0.12 control standard deviations. Signs were broadly consistent across the two bootcamps, although no single-bootcamp estimate survived correction on its own.
Other outcomes did not clear the threshold. Learner questions and the proportion speaking moved in a favourable direction but were not conventionally significant after correction. Tutor speaking share changed by minus 0.001, p=.803. Attendance, cohort or learner practice-test performance, ratings and net promoter scores showed no significant pooled effect. A few uncorrected satisfaction contrasts favoured learner feedback, but none would survive correction. The authors appropriately conclude that observable discourse changed while measured learning and the balance of talk did not.[1]
Interviews explain barriers, but they are not a second experiment
The team interviewed 17 treated tutors and 13 learners from the first bootcamp. Most tutors said the summary statistics and concrete generated suggestions were useful, and several described replacing a closed invitation such as ‘any questions?’ with wording that assumed questions existed. Ten tutors named learner engagement as their biggest challenge. The highlighted transcript excerpts were less valued: most interviewed tutors said they ignored them or saw them as redundant beside the generated suggestions.
Learners described why a group-level participation number was hard to act on: one learner could speak more yet see little movement if peers stayed silent, some preferred listening and learner-to-learner interaction was rare. But the interviews were small, conducted by a platform staff member and coded by one person; no control tutors were interviewed. They help generate design explanations, not estimate effects. The platform later changed its feedback based partly on these accounts, but the paper does not evaluate whether that redesign works.[1]
The practical lesson for schools and tutoring platforms
For tutoring programmes, the experiment supports a limited use case: automated transcript feedback can increase selected, observable talk moves at scale. Adoption should still start with reliable delivery, consent, transcript security, accessibility and audits of classifier errors. Programmes should not equate a higher count of questions or reasoning-labelled utterances with better teaching. The classifiers detect whether a move appears, not its quality, timing or mathematical correctness. Only two classifiers were validated on this platform, and speech-recognition errors can affect every downstream measure.
The study cannot show that feedback raises SAT scores because subsequent examination scores were unavailable and practice-test estimates were null. It lasted four weeks on one nonprofit platform with mostly high-school volunteer tutors. Schoolhouse.world staff co-authored the work, though the quantitative analysis was led by an unaffiliated researcher and the plan was preregistered. Confidence would rise with independent replication, complete delivery logs, validated learning measures, longer follow-up and trials of session structures that genuinely give learners more opportunities to speak. Until then, the intervention is a modest coaching aid—not an automated substitute for educator development.[1]