Therapy AI struggled with unfamiliar children. That matters for care

Every child is more than a pattern of movements

Models that score an autistic child’s engagement during robot-assisted therapy struggled when tested on a child they had never seen. The finding, from a benchmark in Kazakhstan, raises a question for therapists and families: how much trust should a score deserve before it shapes care?

A peer-reviewed benchmark from Kazakhstan exposes another problem: a model that has seen one child does not necessarily work for another. Across 47 hours of robot-assisted autism therapy, performance remained near chance when the test child was genuinely unseen.

How researchers divided the videos into training and testing changed the apparent result at least as much as their choice of classifier. For the five-level engagement task, mean macro-F1 varied by 0.12 across evaluation protocols, compared with 0.07 across the discriminative models. Macro-F1 balances performance across the labels; a better score on familiar frames still does not establish understanding of a new child.

The central problem is leakage of familiarity. If frames from the same child and session appear on both sides of a random split, a model can exploit person- or recording-specific cues. That score does not show it will interpret another child. The authors describe the random result as a leakage-inflated upper bound and report near-chance performance when the test child is genuinely unseen.[1]

What the dataset contains

The QAMQOR dataset covers 187 robot-assisted autism-therapy sessions—47 hours in total—with 34 children aged three to 12. Thirty-two were boys and two girls; 23 were described as non-verbal, 11 verbal and 12 had co-occurring ADHD. Sessions lasted roughly 10 to 20 minutes and used a NAO humanoid robot for activities including songs, dance, storytelling, imitation, emotion recognition, turn-taking and identifying the robot's body parts.

A wide-angle camera recorded the sessions. Frames were sampled once per second, cropped to the child and labelled on a five-level scale from non-compliance through high engagement, as well as a binary engaged/disengaged outcome. OpenPose and MediaPipe converted face, body and hand positions into numerical keypoints. The paper reports institutional ethics approval and written consent from parents or caregivers.[1]

The benchmark tests four different questions

A random split asks whether a model can classify held-out frames drawn from a familiar mixture of people, sessions and activities. A child-independent split asks whether it works for a person it has never seen. Session-independent and activity-independent splits test new encounters and tasks. Those distinctions matter because engagement is expressed differently through gaze, posture, hand movement and response timing for different children and contexts.

The team compared logistic regression and several gradient-boosted tree methods with temporal XGBoost, a recurrent LSTM and a Temporal Convolutional Network. Every model used the same split protocols. Class balancing improved macro-F1, balanced accuracy and Cohen's kappa while lowering raw accuracy, illustrating why an imbalanced dataset can make an uninformative majority prediction look respectable if accuracy is reported alone.[1]

More complex sequence models did not solve the problem

Under the matched, untuned settings, XGBoost and Temporal XGBoost were the strongest overall classifiers, while the LSTM and TCN tended to collapse toward the majority class. In the binary task, the discriminative-model macro-F1 range ran from 0.47 to 0.53; in the five-level task it ran from 0.23 to 0.30. Those are weak results even before asking whether an error would change a therapy interaction.

LightGBM trained in a reported median 26 seconds on CPU and reached mean macro-F1 of 0.50 for binary and 0.29 for multi-class classification, compared with 186 seconds and 0.45 and 0.14 for the TCN. The authors stress that the neural models were not tuned per protocol, so these numbers do not prove that deep sequence methods are inherently worse. They show only that added architecture did not overcome limited independent participants, imbalance, coarse sampling and behavioural heterogeneity under the common benchmark.[1]

What therapists and families should know

The dataset does not demonstrate that a robot can understand a child's inner state or deliver therapy autonomously. Engagement labels are human interpretations of observable behaviour sampled once per second. A child may look away while processing, move for sensory regulation or decline an activity for reasons a pose model cannot observe. Reducing those behaviours to a score risks confusing compliance with benefit.

The authors recommend using a model, if at all, as an assistive signal for therapist judgement and say unseen-child performance is not deployment-ready. That is the right boundary. A therapist needs to see uncertainty, override the system and understand what action a score triggers. Families also need clear governance for sensitive child-derived data, including what is shared, whether raw video is retained and how future researchers may reuse extracted features.[1]

Representation and privacy remain material limits

Thirty-four children from one rehabilitation centre, using one robot and one therapy protocol, cannot represent the diversity of autistic children or services. The extreme sex imbalance prevents meaningful evaluation for girls. The pipeline excludes speech, prosody and physiological signals, while one-frame-per-second sampling may miss fast cues. Results came from single fixed splits, and most models lack cross-validated uncertainty estimates.

The public repository makes reproducibility possible, but openness does not eliminate ethical responsibility. The paper reports consent and releases derived visual representations rather than making the report a clinical deployment. Future reuse should preserve purpose limitations and avoid turning a research engagement label into a diagnosis, eligibility decision or behaviour-control tool.[1]

What would change the assessment

Stronger evidence would recruit larger multi-site cohorts with better sex representation, repeat every evaluation across multiple child-independent folds and test a locked model at a new service. Researchers should report calibration, uncertainty and error by age, communication profile and activity, and compare model-assisted sessions with therapist-only practice on outcomes agreed with autistic people and families—not just classification scores.

For now, QAMQOR's useful result is methodological. It gives researchers a common way to expose whether an engagement model transfers beyond familiar frames. The benchmark does not show that machine recognition is ready to steer care, but it makes inflated claims harder by requiring the most important test: can the system cope with a child it has not already seen?[1]