What should AI companies prove before giving conversational agents to children?
A new World Economic Forum white paper sets out child-focused practices spanning design, privacy, learning, harmful interactions and accountability. It is useful guidance, but not a binding standard or an outcome study, and its public summary reports no sample or systematic-review method.
The short version
- What happened
- The 30 September white paper covers age-appropriate design, beneficial use, harmful interactions, privacy, learning, accountability and inclusion across the AI life cycle.
- Who it affects
- Children could gain more useful learning and accessibility tools if benefits are tested and persuasive, data-intensive or autonomous features are constrained by default.
- What remains unproven
- The source is a WEF policy white paper rather than a peer-reviewed experiment, law or binding technical standard.
Keep in mind
The source is a WEF policy white paper rather than a peer-reviewed experiment, law or binding technical standard.
The public summary does not report a participant sample, child denominator, systematic-review protocol or measured effect on safety, learning or wellbeing.
Implementation and enforcement will differ across countries, schools, families, platforms and regulatory systems.
Source analysis · 1 linked source. Primary analysed source date: 30 September 2026.
Read the full report · 6 min The evidence, methods, implications and limitations
The report treats child safety as a product-lifecycle duty
A World Economic Forum white paper published on 30 September argues that children's safety and wellbeing must be considered throughout the AI life cycle, not added after a conversational product has reached families and schools. The report groups its proposed practices around age-appropriate design, beneficial use, harmful interactions, privacy, learning, accountability and inclusion. It covers generative, conversational and agentic systems because memory, personalisation, persuasion and autonomous actions can change both the usefulness and the risk of an interaction.
That framing is more practical than a generic promise to make AI 'safe'. Developers decide what data a system remembers, how it responds to distress, whether it can contact other services, which commercial prompts it displays and when a human is brought in. Deployers decide where the system appears and what claims are made about it. Schools, parents and children cannot compensate for every hidden design choice. The paper therefore places duties across organisations rather than treating safety as a matter of individual self-control.[1]
Age-appropriate design is wider than an age gate
Age assurance may help a service apply different protections, but it can also create privacy risks if platforms collect identity documents, biometric estimates or more behavioural data. A child-centred design approach must ask what the product can do after a user enters: whether persuasive features are limited, sensitive memories expire, advertising is separated from advice, default settings minimise disclosure and high-risk actions require adult or institutional oversight.
The report's emphasis on inclusion is important because one age check will not work equally for every child. Some young people lack standard identity documents, share devices or use assistive technology. Language, disability and cultural context can change whether warnings are understood and whether a complaint route is usable. A protection that excludes vulnerable children or pushes them towards unregulated services may create a different harm rather than solving the first one.[1]
Benefits should be demonstrated, not assumed from engagement
AI tools can support learning, creativity and accessibility, but time spent with a product is not evidence of educational benefit or wellbeing. A credible evaluation should define the intended benefit, compare it with an appropriate alternative and measure whether gains persist. For a tutoring system, that could mean independent assessment of learning rather than session length. For an assistive tool, it could mean whether a child completes a task with more autonomy without surrendering unnecessary data.
The same principle applies to AI companions. A fluent response may feel supportive while providing inaccurate information, encouraging dependency or mishandling a disclosure of abuse or self-harm. Companies need pre-release tests that reflect children's actual language and situations, clear escalation rules and monitoring for harms after launch. Serious incidents should be reportable and auditable without forcing families to prove a product's internal behaviour from screenshots alone.[1]
Incentives determine whether practices survive commercial pressure
The WEF paper looks beyond voluntary checklists to the incentives created by regulation, procurement, assurance, investment, shared infrastructure and organisational accountability. Procurement is especially concrete: a school or public agency can require a documented risk assessment, data-retention limits, independent testing, incident reporting and an exit plan before buying a service. Investors and boards can ask whether safety teams have authority, resources and access to product decisions rather than existing only as advisers.
Assurance must be specific about scope. A one-time audit of a model does not automatically cover the surrounding app, its memory, advertising, tool access or later updates. Shared testing infrastructure could help smaller organisations evaluate risks without recreating every benchmark, but common tools should not become a ceiling. Products used by children still need local testing, accessible redress and clear responsibility when a supplier, school and platform each control part of the experience.[1]
What the paper establishes—and what would change our assessment
This is a policy white paper, not a clinical or educational trial. The public publication page does not provide a participant sample, child denominator, systematic evidence-search method or measured reduction in harm. Its value is to organise practices and institutional levers; it cannot show that companies already follow them or that any one practice improves outcomes. The recommendations are not binding law, and enforcement will depend on the countries and institutions that adopt them.
Confidence would rise if companies publish product-level child-risk assessments, independent red-team results, incident rates, subgroup testing and evidence that protective changes improve outcomes without excluding children. Regulators and procurers should state measurable requirements and preserve access for independent researchers. The assessment would weaken if the white paper becomes a badge used without public evidence, if age checks expand surveillance, or if organisations cite engagement as proof of benefit. The test is whether children are safer and more capable in practice, not whether a company can point to a principles document.[1]