Should a teaching chatbot refuse quick answers?
In a small Swedish case study, university teachers found a reflection-first chatbot easy to use but disagreed about its value. Eleven reported using it; the study measured perceptions, not learning gains.
The short version
- What happened
- All 48 participants in two cohorts of an eight-week professional-development course were invited; 21 answered the questionnaire, 11 reported using the chatbot and nine were interviewed.
- Who it affects
- Teachers should be told when an AI tool is designed to slow down answer generation and what learning purpose that serves.
- What remains unproven
- One course at one institution with 21 questionnaire respondents
Keep in mind
One course at one institution with 21 questionnaire respondents
Only 11 respondents reported chatbot use and nine were interviewed
No control group or measured learning outcome
No systematic interaction logs and an unknown number of total users
Technical access problems affected an unknown number of participants
Source analysis · 1 linked source. Primary analysed source date: 15 October 2026.
Read the full report · 6 min The evidence, methods, implications and limitations
The answer: sometimes—but the purpose must be explicit
A chatbot that withholds a summary and asks a teacher to reason through a problem can create useful space for reflection. It can also feel obstructive when the user expected fast information. A small Swedish case study found both reactions in the same professional-development course. Participants generally found the system technically easy to use, but they did not agree that its deliberately reflective behaviour was useful. The researchers call this tension “pedagogical friction.”
The result is a design warning, not proof that refusal improves learning. Only 11 questionnaire respondents reported using the course-specific chatbot, nine respondents were interviewed, and no learning outcome was measured. The study nevertheless exposes a practical choice hidden inside many education products: whether the system should optimize for completing a request or for making the learner do intellectual work. Institutions should state that choice openly and test whether it serves the course objective.[1]
How the eight-week case study worked
The study took place in two cohorts of an introductory university teaching-and-learning course at one Swedish institution. The course lasted eight weeks and expected 60 hours of work across reading, seminars, assignments and reflective journals. Its 48 participants included PhD students, postdoctoral researchers, lecturers and other teaching staff from several disciplines. One cohort was Swedish and the other internationally mixed, but both followed the same course structure.
All 48 participants could use the chatbot and were invited to the research. Twenty-one completed an end-of-course questionnaire; 11 of those said they had used the course-specific tool, while ten had not. Nine questionnaire respondents volunteered for semi-structured interviews lasting about 35 to 45 minutes. Because people could use the tool without joining the study, the total number of users across the course is unknown. A technical access problem also affected an unknown number of participants.[1]
What the chatbot was designed to do
The researchers configured a custom GPT available through the university’s ChatGPT Edu access. They uploaded course readings, assignment descriptions, schedules and other course materials, then specified four functions: administrative support, literature discussion, assignment support and reflective coaching. Administrative questions could receive direct answers. For learning tasks, the system was instructed to encourage dialogue, reference course material and avoid writing summaries or ready-made assignment text.
The authors tested and refined the instructions before participants received access, but this was formative testing based on researcher judgment rather than a formal validation rubric. The model, uploaded knowledge and instructions then remained fixed during the study. The chatbot was optional and was introduced as part of the research rather than embedded in assessment. That limits pressure to use it, but it also means low uptake may reflect visibility, access or course integration as much as the merits of reflective dialogue.[1]
Ease of use did not settle usefulness
Among respondents who used the tool, technical ease was not the main dispute. The sharper divide concerned expectations. Some participants approached generative AI as an efficiency tool for rapid answers and summaries. When the chatbot responded with questions, they experienced the guardrail as restrictive or unhelpful. Others understood the questioning as aligned with the course’s emphasis on reflection and valued a conversational partner that did not immediately complete the task.
This is a useful reminder that a smooth interface can still conflict with pedagogy. If a university advertises an assistant as a shortcut, users may reasonably judge reflective resistance as failure. If teachers understand that the system is designed to slow them down, the same behaviour may be interpreted as coaching. The study’s interviews help describe that mismatch, but self-reported impressions cannot tell whether either group learned more, reflected more deeply or changed their teaching.[1]
Important evidence limits
The questionnaire was created for this project and analysed descriptively; there was no inferential test or population estimate. Interview coding was led by the first author and discussed across the team, without a formal inter-coder reliability exercise. Conversation logs were not collected systematically, so the researchers could not compare reported experience with actual prompts, response quality or time spent. Participants sometimes shared excerpts voluntarily, but those were illustrative rather than a complete dataset.
There was no control group using an ordinary answer-oriented chatbot, no baseline or final assessment of reflective skill and no independent measure of teaching practice. The same team helped design and evaluate the tool, although the lead interviewer did not teach, assess or develop it and told participants that criticism was welcome. The single-course setting and 21 questionnaire respondents make generalization inappropriate. The findings describe how this particular group negotiated a particular design.[1]
What teachers and institutions can take from it
Before deployment, course teams should write a one-sentence learning purpose for the chatbot and show examples of the intended interaction. If the aim is reflection, explain why the system may ask questions before giving information and when a direct answer remains appropriate. Provide an easy route to course staff when the tool is wrong or circular. Access problems should be tracked separately from voluntary non-use, and students or staff should not be disadvantaged for declining an experimental system.
A stronger evaluation would randomly assign willing participants to reflective, answer-oriented and no-chatbot conditions while preserving normal teaching. Researchers could score reflective writing with blinded raters, examine consented interaction logs, measure workload and frustration, and follow whether behaviour changes in later teaching. The current case supports a modest conclusion: the pedagogical character of a chatbot matters, and users need to understand it. It does not show that one character produces better learning.[1]