Does open AI access still let teachers see student judgement?
In one 33-manuscript university course, structural writing clustered near the top while evidence use and theoretical interpretation still varied. The case supports assessment redesign, not a claim that AI improved learning.
The short version
- What happened
- The case study analysed 33 final manuscripts from one 16-week undergraduate qualitative-methods course in California; all 33 students consented to research use of their work.
- Who it affects
- Teachers may need to grade evidence use and decision quality more explicitly when AI makes formal structure easier to produce.
- What remains unproven
- One 16-week course and 33 consenting manuscripts cannot establish general effects across subjects or institutions.
Keep in mind
One 16-week course and 33 consenting manuscripts cannot establish general effects across subjects or institutions.
There was no non-AI comparison, pre-test or causal design, so learning gains cannot be attributed to AI access.
The three rule-based indicators were rubric-proximal proxies, not validated measures of comprehension or overall manuscript quality.
The instructor designed, taught and analysed the course, and the four-case process comparison was not independently coded.
Students could use different models and platforms, and final-manuscript AI interaction logs were not available.
Source analysis · 1 linked source. Primary analysed source date: 11 October 2026.
Read the full report · 7 min The evidence, methods, implications and limitations
The answer: some differences remained visible, but this one course cannot prove learning improved
A peer-reviewed case study from California State University, Sacramento found a distinctive pattern in 33 final research manuscripts produced with unrestricted access to generative AI. Nearly every manuscript specified its method in considerable detail, yet students still differed in how tightly claims were connected to evidence and how well theory informed interpretation. That makes a practical case for assessing judgement-bearing work instead of treating polished structure as sufficient evidence of mastery.
It does not show that unrestricted AI access improved learning. The study followed one class for one semester, had no control group and did not compare the same students with and without AI. Its three indicators were rule-based proxies designed to estimate dispersion, not validated measures of comprehension or final course grades. The useful finding is narrower: within this deliberately AI-integrated assignment, student products did not become uniform on every dimension that the instructor considered important.[1]
What the course and dataset actually contained
The Fall 2025 course was a 16-week undergraduate qualitative research methods class in child and adolescent development. Thirty-six students enrolled; 33 submitted a final manuscript and consented to inclusion, making 33 the full quantitative denominator. Students worked towards a journal-oriented paper of at least 5,000 words through staged assignments covering research questions, data sources, theory, methods, findings and discussion. They also had a course-specific Class Companion and could use ordinary ChatGPT or other major systems without limitation.
The platform and model were not standardised or systematically recorded. For selected earlier assignments, students submitted AI conversation logs alongside their work, but the final manuscripts did not include logs. The quantitative analysis therefore describes properties of the final texts rather than the amount or quality of AI assistance that produced each one. Four contrasting student cases supplied a qualitative process comparison, principally using mid-semester logs and later products, but this was not a formal frequency-coded behavioural study.[1]
The numbers distinguish structure from evidence and interpretation
The Methods Specificity Index ran from zero to six and assessed whether a manuscript named and explained elements such as research design, data source and analytic procedure. Its median was six; 23 of the 33 manuscripts, or 69.7%, reached the maximum, and 32 of 33 scored either five or six. That ceiling pattern suggests that structural methods language was no longer very discriminating in this setting, although the index does not establish whether every methodological choice was appropriate.
The other two indicators spread more widely. Claim–Evidence Coupling ranged from zero to 0.750, with a median of 0.333 and an interquartile range from 0.273 to 0.542. Theory–Interpretation Linkage ranged from zero to 0.917, with a median of 0.444 and an interquartile range from 0.250 to 0.500. These proportions were generated through specified text rules. They show variability in the corpus, not a threshold between competent and incompetent work, and they should not be interpreted as effect sizes for AI.[1]
Four cases suggest processes teachers can inspect
The author selected four contrasting cases to examine how different outcomes emerged under the same permissive AI policy. Stronger work was associated with three recurring processes: directing the system against explicit criteria, providing project-specific material the model could not invent, and using substantive multi-turn revision rather than accepting a first draft. Those behaviours place responsibility in selection, checking, integration and revision rather than in typing every sentence independently.
Association is the right word. Four purposively contrasted cases cannot estimate how common a behaviour was, whether it caused a stronger manuscript or whether another teacher would identify the same process. The instructor designed the course, taught it, authored the paper and performed the analysis. That unusually close access provides context, but it also creates a risk that the analysis reflects the designer's expectations. Independent coding and replication in other subjects would make the process claims more convincing.[1]
What this means for teachers and students now
For teachers, the study argues against using polished organisation as the main signal of student capability when AI can reliably scaffold it. An assessment can instead require learners to expose their evidence chain, justify why a method fits the question, identify what came from the actual dataset, test an alternative interpretation and explain consequential revisions. Selected process records, short oral checks or staged decisions may help, provided they are proportionate, accessible and not treated as surveillance or automated authorship detection.
For students, the result does not mean that prompt technique replaces subject knowledge. The dimensions that remained dispersed were precisely those that require reading the evidence, understanding theory and judging whether a claim is warranted. Institutions should also avoid assuming equal access, fluency or confidence with AI tools. If use is required, students need a supported route to learn the tools, protect sensitive material and challenge generated errors, plus a fair alternative when a commercial service is inaccessible or inappropriate.[1]
Limits and what would change the assessment
This is a single-semester case at one US public university, in one writing-intensive subject, with 33 manuscripts and no counterfactual. The measures were created for this analysis and intentionally sit close to the course rubric, but they were not validated against independent expert ratings, retained learning or later professional performance. Models and student tool choices varied. The study did not report demographic comparisons, and sex and gender were not collected, so it cannot assess whether the design worked equally well across groups.
Confidence would rise with pre-registered multi-course studies that compare assessment designs rather than merely comparing access policies. Useful outcomes would include blinded expert ratings, student explanations of their decisions, delayed knowledge tests, workload for teachers, accessibility, and false-positive or inequity risks in process checks. Repeating the three indicators with independent raters and publishing analysis code would test robustness. Until then, the paper is best read as a detailed design case and a hypothesis about where human judgement remains visible.[1]