AI Tutors: What the Randomized Trials Found, and What They Cost

What the trials measured

Since 2023, researchers have run randomized controlled trials of AI tutors built on large language models (LLMs, the text-generating systems behind chatbots such as ChatGPT) in Nigeria, Ghana, the United States, Turkey and England. A randomized controlled trial assigns students, classes or tutors to the AI tool or to a comparison group by lottery, so that differences in results can be put down to the tool. Most of these studies report an effect size in standard deviations (SD): the gap between the groups' scores divided by the spread of scores, which puts different tests on one scale.⁸⁹

That common unit doesn't make the trials comparable: they ran from two weeks to about eight months, from primary school to university, against comparison groups that got anything from nothing extra to a full class session. The table above therefore lists each result as stated rather than charting or ranking them.

Nigeria: six weeks after school

The trial most often cited for AI tutoring in low-income settings ran in June and July 2024 in nine public schools in Benin City, Edo State. First-year senior secondary students, who are typically 15, were invited to an after-school English program; 52% expressed interest, and a lottery among them assigned 657 to the program and 671 to a control group that "did not receive any intervention but continued their regular learning in the classroom". Students worked in pairs at a shared computer with Microsoft Copilot, then powered by GPT-4, over twelve 90-minute sessions. Teachers guided the sessions but "did not provide direct instruction".⁸

Of those assigned, 422 program students and 337 control students sat the final test. The World Bank researchers report scores 0.31 SD higher for students selected for the program, 0.24 SD in the English section, and 0.21 SD higher on the schools' own end-of-year English exam, which covered more than the program did. The authors equate the gain to 1.5 to 2 years of "business-as-usual" schooling, and their blog says the program outperformed 80% of the interventions in a comparison set of randomized trials in developing countries.⁸¹⁵ They also write that they "cannot isolate the effect of the chatbot alone": the result reflects the whole package of structured prompts, curriculum alignment, teacher support and peer work.¹⁶

Ghana and Harvard: AI tutors working alone

In Ghana, Rising Academies, which runs a network of schools there and built the WhatsApp math tutor Rori, randomly assigned 11 of its schools: students in five used Rori for two 30-minute sessions a week during study hall from February to August 2023, while six kept their usual routine. Among the 477 students who sat both tests, the Rori group's scores grew by 5.13 points on a 35-question test against 2.12 for the control group, an effect size of 0.36. The authors, who include Rising Academies staff, call the study preliminary and note it covers the first year only.⁹

At Harvard, 194 students in an introductory physics course each learned one topic from a custom GPT-4 tutor at home and another in an in-class active-learning session, swapping in the second week. Median post-test scores were 4.5 for the AI lessons and 3.5 for the class sessions, with median learning gains "over double", and the median student spent 49 minutes with the tutor against a 60-minute class. The team puts the effect at 0.73 to 1.3 SD and writes that it does not "presume that structured AI tutoring will always outperform in-class active learning in all contexts".¹⁰

When AI access didn't help

Two trials found no gain or a loss. In a large Turkish high school in fall 2023, about 50 math classes were assigned to a ChatGPT-style interface ("GPT Base"), a version prompted to give teacher-designed hints instead of answers ("GPT Tutor"), or no AI. With the tools available, practice scores rose 48% and 127%. On a later exam taken without AI, GPT Base students scored 17% lower than the control group, a statistically significant gap, and GPT Tutor students were "statistically indistinguishable" from it. The authors found that most GPT Base students used the chatbot to get solutions, as a "crutch".¹¹

The newest result comes from the University of Maryland, College Park. In fall 2025, 30 instructors' sections were randomly given a GPT-4o study assistant that draws on each course's own materials. Students with access had final grades 0.37 SD lower in the same-course sample, and their participation in instructor-designed online activities fell by 0.90 SD. Only about 15% of students offered the tool used it at least once, and all students could already use ChatGPT and Gemini through the university. The authors conclude that the tutor "may displace engagement with existing learning activities without improving academic performance".¹⁴

AI as the tutor's assistant

Two trials kept a human tutor in every conversation. Tutor CoPilot, built at Stanford, ran from the end of March 2024 for two months with FEV Tutor, a virtual tutoring company, and a large school district in the southern US, serving students in grades 3 to 8 who had scored below grade level. Tutors were randomly given access to a button that drafts a response using a chosen strategy, such as a hint, a worked example or encouragement. Students of tutors with access passed their end-of-lesson exit tickets 66% of the time against 62%, and by 9 percentage points more where tutors had lower quality ratings. The study found no statistically significant improvement in end-of-year math test scores, which the authors attribute partly to the two-month length.²

On Eedi, a UK math platform with remote human tutors, Google's LearnLM model drafted messages in a May–June 2025 trial with 165 students aged 13 to 15. Seventeen expert tutors reviewed every draft before it was sent and approved 76.4% with no or minimal edits. Students in supervised LearnLM sessions solved novel problems on later topics 66.2% of the time, against 60.7% for those tutored by humans alone. The authors, from Google and Eedi, describe it as an exploratory trial.¹²

Who ran the trials, and who paid

Three of the eight trials were written or paid for by the company whose product was tested. Rising Academies staff co-wrote the Rori study, Google and Eedi wrote the LearnLM study, and Medly paid What Worked Education to evaluate its GCSE science platform; that four-week trial in English secondary schools reported Hedges' g = 0.33 (an effect size in SD with a small-sample correction), with 30.7% attrition and tests aligned to the curriculum rather than standardized.⁹¹²¹³ The other five were run by university or World Bank researchers, and they include both the larger positive results (Nigeria, Harvard) and the null or negative ones (Turkey, Maryland). The Nigeria study used a free Microsoft product, and its stated support came from the Mastercard Foundation.⁸

From trials to classrooms, and the price

School systems have bought AI tutors without waiting for more trials. The School District of Palm Beach County put Khan Academy's Khanmigo, an AI tutor and teacher assistant, into every middle and high school in the 2024–25 school year. It told WPTV the first year cost $2,545,000, half of it covered by the Stiles-Nicholson Foundation, and that it would pay $1,138,000 for 2025–26. "It's a sizable expense for us, but the reach is far," Superintendent Mike Burke said.⁴ In Arizona, State Superintendent Tom Horne used $1.5 million of federal pandemic relief funds to pay for two years of Khanmigo in some districts; about 16% of the state's public school students had signed up by October 2025, though KJZZ noted it wasn't clear how many used it regularly. "I can't afford 1,200,000 tutors for 1,200,000 students," Horne said.⁵

Others are still piloting. Newark Public Schools extended Khanmigo from one school in 2023–24 to 13 more North Ward schools in 2024–25, at no cost to the district, and approved a data-sharing agreement with Khan Academy to study its effect; the district had not released that data.⁶ Indiana's Department of Education ran a nearly $2 million AI pilot grant across 36 districts, funded by a one-time $1.8 million allocation of federal pandemic relief; 20 of the districts chose Khanmigo, and about 53% of 625 teachers surveyed rated the effect on students as positive.³ Estonia's national AI Leap program says its 2025–26 pilot year reaches 154 of the country's 156 upper-secondary schools, 4,800 teachers and 20,000 students, who gain access from January 2026 to an app "intentionally configured not to provide direct answers", built with partners including OpenAI and Google.¹⁷

Published prices vary with what's counted: Khanmigo's list price is free for teachers and $44 a year for learners, with district prices given only on request, while the trial cost figures in the scorecard range from AI usage alone to whole programs with staff and training.⁷⁸ None of the deployment sources above reports a randomized evaluation of that deployment's own learning outcomes.

What it means for the tutor's tasks, by the data

O*NET lists 16 core tasks for tutors, led by giving feedback, reviewing class material and assessing progress. The US had 208,400 tutors in 2025 with a median wage of $43,350, and the Bureau of Labor Statistics projects "little or no change" in employment from 2025 to 2035, with 34,200 openings; the projection doesn't isolate any effect of AI.¹

On the graded cases in the scorecard, chatbot tutors conduct sessions on their own in production at two school systems, covering two of the eight tasks: reviewing material and one-to-one instruction. The other cases are AI assisting a teacher or tutor at pilot stage, or, for progress reports, only a vendor's feature description. In the trials, most AI tutoring sessions still had an adult in the room or in the loop: Nigeria's teachers ran the sessions, Eedi's tutors approved each message, and Tutor CoPilot's largest gains came for lower-rated tutors.⁸¹²²