The open-source tool, which researchers say other educators can recreate and integrate into their tutoring systems, made the human tutors slightly more effective. And the weakest tutors became nearly as effective as their more highly-rated peers, according to a study . 

The tool, dubbed Tutor CoPilot, prompts tutors to think more deeply about their interactions with students, offering different ways to explain concepts to those who get a problem wrong. It also suggests hints or different questions to ask.

The new study offers a middle ground in what鈥檚 become a polarized debate between supporters and detractors of AI tutoring. It鈥檚 also the first randomized controlled trial 鈥� the gold standard in research 鈥� to examine a human-AI system in live tutoring. In all, about 1,000 students got help from about 900 tutors, and students who worked with AI-assisted tutors were four percentage points more likely to master the topic after a given session than those in a control group whose tutors didn鈥檛 work with AI.

Students working with lower-rated tutors saw their performance jump more than twice as much, by nine percentage points. In all, their pass rate went from 56% to 65%, nearly matching the 66% pass rate for students with higher-rated tutors.

The cost to run it: Just $20 per student per year 鈥� an estimate of what it costs Stanford to maintain accounts on Open AI鈥檚 GPT-4 large language model.

The study didn鈥檛 probe students鈥� overall math skills or directly tie the tutoring results to standardized test scores, but Rose E. Wang, the project’s lead researcher, said higher pass rates on the post-tutoring 鈥渕ini tests鈥� correlate strongly with better results on end-of-year tests like state math assessments.聽

Wang said the study鈥檚 key insight was looking at reasoning patterns that good teachers engage in and translating them into 鈥渦nder the hood鈥� instructions that tutors can use to help students think more deeply and solve problems themselves.聽

鈥淚f you prompt ChatGPT, ‘Hey, help me solve this problem,’ it will typically just give away the answer, which is not at all what we had seen teachers do when we were showing them real examples of struggling students,鈥� she said.

Essentially, the researchers prompted GPT-4 to behave like an experienced teacher and generate hints, explanations and questions for tutors to try out on students. By querying the AI, Wang said, tutors have 鈥渞eal-time鈥� access to helpful strategies that move students forward.

鈥滱t any time when I’m struggling as a tutor, I can request help,鈥� Wang said.

She said the system as tested is 鈥渘ot perfect鈥� and doesn鈥檛 yet emulate the work of experienced teachers. While tutors generally found it helpful 鈥� particularly its ability to provide 鈥渨ell-phrased explanations,鈥� clarify difficult topics and break down complex concepts on the spot 鈥� in a few cases, tutors said the tool鈥檚 suggestions didn鈥檛 align with students鈥� grade levels. 

A common complaint among tutors was that Tutor CoPilot鈥檚 responses were sometimes 鈥渢oo smart,鈥� requiring them to simplify and adapt for clarity.

鈥淏ut it is much better than what would have otherwise been there,鈥� Wang said, 鈥渨hich was nothing.鈥�

Researchers analyzed more than half a million messages generated during sessions, finding that tutors who had access to the AI tool were more likely to ask helpful questions and less eager to simply give students answers, two practices aligned with high-quality teaching.

Amanda Bickerstaff, co-founder and CEO of , said she was pleased to see a well-designed study on the topic focused on economically disadvantaged students, minority students, and English language learners.  

She also noted the benefits to low-rated tutors, saying other industries like consulting are already using generative AI to close skills gaps. As the technology advances, Bickerstaff said, most of its benefit will be in tasks like problem solving and explanations. 

Susanna Loeb, executive director of Stanford鈥檚 National Student Support Accelerator and one of the report鈥檚 authors, said the idea of using AI to augment tutors鈥� talents, not replace them, seems a smart use of the technology for the time being. 鈥淲ho knows? Maybe AI will get better,鈥� she said. 鈥淲e just don’t think it’s quite there yet.鈥�

At the moment, there are lots of essential jobs in fields like tutoring, health care and the like where practitioners 鈥渉aven’t had years of education 鈥� and they don’t go to regular professional development,鈥� she said. This approach, which offers a simple interface and immediate feedback, could be useful in those situations. 

鈥�The big dream,鈥� said Wang, 鈥渋s to be able to enhance the human.鈥�

Benjamin Riley, a frequent AI-in-education skeptic who leads the AI-focused think tank and writes a on the topic, applauded the study’s rigorous design, an approach he said prompts 鈥渆ffortful thinking on the part of the student.鈥�

鈥淚f you are an inexperienced or less-effective tutor, having something that reminds you of these practices 鈥� and then you actually employ those actions with your students 鈥� that’s good,鈥� he said. 鈥淚f this holds up in other use cases, then I think you’ve got some real potential here.鈥�

Riley sounded a note of caution about the tool鈥檚 actual cost. It may cost Stanford just $20 per student to run the AI, but he noted that tutors received up to three weeks of training to use it. 鈥淚 don’t think you can exclude those costs from the analysis. And from what I can tell, this was based on a pretty thoughtful approach to the training.鈥�

He also said students鈥� modest overall math gains raises the question, beyond the efficacy of the AI, of whether a large tutoring intervention like this has 鈥渕eaningful impacts鈥� on student learning. 

Similarly, Dan Meyer, who writes a on education and technology and co-hosts a on teaching math, noted that the gains 鈥渄on’t seem massive, but they’re positive and at fairly low cost.鈥�

He said the Stanford developers 鈥渟eem to understand the ways tutors work and the demands on their time and attention.鈥� The new tool, he said, seems to save them from spending a lot of effort to get useful feedback and suggestions for students.

Stanford鈥檚 Loeb said the AI鈥檚 best use is determining what a student knows and needs to know. But people are better at caring, motivating and engaging 鈥� and celebrating successes. 鈥淎ll people who have been tutors know that that is a key part about what makes tutoring effective. And this kind of approach allows both to happen.鈥�