Principal Investigator
Susanna Loeb – Stanford University, National Student Support Accelerator
Project Description
This study uses an RCT to estimate the impacts of Tutor CoPilot—a novel human-AI approach that leverages a model of expert thinking to provide guidance to tutors as they tutor—on tutoring quality and students’ mastery of content knowledge. The study involved 900 tutors and 1,800 K-12 students from historically underserved communities. All students in this study received tutoring but the research team randomly assigned about half of the tutors to receive access to Tutor CoPilot in their tutoring sessions and the half to tutor as normal with no additional support.
Research Questions
- To what extent does Tutor CoPilot affect student learning?
- Does the effect of Tutor CoPilot on student learning differ by the initial effectiveness of tutors?
- How does Tutor CoPilot change tutoring quality as measured by the language tutors use with students?
- How do tutors perceive Tutor CoPilot?
Key Findings
The study found that students working on mathematics with tutors randomly assigned to have access to Tutor CoPilot were four percentage points more likely to master topics. Notably, students of lower-rated tutors experienced the greatest benefit, improving mastery by nine p.p. relative to the control group. Tutor CoPilot costs only $20 per tutor annually, based on the tutors’ usage during the study. Exploratory analysis of 550,000+ messages using classifiers to identify pedagogical strategies revealed that tutors with access to Tutor CoPilot were more likely to use strategies that fostered student understanding (e.g., asking guiding questions) and less likely to give away the answer to the student, aligning with high-quality teaching practices. Tutor interviews qualitatively highlight how Tutor CoPilot’s guidance helps them to respond to student needs, though tutors flagged common issues in Tutor CoPilot, such as generating suggestions that were not grade-level appropriate.
Study Citation
Wang, R. E., Ribeiro, A. T., Robinson, C. D., Loeb, S., & Demszky, D. (2024). Tutor CoPilot: A human-AI approach for scaling real-time expertise. arXiv preprint arXiv:2410.03017.
The Key Findings above were reproduced from the published report and do not necessarily reflect interpretation of Overdeck Family Foundation staff.
