2023 End-of-Year Research Update

Our grantees continue to generate cutting-edge research and evidence across our investment areas of early childhood, kindergarten through ninth grade, and out-of-school STEM. Within our vision for research, these research projects fall into three categories:

Validation of program models;
Research aligned to each portfolio’s logic model and priority outcomes; and
Research about cross-cutting topics that have implications for all portfolios.

Each area of research adds valuable insights to grantees’ respective fields while bolstering our foundation’s understanding of the education landscape and best practices that can support improvement in key academic and socioemotional outcomes for all children. Aligned with our core value of “learn better, together,” Overdeck Family Foundation is committed to promoting transparent research practices by lifting up timely findings. We hope that other organizations and funders can use this data to inform strategic decision-making and investments for the future.

Below are the results of 15 grantee research efforts that concluded at the end of this year. To learn more and see additional research from our grantees across portfolios, visit Overdeck Family Foundation’s Research Repository.

Validation research

FIRST® Longitudinal Study: 2022 Survey Results (108-Month Follow-Up)

FIRST seeks to inspire young people to consider STEM experiences and careers by engaging them in hands-on robotics programs that build science, engineering, and technology skills. For the last nine years, the organization has been working with a team of researchers from Brandeis University to study associations between participation in programming and STEM-related outcomes.

This is a correlational study reporting on how enrollment in FIRST is associated with a range of STEM-related outcomes nine years after youth initially participated in the program. To conduct the study, researchers compared the outcomes of students who chose to enroll in FIRST with a comparison group who chose not to participate, controlling for potential confounding demographic characteristics. The researchers binarized outcomes of interest—STEM attitudes and STEM career interest—and fit logistic regression models to estimate associations between FIRST participation and these outcomes.

New Jersey Tutoring Corps School Year 2022-23 Efficacy Report

The New Jersey Tutoring Corps provides high-dosage tutoring in math and literacy to Pre-K through fifth grade students across New Jersey. During the 2022-23 school year, the organization piloted its embedded school-day program designed to provide high-dosage tutoring as part of the school day for students. Groups of one to four students met for 30-60-minute sessions two to three times per week with the same tutor for the duration of the program. Students received 12-15 weeks of tutoring and participated in assessments of literacy and math skills before tutoring began and after conclusion of the tutoring support.

This is a pre-post descriptive study examining growth in literacy and math skills for the 500 students who received in-school tutoring through the New Jersey Tutoring Corps during the 2022-23 year.

OnYourMark – The Effects of Virtual Tutoring on Young Readers

This is a randomized controlled trial (RCT) of OnYourMark Education, which provides high-impact virtual tutoring for 20 minutes a day, four days a week, to students in kindergarten through second grade. Tutors virtually work with students via video chat during the school day, regularly monitoring progress towards reading proficiency.

The study evaluated the impact of OnYourMark high-dosage tutoring on language and literacy skills for students in kindergarten through second grade. About 1,000 students were randomly assigned to receive tutoring from OnYourMark, with half that group receiving one-on-one tutoring and the other half receiving two-on-one tutoring. The remaining students were randomly assigned to the control group, and continued to receive instruction as usual. Control students did not receive any OnYourMark tutoring. Students completed two assessments of early literacy skills before and after tutoring—the DIBELS which captures foundational literacy skills and NWEA MAP Reading Fluency which also measures reading fluency and comprehension.

The researchers fit a series of regression models, adjusting for demographic characteristics and the combination of school x grade (clustering at the pair level to account for students nested within tutor groups) to estimate differences in the outcomes between the group randomly assigned to receive tutoring and the control group. The researchers also estimated the separate impacts of receiving one-on-one tutoring and two-on-one tutoring and explored heterogeneity in impacts across grade, baseline reading performance, and sub-domains of literacy skills as measured with the DIBELS.

Effects of a Quill.org Intervention on Paragraph Revision

This is a randomized controlled trial (RCT) of Quill, a free, open-source digital platform that provides research-based writing instruction to students. Researchers from The College Board aimed to estimate the impact of Quill’s sentence-combining intervention on high school students’ paragraph revision skills as assessed immediately after receiving the intervention, as well as two weeks and two months post-test. The study included a sample of 82 students who were all at least 14 years old and whose parents provided active consent for them to participate. After recruiting students, the research team randomly assigned about half to participate in a four-week sentence-combining intervention with one week focused on appositive phrases, one week on participial phrases, one week on complex sentences, and one final week on a mix of appositive phrases, relative clauses, and participial phrases activities. Students in the control condition received filler activities involving proofreading, punctuation, and grammar. These activities were thought to be educationally enriching, though not directly related to target study outcomes.

Students in both conditions practiced skills five times per week when participating in the intervention. The researchers used a series of whole-discourse exercises to assess students’ paragraph revision skills prior to the start of the four-week intervention, immediately after the intervention, and then two weeks and two months post-test. They then fit a series of regression models to estimate the impact of assignment to the sentence-combining intervention on students’ paragraph revision skills at each of these follow-up time points.

The Effect of Exposure to Reach Out and Read on Shared Reading Behaviors

This is a correlational study that aims to leverage existing survey data from parents of children ages six months to five years old who had brought their child to a well visit at one of 427 pediatric clinics in North and South Carolina. Clinics collected survey data when conducting intake procedures with parents during well visits. Parents reported on whether they had ever received a book at a well visit before, and what their current reading behaviors were with their child. All data were collected during six-week periods in the fall and spring of each study year. The researchers binarized all outcomes and used logistic regression to examine whether there were any differences in reading related behaviors for parents who had received a book before (the Reach Out and Read group) or not (the comparison group). Models included controls for site, child age, insurance status, caregiver education, and survey language. There were 98,214 parent-child dyads included in the study.

Saga Education – Not Too Late: Improving Academic Outcomes among Adolescents

Researchers from the University of Chicago conducted two separate randomized controlled trials (RCTs) of Saga Education, which offers high-dosage tutoring to support student learning. The studies aimed to evaluate the impact of Saga’s two-on-one tutoring model—delivered by trained paraprofessionals—on high school students’ math performance. The first trial (“Study One”) was conducted during the 2013-14 school year; the researchers randomly assigned 2,633 rising ninth or tenth graders in 12 Chicago Public Schools (CPS) high schools to either receive Saga math tutoring or to continue business-as-usual instruction. The sample consisted almost entirely of male high school students of color from families with lower incomes. The researchers accessed students’ scores on CPS standardized tests and also accessed their grades in math and non-math courses to measure outcomes of interest.

In a second study during the 2014-15 school year, researchers randomly assigned a separate sample of 2,710 ninth and tenth graders (“Study Two”) across 15 schools to receive the same Saga tutoring intervention or to continue business-as-usual instruction. The researchers used a similar design and collected the same post-test outcome data. Given that only about 40 percent of students in Study One and 37 percent of students in Study Two actually participated in Saga tutoring (because they ended up not attending a school offering the program or their schedule did not allow for tutoring), the researchers also used instrumental variables estimation to calculate treatment of the treated effects for the group of students who did participate in at least one Saga tutoring session.

Effects of ST Math on Growth in Math Performance at the Grade-Level

This is a matching study with retrospective data, conducted by a researcher at the MIND Research Institute, which leverages data from 2018-19 through 2021-22 to estimate the impact of ST Math, a game-based, instructional software designed to boost math comprehension through visual learning, on third, fourth, and fifth grade students’ math performance in California, measured at the grade level. After accessing historical data on students’ standardized test scores in California, the researcher used propensity score matching to match full grades (e.g., third grade, fourth grade, fifth grade) that had adopted ST Math to grades that had not adopted the software for their classrooms. The researcher then compared changes in math proficiency and scaled test scores between the grades implementing ST Math and those in the comparison group.

WNET – Cyberchase’s Impact on Environmental and STEM Learning

Cyberchase is a long-running PBS KIDS program created and produced by The WNET Group. The program strives to promote problem-solving skills through the use of mathematics and has recently incorporated environmental themes with the goal of advancing knowledge and inspiring actions related to environmental health and sustainability.

This is a pre-post descriptive study constructed to understand potential benefits of Cyberchase on children’s STEM knowledge, attitudes, and behaviors. Researchers conducting the study asked teachers to collect pre-test assessment data from 234 first and second grade students prior to viewing four full-length episodes of Cyberchase (23-24 minutes/episode) and four three-minute “For Real” videotaped segments. The assessments captured student-reported information on STEM knowledge, attitudes, and behaviors. The teachers then collected the same data following the conclusion of each Cyberchase Episode and “For Real” segment. The research team recruited students from three Title 1 elementary schools in Arizona serving primarily Hispanic students and one Title 1 elementary school in Indiana serving primarily Black students.

Efficacy Analysis of Zearn Math: Findings from implementation in Louisiana

This is a matching study with retrospective data that aimed to estimate impacts of using Zearn Math—a computer-based math platform that provides individualized supports to students—at the recommended dosage on students’ math performance. A group of researchers from Zearn leveraged data on more than 14,000 elementary and middle school students attending schools across 31 Louisiana parishes to conduct the analysis. The researchers accessed students’ scores on the Louisiana state standardized tests (LEAP) to assess math performance at the beginning and end of the 2021-22 school year. The researchers then used coarsened exact matching (CEM) to match students who completed the recommended number of Zearn math sessions—at least three times per week on average—to students who participated in less than one session per week on average. They then compared the test score outcomes of the students in their matched treatment and control groups in order to estimate impacts of participating in Zearn math on LEAP test scores. The research team also explored whether impacts were larger for students who started the year with lower levels of math performance.

Logic model & priority outcomes research

The Canopy Project – Evaluating the Impact of Innovative Schools

This is a landscape scan conducted by The Canopy Project, which used qualitative data and review of internal and external quantitative data on school performance to achieve three aims. First, the researchers sought to identify the types of student performance outcomes that Canopy schools chose to prioritize. Second, they examined how Canopy schools measured success on those outcomes of interest. And third, they explored whether they could demonstrate any evidence of impact for Canopy schools in their network.

EdReports – State of the Instructional Materials Market: Use of Aligned Materials in 2022

This is a descriptive study that sought to understand the degree to which school districts used high-quality instructional materials in 2022, as well as the factors associated with adoption of these materials. The authors defined “aligned” high-quality instructional materials as core comprehensive ELA, math, and science instructional materials that met expectations for EdReports’ review criteria (which align to college- and career-ready standards and other dimensions of quality, such as supports for diverse learners). The study leveraged data from EdReports reviews, copyright dates, ESSER spending reports, and data from the RAND Corporation American Instructional Resources Survey (AIRS) on curriculum use, teacher perception, and school context.

Education First – Strategic School Staffing Solutions

This is a landscape scan of strategic staffing models, conducted by Education First, which aimed to learn how to better meet the needs of students by redesigning critical adult roles and ensuring that those roles are attractive, sustainable, and professionally rewarding. As part of this work, the team identified features of potentially effective models and bright spots of strategic school staffing.

PERTS – Learning Conditions Are an Actionable, Early Indicator of Math Learning

PERTS and the University of Chicago Consortium on School Research (UCCSR) partnered together to develop a continuous improvement framework, Elevate, which seeks to provide educators evidence-based recommendations for improving learning conditions that are associated with students’ academic performance. In Elevate, these learning conditions are offered alongside practical measures that equip educators to monitor conditions in real-time and make data-driven decisions about what practices to adopt, adapt, or abandon in the service of building an optimal learning environment for students.

This report sought to quantify the relationship between math achievement and learning conditions in the Elevate framework, and explore variability in conditions over time. PERTS and the researchers collected data from over 4,000 U.S. students in eighth through 12th grade during October through March of the 2019-20 academic year. After accessing data on schools and students, the team examined associations between three learning conditions—Teacher Caring, Meaningful Work, and Feedback—and students’ likelihood of earning a grade of B or better in mathematics.

RAND – Expanding Afterschool Opportunities: Connecting STEM Afterschool Providers and Schools

This is a mixed methods descriptive survey, which aimed to understand how district and school leaders partner with external organizations to provide STEM afterschool programs. RAND conducted three research activities to provide a national picture of why and how principals and district leaders partner with STEM afterschool providers.

First, the research team distributed a survey to a nationally representative sample of public school principals for kindergarten through eighth grade in November and December 2022 through the American School Leader Panel (ASLP). Of the 6,040 surveys sent, 994 school leaders were both eligible to complete the survey (meaning that they both offered afterschool programming and partnered with an outside provider for at least one program) and then did so. RAND also interviewed 30 school leaders, representing rural, suburban, and urban schools and asked them questions about partnering with afterschool STEM providers, including why they partner, how they learn about partners, and what they look for in partners. Finally, RAND included questions on afterschool programming in a spring 2023 American School District Panel (ASDP) survey administered to a nationally representative panel of 1,107 districts; 222 district leaders participated (ASDP, 2023). These questions aimed to capture additional information from district leaders, unique from principals.

Opportunity Insights – Diversifying Society’s Leaders? The Determinants and Causal Effects of Admission to Highly Selective Private Colleges

This is a QED study that leverages multiple sources of data to understand inequities in admission to highly selective colleges and universities by family income. The researchers examined the effects of enrolling in a highly selective college on employment and earnings for families from diverse socioeconomic backgrounds. To achieve these objectives, the researchers assembled a dataset linking several sources of administrative data: 1) information from parents’ and students’ federal income tax records; 2) college attendance information from the Department of Education; 3) data from the College Board and ACT on standardized test scores; and 4) application and admissions records from several highly selective public and private colleges covering 2.4 million students.

This dataset included longitudinal information on a set of pre-college characteristics (parental income, students’ SAT and ACT scores, high school grades, academic and non-academic credentials) as well as post-college outcomes (earnings, employers, occupations, graduate school attendance). Within this dataset, the researchers focused on the 12 Ivy-Plus colleges, 12 other highly selective private colleges, and highly selective state flagship public institutions.

The study focused on the entering classes of 2010-15 when analyzing attendance patterns and included earlier cohorts when analyzing long-term post-college outcomes. The first part of the analysis focused on exploring how family income predicted admission to highly selective colleges—controlling for all confounding characteristics—and understanding why higher-income students garnered an admissions advantage. The second part of the study leveraged idiosyncratic variation in admissions decisions for waitlisted applicants to estimate the effect of attending a highly selective college on long-term earnings and employment outcomes. The researchers focused in particular on understanding how this pattern might have varied for students from the bottom of the income distribution.

Thank you Meghan McCormick, Lina Eroh, and Brittany Sullivan for your contributions to this post.

Header image courtesy of FIRST and Argenis Apolinario