2023 Bright Spots: Unlocking Evidence

Here, we highlight stories of how our funding helped four grantees unlock evidence to advance field knowledge or validate program models through studies and research. To learn more about our grantmaking efforts and read additional stories from our grantee partners, explore the 2023 Grantmaking & Impact Report.

Read the Report

FIRST

Science, technology, engineering, and math (STEM) occupations are growing rapidly, with STEM outpacing growth of non-STEM by over four times. But lack of interest in STEM is a major reason people don’t enter STEM professions.

FIRST^® seeks to inspire young people to consider STEM experiences and careers by engaging them in robotics programs that build science, engineering, and technology skills and inspire innovative solutions to real-world problems. Reaching 336,000 U.S. students in 2023, FIRST uses evidence-based strategies to increase student interest in STEM, including hands-on learning, teamwork on real-life problems, exposure to careers and adult mentors, and a robotics competition that completes the experience.

FIRST’s commitment to building evidence of impact differentiates it in the out-of-school time field. Since 2018, Overdeck Family Foundation has supported the organization’s longitudinal study examining how participation in FIRST during middle and high school is associated with long-term outcomes. This year, FIRST completed the ninth wave of follow-up surveys.

Nine years after participants’ original enrollment, the study continues to find sustained positive benefits of FIRST across all identity groups (gender, race/ethnicity, and family income level). Compared to demographically similar students who did not enroll in the program, students who participated in FIRST:

• Reported more positive STEM attitudes (2.2 times more interested in STEM, 1.9 times more likely to report strong STEM identity);
• Were more likely to declare a STEM-related major in college, with 81 percent of FIRST alumni declaring a STEM major, compared to 64 percent in the comparison group; and
• Participated in STEM careers at higher rates after finishing college, with 61 percent of FIRST alumni working in a STEM field, compared to 44 percent in the comparison group.

FIRST has also been building the evidence base for its FIRST LEGO® League Explore (ages six through 10) and Challenge (ages nine through 16) programs. A recent implementation study indicates positive gains in outcomes like interest in STEM, programming and coding skills, and understanding STEM content, as well as improved social-emotional outcomes like creativity, teamwork, and problem-solving.

Overdeck Family Foundation has played an instrumental role in supporting FIRST in its pursuit of scale and deeper evidence of impact by supporting the organization to increase its reach (especially in New Jersey), pursue a multi-year longitudinal evaluation, and navigate the challenges of rebuilding reach after the significant impact of shutdowns caused by the COVID-19 pandemic.

LENA

Responsive adult-child interactions, known as conversational turns, are among the most predictive metrics of child outcomes. Yet only four percent of children 18-24 months old consistently receive the optimal level of interaction in childcare settings.

LENA supports young children’s parents, caregivers, and educators with evidence-based programs designed to encourage meaningful adult-child interactions. Its small wearable device for children, often referred to as a “talk pedometer,” measures conversational turns, delivering language environment data that helps adults increase in interactive conversation and highlights opportunities to improve classroom quality and child outcomes.

LENA Grow—the organization’s newest and fastest-growing program—utilizes this talk pedometer technology to offer professional development and individualized strengths-based coaching for early childhood educators. While other LENA programs already had robust evidence of effectiveness, Grow demonstrated potential impact through internal data but had limited external research.

In 2021, LENA launched two independent studies to assess the impact of Grow: a randomized controlled trial (RCT) in collaboration with SproutFive, a childcare provider in Columbus, OH, and a quasi-experimental study in collaboration with Next Door, a childcare provider in Milwaukee, WI. Both studies found positive outcomes, aligned to three areas:

• Grow teachers increased conversational turns after participating in the program;
• Grow teachers increased their job satisfaction compared to non-Grow teachers; and

The RCT in Columbus found that children in Grow classrooms experienced larger gains in language and literacy skills, compared to children in non-Grow classrooms. The quasi-experimental study in Milwaukee found similar benefits of Grow on Head Start students’ literacy skills.

In Columbus, children enrolled in Grow classrooms saw a statistically significant increase in conversational turns of 13 percent. Children with interaction levels below the national average saw even greater gains, experiencing 80 percent more interaction by the end of the program than they had at the beginning. This increased interaction was also associated with child-level outcomes on standardized measures. Participating children in Columbus saw pre/post average scaled score increases of 39 points on the TS GOLD® child literacy assessment, compared to a loss of two points for their counterparts who did not participate in Grow.

Although more research is needed, this enhanced strength of evidence provides current and potential LENA partners and policymakers a better understanding of the Grow program’s potential impact and the value of expanding its reach to even more families and educators.

Overdeck Family Foundation has supported LENA since 2017, when it reached around 6,000 children through its programming. Our funding has helped the organization continue to innovate on its program model and generate rigorous evidence to boost performance of early language outcomes, as well as scale to reach 29,000 children in 2023. Carly Roberts, Associate Program Director of Out-of-School Learning at the Foundation, has served on LENA’s Board of Directors since 2020. In addition, the Foundation has provided capacity-building support focused on helping LENA better understand and reduce its cost per user as well as enhance its data capacity.

New Jersey Tutoring Corps

Recent New Jersey Student Learning Assessments data found that only 46 percent of third grade students scored proficient or better in math, down from 51 percent in 2019. In English language arts, that rate dropped from 50 percent in 2019 to 42 percent in 2023.

The New Jersey Tutoring Corps provides high-dosage tutoring in math and literacy for Pre-K through eighth grade students across New Jersey. The program uses one of three evidence-based models to deliver its support directly to students: embedded during the school day, scheduled after school, or integrated into summer programming.
In SY 2022-23, the New Jersey Tutoring Corps piloted its embedded school-day program designed to provide high-dosage tutoring as part of the school day for students. Groups of one to four students met for 30- to 60-minute sessions two to three times per week, with the same tutor for the duration of the program, which typically lasted 12 to 15 weeks.

An external evaluation of the school-day program, conducted by Dr. Ellen Behrstock-Sherratt, found that, of the approximately 500 students served during the 12 to 15 week evaluation period, the percentage performing at grade level in math improved from 16 to 40 percent across grade levels; in literacy, the percent of students at grade level grew from 23 to 40 percent. Proficiency gains were particularly large in math, an important finding given a substantial body of research linking early math skills to long-term benefits for students. Importantly, students’ confidence in their own skills as learners and their academic content knowledge also improved.

Overdeck Family Foundation helped conceive and found the New Jersey Tutoring Corps in 2021, as a response to COVID-induced learning loss across the state. Since then, we have provided consultative support and general operating grants, which helped the organization unlock additional sources of funding, establish its 501(c)(3) status, and work towards a long-term strategy to improve sustainability and increase reach.

OnYourMark

In 2022, only 33 percent of fourth grade students nationwide performed at or above the NAEP Proficient level on the reading assessment.

OnYourMark Education provides high-impact virtual tutoring for 20 minutes a day, four days a week, to students in kindergarten through second grade. The organization recruits, trains, and manages tutors from across the country, matching them with schools seeking to increase students’ reading ability. Throughout the school year, tutors meet with students using a video conferencing platform during regularly scheduled small-group instruction or independent learning. Grounded in the science of reading with a focus on foundational literacy skills, OnYourMark’s tutoring reaches more than 2,000 students at 23 partner schools across the country.

In 2022, Overdeck Family Foundation was the sole funder of a randomized controlled trial (RCT) designed to understand the potential impact of OnYourMark’s virtual tutoring on student learning. Conducted by a team of researchers from Stanford University led by Susanna Loeb, the study examined over 2,000 students across 12 schools. About half of the students were randomly assigned to receive tutoring from OnYourMark while the other half were randomly assigned to receive instruction as usual. Study findings revealed that one-on-one virtual OnYourMark tutoring had a positive impact on students’ early literacy skills, as measured with an assessment called the DIBELS. OnYourMark had the largest effects on kindergarteners’ knowledge of letter sounds and first graders’ decoding skills. Researchers concluded that impacts on these domains translated into as much as 24 extra days of learning letter sounds for kindergartners and 51 extra days of decoding for first graders.

OnYourMark’s RCT is the first of its kind to test the impact of virtual tutoring on young learners. The results point to the potential of virtual instruction to reach more students, whether they are in rural schools or schools that are struggling to recruit and hire tutoring staff, as well as potential insight into how high-dosage tutoring can become more cost-effective and sustainable in the future through the use of technology.

Bolstered by this evidence, OnYourMark continues to innovate on ways to improve the cost-effectiveness of high-quality tutoring. This year, the organization launched a pilot with Overdeck Family Foundation grantee Illuminate Literacy building on previous Foundation-funded research into the effectiveness of blended tutoring models. The pilot will unearth additional insight into the efficacy of tutoring programs when paired with technology platforms.

In addition to providing funding for OnYourMark’s RCT, which sets up the organization to meet “Tier 1” evidence standards as established by ESSA, Overdeck Family Foundation also provides OnYourMark with general operating support that has enabled it to iterate on the delivery of its model, improve its data systems to handle increased reach, and invest in its growth teams, resulting in more students receiving high-impact virtual tutoring and, potentially, reading on grade level.

Header image courtesy of Imagine Science