Download: Scaling Out-of-School Time STEM Programming

Last June, the Inspired Minds portfolio, which focuses on supporting out-of-school time (OST) STEM learning, called for proposals from researchers to help us understand how OST STEM programs have successfully scaled and under what contextual conditions. After reviewing 31 proposals, we selected Education Northwest to conduct the landscape analysis. The ensuing research report, Scaling Out-of-School Time STEM Programming, focuses on findings from youth-serving programs (ages 5-14) that meet the minimum dosage or duration needed to create a significant difference on STEM mindsets and skills.

As a national funder, we support a wide swath of out-of-school STEM programs across the country. Some occur afterschool, while others take place in the summer, in museums, or during informal time with family.  Some of the organizations we fund are early-stage with a few hundred students; others are national with hundreds of thousands of students. All share the goal of positively impacting students’ STEM knowledge, skills, and mindsets. 

Our intent with this research was to better understand whether there were shared elements that help a program find success in scaling up to regional and national levels. We were also interested in the evidence that these programs meet their academic and social goals with students, particularly students underrepresented in STEM.

We hope this report and webinar presentation help both early-stage and growth-stage organizations, including those in our portfolio and in the wider STEM ecosystem. For programs just starting their scaling plans, we believe that learning about the best practices in the landscape will help you iterate your model and create a scaling plan that maximizes your reach and impact. For more established organizations, we believe this report will give you the tools and resources you need as you plan on growing responsibly and sustainably. And for intermediaries and funders, this analysis can guide what to look for when supporting the field with technical assistance and capacity building.

We understand that not all programs are created with the intent to scale, but many organizations do have growth as a long-term goal. Of the 25 organizations interviewed by Education Northwest, 96% expressed a vision for additional growth and scale beyond where they are today. For those programs that do intend to increase their reach, we believe that a well-designed scale strategy must account for evidence of impact, not just at one location, but across sites. Whether on a per-student or per-site basis, the goal of scaling should be to build efficiency and increase access to high-quality programming.  

The models for program scale outlined in this report give us a framework for strengthening the OST STEM field. Additionally, the five themes below surfaced as opportunities to advance individual STEM organizations’ impact. As a foundation, we commit to challenging and supporting our grantees to set goals related to these themes.

1. Programs should aim for evidence of effectiveness at all levels of scale

It was clear from our analysis that many STEM programs that have scaled had limited prior rigorous evidence of effectiveness. We believe that, while backsolving for fidelity is not ideal, data can and should be collected at all stages of an organization’s journey in order to inform and improve the program model. It’s never too late to learn.

The recent NSF-funded convening on Common Measures in STEM articulated a range of intended outcomes from STEM learning and mapped on existing measurement and evaluation tools. This report layers on additional dimensions of the Coburn model when thinking about scale, specifically spread, depth, sustainability, and ownership. 

As a foundation, we aim to support programs in collecting and analyzing data that is relevant to testing their theory of change, meaningful to children and families, and predictive of later academic success. 

2. Family engagement leads to more impactful program design

Educator, family, and community outcomes are currently under-evaluated by programs, even though their importance is noted throughout the literature and program interviews. We commit to supporting grantees to use data to continuously improve their quality of service delivery and effectiveness, and expand measurement capacity to include community/family level and social-emotional outcomes.

Relatedly, a majority of programs reported that families were engaged as recipients of communication or guests at events, but not central to the program experience. 12% did not include any family or community engagement. We recommend this resource by STEM Next to all STEM grantees, and commit to supporting programs to empower families as active players and participants in their child’s STEM learning.

3. Listening and adapting to local needs is key

Based on their analysis, Education Northwest recommends needs-sensing to ensure a program fills a need in each local community and is a viable option for long-term investment. Programs should foster partnerships and provide data and evidence that include sample characteristics so potential future partner sites can understand the likelihood that an expansion of the program to their community will demonstrate similar results. 

We believe listening and learning from the local community is critical. We recommend and commit to supporting the Listen4Good tool to create and close feedback loops between participants and staff. 

4. Connecting to the school day deepens learning 

While 64% of programs connect OST and school-day learning, only 32% do so through direct communication between OST and in-school educators. While alignment to Next Generation Science Standards is a great start, better coordination can create a mutually beneficial partnership to deepen learning. 

We commit to supporting grantees who are testing innovative models to link in-school and out-of-school systems to deepen student learning.

5. Strong staffing is crucial to sustainability

Programs described staffing challenges as one of the largest barriers to scaling, specifically when it comes to decreasing turnover and recruiting skilled staff. The infographic here outlines strategies for staff training that prioritize sustainability, such as a train-the-trainer model to build long-term capacity in educators. 

We encourage and support our grantees to invest in their staff, including recruiting and retaining staff who reflect the identity of program participants. 

At the moment, the afterschool field is in a period of both suspension and innovation. Much of what makes the field unique, including hands-on group learning, is no longer possible in the wake of COVID-19. As you plan for the future, we invite you to consider how this report and infographic can support your strategic thinking. Do these scale strategies resonate? Are you thinking differently about scaling and program delivery post-COVID? Are there things we missed? Make sure you have shared how COVID has affected your program with the Afterschool Alliance through the survey here.

Additionally, we invite you to reach out with feedback to Gemma Lenowitz, Associate Program Officer of the Inspired Minds portfolio. Additionally, we want to express a deep thank you to Ashlie Denton, Elizabeth Gandhi, Christopher Mazzeo, Julie Petrokubi, Rebecca Moyer, Sara Taylor, Traci Fantz, and Lisa Rummler at Education Northwest for their work on this report.

Download the Report