As a leading education funder, we are dedicated to advancing the belief that “evidence matters” through novel, methodologically sound, and actionable research that uncovers programs and models with the potential to positively improve outcomes for all children. We believe research is a key way to increase awareness of what’s effective and to impact evidence-based decision making and practice across our four main grantmaking areas: early childhood, informal STEM education, and K-9 programs that include supporting expert educators and student-centered learning environments.
As we look to the year ahead, our vision for research is focused on three main areas: the validation of portfolio grantees, research aligned to each portfolio’s logic model and priority outcomes, and research about cross-cutting topics that have implications for all portfolios.
Validation of portfolio grantees
We approach program and grantee validation through high-quality evaluations that are designed to help grantees improve their program models and gain access to additional funding opportunities. Within this context, we focus on validations with the greatest possible rigor—such as largest sample size or most rigorous design—to yield impactful gains in ESSA tier, without timing or financial tradeoffs. For grantees, this can include:
- Validations that can be completed during the grant term and at a reasonable cost
- Validations that support programs moving from ESSA Tier 4 to 3, or from 3 to 2
- Partnering with other research-oriented foundations/organization to conduct RCTs of organizations or programs when they are ready to move to ESSA Tier 1
Research aligned to portfolio logic models and priority outcomes
As you may have seen in a recent blog post by our Vice President, Anu Malipatil, our team has been monitoring the key trends and shifts we believe are guiding the education landscape. In response, each of our portfolios has pivoted its approach to grantmaking efforts for 2022. To learn about the success of these updated strategies, we are prioritizing research that allows for insight on interim and priority outcomes for each strategy, crafting research questions aligned to the logic model of each portfolio.
Throughout the year, we will also consider and track research that comes to us organically that addresses questions relevant to each portfolio strategy.
Research on cross-cutting topics
For our last category, we are seeking out opportunities to fund research that addresses strategic questions relevant to several portfolios, such as:
- Family engagement in school and out of school: What are the issues that most engage families (across Pre-K through 8th grade-aged children)? What are productive ways to engage and build trust/engagement with families? How can systems and leaders support families to engage more on student academic priorities?
- Staffing aligned to grantee programs: What are the key drivers of staffing challenges in different settings (early childhood learning environments, K-12, after school/summer)? Can any creative interventions demonstrate efficacy at better retaining existing staff/recruiting and onboarding new staff?
- Value of in-person versus virtual delivery of programs: What are differences in efficacy and cost-effectiveness between in-person and virtual delivery of similar programs? Are some programs better delivered virtually than others? What are the characteristics and best practices for their delivery?
With the above framework in mind, I am excited to share the results of three grantee research studies that have concluded in Q1 of this year:
Validation research
Young Mathematicians in Worcester
In 2019, the Education Development Center (EDC) partnered with Worcester Child Development Head Start, Worcester Family Partnership, Worcester Public Libraries, and Quinsigamond Community College to create a Family Math Partnership: Young Mathematicians in Worcester, MA. This initiative was designed “to transform the way that educators and families interact with children around math—making it a common and doable family activity.”
After two years, EDC and Goodman Research Group evaluated the partnership and the impact of the initiative’s family math resources and professional development opportunities. The evaluation revealed an increased understanding of the importance of math among families and educators of preschool-aged children. Notably, families showed increased interest in and knowledge of early math, increased comfort helping their children with math, and an improved ability to come up with fun math activities to do with their children.
Courtesy of EDC
In addition, 90 percent of the parent survey respondents who used the materials agreed that the Young Mathematicians’ games and resources helped them talk with their children about math. Furthermore, 88 percent also agreed that the games and books helped them feel less anxious about math.
The evaluation also found that the Young Mathematicians Family Math Partnership promoted an increased understanding of the importance of math for educators and educators showed an increased interest in early math. Educators in the program also reported increased comfort engaging in math with young children and supporting family math. Importantly, educators grew in their confidence to support families to do math at home with their children.
Courtesy of EDC
This project began in the fall of 2019 and continued without pause despite Covid and program closures. Early childhood programs in Worcester shifted to remote learning from March 2020 through April 2021. In response to educators’ need for support in virtual instruction, the partnership immediately adjusted the professional learning sessions to an online format and worked with educators to develop virtual mathematics instructional strategies and practices. This approach enabled educators to support children and families to learn and practice math at home.
Unfortunately, the evaluation was limited in its ability to assess the impact of the initiative on children’s math learning. A planned secondary analysis of the partner programs’ child assessment data was not able to be conducted because, due to the pandemic, programs canceled one full assessment cycle and had inconsistent participation in subsequent virtual assessments. However, educators reported that the program mitigated the risks to children’s learning posed by the pandemic and helped to strengthen family math engagement and learning. Moving forward, the team at EDC will explore potential additional research partnerships to better understand the impact of the program on children’s math learning.
Image courtesy of SRI Education
Logic model & priority outcomes research
Evaluations of Mobile Messaging on Responsive Caregiving
Mobile messaging programs are a low-cost, scalable approach to building parents’ knowledge and capacity to support their children’s development, but there is little research on how mobile messaging affects parents’ engagement in responsive caregiving and other positive parenting behaviors, particularly with infants and toddlers. To better understand mobile messaging in this context, SRI Education partnered with Bright by Text (BBT) and Consejos de Univision (developed with Too Small to Fail) to evaluate the impact of mobile messages on key parenting outcomes.
BBT provides “just-in-time” child development and parenting guidance to caregivers via text messages that focus on child milestones, early learning, social-emotional skills, and health and wellness. To understand BBT’s impact, SRI conducted a randomized controlled trial with parents of 18- to 36-month-old children, and who were new enrollees to BBT. Participants in the treatment group received 12 weeks of BBT messages, while participants in the control group received placebo messages. Data collected through parent surveys, implementation surveys, and parent interviews showed:
- Relative to the control group, participating in BBT did not significantly impact parents’ responsive caregiving, or their attitudes, knowledge, and confidence.
- Although the sample sizes were small, moderation analysis suggests that BBT did differentially impact: days in a typical week spent reading for parents who reported an annual household income >$50,000 and for parents who had at least an associate’s degree or higher; minutes spent reading on a typical day for parents who reported an annual household income <$50,000; engagement in shared reading activities for parents of boys; and attitude toward parenting for parents who reported a household income <$50,000.
- Interviewees reported the primary benefit of the BBT messages was that they could access the content on their own time and that the messages primarily helped them as parents.
- Most parents (60 percent) reported saving the message content for later, whereas a third (33 percent) reported trying it right away. Nearly all parents (96 percent) said they read every message or most of the messages and most parents (90 percent) reported rereading the messages often or sometimes.
These findings add to the small body of literature on mobile messaging as a support for parents. Although the BBT study did not find significant impact on changing parent behaviors and outcomes, the overall results of both BBT and Consejos de Univision studies suggest that “providing parents with direct guidance and conversational prompts they can apply to a range of activities may be an effective strategy to promote responsive caregiving.”
Image courtesy of BellXcel
Exploring How a Partnership with BellXcel Influences Systems Changes
BellXcel provides an evidence-based software platform designed to help schools and community-based partners design, manage, and deliver high-quality summer and afterschool programs for youth, including programs focused on improving math skills and engagement.
In 2021, BellXcel commissioned the Sperling Center for Research and Innovation (SCRI) to study BellXcel’s organizational impact on partners and the impact the BellXcel partnership model has on building bridges to community stakeholders.
SCRI conducted 30-60 minute interviews with program leaders from nine BellXcel summer youth development partners. Using qualitative data analysis, they learned that BellXcel partners benefited most in these five impact categories:
- Program outcome and evidence: Program evaluation, program outcomes, and the use of data and evidence
- Community relationships: Relationships with local community partners, such as schools, other youth development organizations, families, retail, and industry
- Organizational confidence: Individual and/or organizational confidence in running new and/or existing youth programs
- Philanthropy and fundraising: Funder relationships and funding for programming
- Program planning and operations: Planning and operations of other youth programming
These categories have the potential to affect staff throughout an organization, lead to greater community reach and engagement, and to foster new relationships and opportunities that may not have happened in the absence of the BellXcel partnership. The organization will use these initial findings to identify components of the partnership and technical assistance model that are levers for change, to identify priority impact outcomes, and to design and implement additional measurement tools.
Overdeck Family Foundation will use these new findings from our grantees to continue to learn and improve. Guided by one of our three core organizational values, “learn better, together,” we believe it is essential to invest in research and to share findings with our grantees and the field—even when results may be different than unexpected. As a foundation, we use these valuable insights to analyze and adjust our grantmaking strategies and investments throughout the year, and we hope this transparent practice can help nonprofits and other funders make informed decisions by reflecting on what has already been learned.
We look forward to continuing this important work unlocking evidence across our three main research areas this year.
Thank you to Laurie Sztejnberg, Katie Lim, and Gemma Lenowitz for your contributions to this blog post.
Header image courtesy of BellXcel