Scale-up Grants Can Be More Than Just Running a Large-scale RCT

Written By: Denis Newman and Jenna Zacamy from Empirical Education Inc.

Our study of the scale-up of an i3-funded innovation (Newman et al., 2015) casts doubt on the value of a large-scale RCT as the only way to understand impact of the innovation at scale. We studied the same innovation simultaneously in two contexts: 1) in the treatment group of an RCT and 2) in a large set of districts who were brought on board outside the constraints of the RCT, which we sometimes called “in the wild.” The program was Reading Apprenticeship, an approach to improving content-area literacy skills, and the innovation was to improve scalability of Reading Apprenticeship by fostering school teams consisting of teachers from different content areas. We found that the growth process and decline of participation in the schools “in the wild” was very different from the RCT schools. Importantly, teachers in the wild had a greater commitment to the success of the innovation in their school, not just in their own classrooms.

This was all in the context of our evaluation of a 2010 cohort Validation grant: Reading Apprenticeship Improving Secondary Education (RAISE). During this time, Validation grants were funded at the level of current Scale-up grants.  The funding allowed the grantee, WestEd’s Strategic Literacy Initiative (SLI), to support not only 42 schools in two states participating in a three-year RCT, but also 239 schools in four states implementing the innovation.  Remarkably, SLI funded not only the RCT, but also a separate four-year evaluation to track processes within schools (called the “scale-up” schools) through multiple surveys of approximately 850 teachers. Many of the same survey questions were asked of the scale-up and RCT teachers.  Scale-up teachers were not asked about their classroom practices and we didn’t collect student-level outcomes for the scale-up schools. This parallel study of the innovation allowed us to compare teacher teams in the RCT treatment schools with those in the scale-up schools, in terms of their attitudes and the growth or loss of participants in the team. 

While the researchers carefully controlled the growth and loss of participants in the RCT schools, scale-up schools saw much more variation. Some scale-up schools more than doubled the original number of participating teachers, while others lost all their participants. While an RCT is heavily penalized for attrition of a school, in our scale-up study, the gains and losses became a topic of the research.  We looked in the first year of the innovation for the predictors of later growth or reduction of the school team as an important mechanism for scaling an innovation.  The strongest predictors were participation in team meetings and a commitment to making Reading Apprenticeship work in the school.  Comparing scale-up to RCT treatment schools, we see these traits more strongly in scale-up schools (on average, i.e., even including schools that saw a net loss in participants).  The attitudes of teachers in schools recruited for the RCT were not favorable for scaling participation over time, and we can speculate that without the effort of researchers, there would have been more attrition than was observed. 

Since we didn’t survey the scale-up teachers about their classroom practices, we couldn’t determine whether the practices that were found to mediate improvements in student outcomes in the RCT were more or less present in scale-up than in RCT treatment schools. Our study of scale-up, however, raises the question of whether the constraints of an RCT may suppress team building, and possibly efficacy, of the innovation. The next round of i3 scale-up projects has the opportunity to examine schools outside of the RCT, systematically surveying teachers under RCT constraints and “in the wild.” Conclusions from such a comparative study would not have the internal validity of an RCT, but it will help us understand the processes by which programs can be put on a path for growth and whether the conditions for success—such as mediators of student growth—can be put into practice.