This Campbell systematic review examines the effects of 'No Excuses' charter schools on students' math and literacy achievement gains compared to similar students in public schools. The review summarizes evidence from 18 studies, including five randomized controlled trials and 13 quasi-experimental studies.
'No Excuses' charter schools, on average, produced larger math and literacy achievement gains for their students than their public school peers – with higher gains for math. These benefits increase for three years, at which point the achievement gains stabilized or returned to lower gains.
In response to the school accountability movement and the move toward market-oriented education policies in recent decades, many schools and districts have felt increased pressure to find effective ways to boost student achievement. However, many districts—largely those serving low-income black and hispanic students in urban settings—have still struggled to make sufficient academic gains (Barton & Coley, 2010; Jacob & Ludwig, 2008; Yeh, 2016). One model of urban school reform aimed at leveling achievement disparities in the public school systems that has garnered attention (both positive and negative) is the No Excuses model. The No Excuses model is driven by the philosophy that there should be no excuses (e.g., poverty) for low academic performance. This model has found its footing in charter schools—publicly funded but privately managed schools exempt from the restrictions of the laws and regulations affecting non-charter public schools. Due to the autonomy afforded to the management of charter schools, educators have the freedom to implement the tenets of the No Excuses model without the restrictions placed on non-charter public schools.
Along with the expansion of charter schools, overall, No Excuses charter schools have been increasing in number since their recognition in the 1990s. No Excuses charter schools primarily draw from low-income and minority communities with the aim of minimizing racial and income-based achievement gaps by improving the academic performance of their students. Although a definitive definition of the No Excuses charter school model does not exist, most definitions share a number of key characteristics—most notably: (a) high academic expectations, (b) rigid and consistent discipline, (c) extended instructional time, (d) intensive teacher training, and (e) parental involvement (Dobbie & Fryer, 2013; Whitman, 2008; Carter, 2000; Golann, 2015).
This systematic review and meta-analysis aims to examine the available evidence on the impacts of No Excuses charter schools on students’ math and literacy achievement, relative to students enrolled in traditional public schools. The focal research questions are as follows:
We conducted a comprehensive systematic literature search to identify all eligible studies, regardless of publication status. For the purposes of this review, the term “studies” is used to refer to separate reports and “samples” to refer to independent samples of students within (and over multiple, in some cases) studies. The search was conducted using electronic databases, internet search engines, citations in previous meta-analyses and literature reviews, citations in research reports screened for eligibility, conference listings, hand searches of relevant journals, and correspondence with experts in the field. We screened redactions and erratum for potentially eligible studies. We used date-of-publication restrictions limiting our search years to 1990 or later as No Excuses charter schools did not exist until the mid-1990s.
Published or unpublished studies using randomized controlled trials (RCTs) or quasi-experimental design (QEDs) to assess the effects of No Excuses charter schools on math and literacy achievement of elementary and secondary students were included in this review. To be eligible for inclusion in this review, studies must have been conducted or reported between January, 1990 and June, 2016. The date range reflects the time period during which No Excuses charter schools have been present in the U.S. School system. In addition, studies must have reported or controlled for baseline and comparison data, and comparisons groups must have comprised students attending traditional public schools. Studies determined not to reflect charter schools using a No Excuses model, or comparing No Excuses charter schools to other charter schools were excluded from this review.
A team of reviewers assessed each title and abstract for preliminary eligibility, which was confirmed during full-text screening. All studies that met the eligibility criteria in the citation and abstract, and full-text screening were coded using a detailed coding instrument developed by the authors. Data extracted from primary studies included information on methodology, design, intervention and comparison group characteristics, math or literacy outcome details and characteristics, etc. Coders met to review the coding agreement and any discrepancies were discussed and resolved by consensus. This review contains a description of the intervention, a summary of the authors’ findings and conclusions, and a narrative synthesis of the identified literature.
We report standardized mean differences with Hedges’ g. Separate meta-analyses were performed to synthesize the studies assessing the effects of No Excuses charter schools on math outcomes and to synthesize the studies assessing the effects of No Excuses charter schools on literacy outcomes. Separate meta-analyses were used to estimate the cumulative effects after one, two, three, and four years of attendance at No Excuses charter schools. Meta-analyses separately report the estimated effects from RCTs and QEDs. We calculated Q, τ2, and I2 to assess heterogeneity.
Moderator analyses were planned to explain between-study heterogeneity in effects for several school-level moderator variables, including percent male, percent minority, percent free/reduced lunch, percent of students provided special education services, school size, average class size, and years that the charter school has been active. However, few studies reported these characteristics at the school level; further, several variables (such as percent male, percent minority, percent free/reduced lunch, and percent special education services) did not have sufficient variability between studies to serve as useful moderators for effect sizes. As such, no planned moderators were consistently significant predictors of math or literacy achievement gains.
The initial search of both academic and grey literature identified a total of 4,554 citations. After duplicates and removal of studies prior to 1990, 2,841 citations remained. Titles and abstracts were screened for initial relevance and 2,386 of the remaining reports were excluded for irrelevance or ineligibility. Full-texts of the remaining 455 studies were retrieved (no unretrievable reports) and screened for eligibility—18 of which passed full-text screening for inclusion in the primary review resulting in 73 independent samples over four years. Analysis of these samples examined the effects of No Excuses charter schools on the math and literacy achievement of 70,121 elementary and secondary students. Five of the included studies were RCTs (Abdulkadiroglu et al. 2011; Angrist et al. 2010; Curto & Fryer, 2014; Hastings, Nekilson, & Zimmerman, 2012; Tuttle et al., 2015) and the remaining 13 used quasi-experimental designs. The weight given to RCTs in the overall estimates (which pool the QED and RCT effects) of math and literacy effects were approximately 8% for year 1, 7% for year 2, 6% for year 3, and 5% for year 5. Studies included nongovernmental reports (9), journal articles (5), dissertations (2), working papers (1), and papers presented at conferences (1). All studies evaluated results using comparison groups comprised of students attending traditional public schools.
The effect of No Excuses charter schools on math and literacy outcomes was found to be significant in this meta-analysis. Similar to previous findings, a substantially smaller but significant effect on literacy was found.
The year 1 math analyses included 73 separate samples. The overall mean effect size for math outcomes was positive and significant (g = 0.202, 95% CI [0.148; 0.257]). The mean effect sizes were significant for QEDs (g = 0.207, 95% CI [0.148; 0.266]) and RCTs (g = 0.149, 95% CI [0.015; 0.283]) and were not significantly different from each other. There was significant heterogeneity in the estimates of the effect size (τ2 = 0.0430). Given the significant heterogeneity and potential bias for non-equivalent groups, the mean effect size should be interpreted with caution (I2 = 89.4, Q = 676.58). At follow-up years, the mean effect on math outcomes remained significant: Year 2 (n = 48), g = 0.293, p < 0.001; Year 3 (n = 33), g = 0.434, p < .001; Year 4 (n = 22), g = 0.299, p < 0.001. The estimated mean effects after adjustment for potential publication bias using trim-and-fill for all four years were also significant: Year 1 (0 trimmed, g = 0.207, p < 0.001), Year 2 (4 trimmed, g = 0.250, p < 0.001), Year 3 (6 trimmed, g = 0.331, p < 0.001), Year 4 (0 trimmed; g = 0.299, p < 0.001).
The year 1 literacy analyses also included 73 separate samples. The overall mean effect size for literacy outcomes was positive and significant (g = 0.069, 95% CI [0.034; 0.104]). The mean effect sizes were significant for QEDs (g = 0.061, 95% CI [0.025; 0.097]) and RCTs (g = 0.164, 95% CI [0.019; 0.309]) and were not significantly different from each other. Heterogeneity was minimal (I2 = 70.6, τ2 = 0.0121, Q = 244.77). At follow-up years, the effect on literacy outcomes remained significant and attenuated from the math results: Year 2 (n = 48), g = 0.120, p < 0.001; Year 3 (n = 33), g = 0.212, p < .001; Year 4 (n = 22), g = 0.178, p < 0.001). The Egger’s test indicated potential publication bias for the Year 1 literacy results (p = .047). After adjusting the mean effect using trim-and-fill, the Year 1 mean effect was still significant (7 trimmed, g = 0.041, p = 0.028), as were all follow-up years: Year 2 (6 trimmed, g = 0.091, p < 0.001), Year 3 (3 trimmed, g = 0.168, p < 0.001), Year 4 (1 trimmed, g = 0.178, p < 0.001).
The present review identified available evidence regarding the effectiveness of No Excuses charter schools on math and literacy achievement. This review found multiple studies providing tentative support for the No Excuses charter school model as an effective intervention for improving students’ math and literacy achievement, with limitations. Further, this intervention was found to improve students’ math achievement more significantly that literacy.
There were several significant limitations to this review. First and of utmost importance, studies were inconsistent in reporting (or failed to report) important sample characteristics (e.g., percent of students receiving special education services, number of students exiting or expelled after enrollment, etc), severely limiting interpretation of the equality of comparison groups and factors impacting differences in outcomes. Second, follow-up years suggest a sustained significant effect on both math and literacy achievement, however these results must be interpreted with caution as the number of samples decreased substantially at follow-up years with one study (Tuttle, Teh, Nichols-Barrer, Gill, & Gleason, 2010), accounting for the majority of the samples. Third, studies included in this review did not report data indicating the degree to which each characteristic of No Excuses charter schools was implemented. This is a significant limitation as it prohibits the interpretation of the presence, absence, or dosage of the individual characteristics of No Excuses charter schools. Fourth, only five of the 18 included studies (approximately 5.75% of the independent samples) employed random assignment, limiting interpretation of the equality of the comparison groups due to the small number in independent samples using random assignment. Fifth, a large number of the included studies examine the effects of the schools of a specific charter management organization: Knowledge is Power Program (KIPP). Specifically, approximately 57% of the weight in Year 1 effect size estimates came from KIPP samples. At year 2, 3, and 4 follow-ups, more that 57% of the weight in the effect size estimates are from KIPP samples. Though informative, the results of this review cannot be considered generalizable to all No Excuses charter schools. Of note, the results of this review reflect one unique model of charter schools and must not be generalized to all charter schools. Finally, most studies measured effects with only one, annual standardized achievement measure.