You are here
Topic Areas
Accountability
Full Abstract
In this paper, we use NWEA MAP test data to examine variation in students’ achievement and growth during the pandemic across multiple dimensions. Consistent with prior evidence, we find that students’ test scores in fall 2021, on average, were substantially below historic averages. Moreover, the average scores of students of color, students attending high poverty schools, and students in elementary school were more negatively impacted, and more so in math than reading. We present novel evidence on the distributions of test scores and growth in fall 2021 relative to pre-pandemic distributions, finding disproportionately larger declines for students with lower previous achievement levels across districts. However, between districts, there was considerable variation in the extent to which their fall 2021 achievement and growth distributions shifted from their historical distributions by subject, student subgroups, and baseline achievement levels. Therefore, accurately targeting students and choosing interventions for pandemic-related recovery will require careful assessment by districts of their students’ achievement and growth in the 2021-22 school year (and into the future): assuming that students in a district reflect the national trends of achievement will often lead to incorrect conclusions about the degree to which they suffered pandemic-related learning losses and the amount of support they will need to recover.
Citation: Dan Goldhaber, Thomas J. Kane, Andrew McEachin, Emily Morton (2022). A Comprehensive Picture of Achievement Across the COVID-19 Pandemic Years: Examining Variation in Test Levels and Growth Across Districts, Schools, Grades, and Students. CALDER Working Paper No. 266-0522
Full Abstract
Using testing data from 2.1 million students in 10,000 schools in 49 states (plus D.C.), we investigate the role of remote and hybrid instruction in widening gaps in achievement by race and school poverty. We find that remote instruction was a primary driver of widening achievement gaps. Math gaps did not widen in areas that remained in-person (although there was some widening in reading gaps in those areas). We estimate that high-poverty districts that went remote in 2020-21 will need to spend nearly all of their federal aid on academic recovery to help students recover from pandemic-related achievement losses.
Citation: Dan Goldhaber, Thomas J. Kane, Andrew McEachin, Emily Morton, Tyler Patterson, Douglas O. Staiger (2022). The Consequences of Remote and Hybrid Instruction During the Pandemic. CALDER Working Paper No. 267-0522
Full Abstract
Testing students and using test information to hold schools and, in some cases, teachers accountable for student achievement has arguably been the primary national strategy for school improvement over the past decade and a half. Tests are also used for diagnostic purposes, such as to predict students at-risk of dropping out of high school. But there is policy debate about the efficacy of this usage, in part because of disagreements about whether tests are an important schooling outcome. We use panel data from three states – North Carolina, Massachusetts and Washington State – to investigate how accurate early test scores are in predicting later high school outcomes: 10th grade test achievement, the probability of taking advanced math courses in high school, and graduation. We find 3rd grade tests predict all of these outcomes with a high degree of accuracy and relatively little diminishment from using 8th grade tests. We also find evidence that using a two-stage model estimated on separate cohorts (one predicting 8th grade information using 3rd grade information, and another predicting high school outcomes with 8th grade information) only slightly diminishes forecast accuracy. Finally, the use of machine learning techniques increases accuracy of predictions over widely used linear models, but only marginally.
Working Paper 235-0520 was originally released in May 2020. This is an updated version, released August 2021.
Citation: Dan Goldhaber, Malcolm Wolff, Timothy Daly (2021). Assessing the Accuracy of Elementary School Test Scores as Predictors of Students’ High School Outcomes. CALDER Working Paper No. 235-0821-2
Full Abstract
In this study, we use microdata from 12 Florida county-level school districts to examine the effects of early grade retention on the short-, medium-, and long-term outcomes of English learners in a regression discontinuity design. We find that retention in the third grade coupled with instructional support substantially improves the English skills of these students, reducing the time to proficiency by half and decreasing the likelihood of taking a remedial English course in middle school by one-third. Grade retention also roughly doubles the likelihood of taking an advanced course in math and science in middle school, and triples the likelihood of taking college credit-bearing courses in high school for English learners. We do not find any adverse effects of the policy on disciplinary problems or absences among English learners.
CALDER WP 211-0119-1 was originally released in January 2019. An updated version was released in March 2020.
Citation: David Figlio, Umut Özek (2019). An Extra Year to Learn English? Early Grade Retention and the Human Capital Development of English Learners. CALDER Working Paper No. 211-0119-1
Full Abstract
Instructional time is a fundamental educational input, yet we have little causal evidence about the effect of longer school days on student achievement. This paper uses a sharp regression discontinuity design to estimate the effects of lengthening the school day for low-performing schools in Florida by exploiting an administrative cutoff for eligibility. Our results indicate significant positive effects of additional literacy instruction on student reading achievement. In particular, we find effects of 0.05 standard deviations of improvement in reading test scores for program assignment in the first year, though long-run effects are difficult to assess.
Citation: David Figlio, Kristian Holden, Umut Ozek (2018). Do Students Benefit from Longer School Days? Regression Discontinuity Evidence from Florida's Additional Hour of Literacy Instruction. CALDER Working Paper No. 201-0818-1
Full Abstract
Educational accountability policies are a popular tool to close the achievement gaps between advantaged and disadvantaged students. However, these policies may exacerbate inequality if families from advantaged backgrounds are better able to advocate for their children and thus circumvent policy. We investigate this possibility in the context of the early grade retention policy in Florida, which requires all students with reading skills below grade level to be retained in the third grade, yet grants exemptions under special circumstances. We find that Florida’s third-grade retention policy is in fact enforced differentially depending on children’s socioeconomic background, especially maternal education. Holding exemption eligibility constant, scoring right below the promotion cutoff results increases the retention probability 14 percent more for children whose mothers have less than a high school degree as compared to children whose mothers have a bachelor’s degree or more. We also find that the discrepancies in retention rates are mainly driven by the fact that students with well-educated mothers are more likely to be promoted based on subjective exemptions such as teacher portfolios.
Citation: Christina LiCalsi, Umut Özek, David Figlio (2016). The Uneven Implementation of Universal School Policies: Maternal Education and Florida’s Mandatory Grade Retention Policy. CALDER Working Paper No. 167
Full Abstract
This paper examines the effect of school turnaround in North Carolina elementary and middle schools. Using a regression discontinuity design, we find that turnaround led to a drop in average school-level math and reading passing rates and an increased concentration of low-income students in treated schools. We use teacher survey data to examine how teacher activities changed. Treated schools brought in new principals and increased the time teachers devoted to professional development. The program also increased administrative burdens and distracted teachers, potentially reducing time available for instruction. Teacher turnover increased after the first full year of implementation. Overall, we find little success for North Carolina’s efforts to turn around low-performing schools under its federally funded Race to the Top grant.
Citation: Helen Ladd, Jennifer A. Heissel (2016). School Turnaround in North Carolina: A Regression Discontinuity Analysis. CALDER Working Paper No. 156
Full Abstract
One of the concerns over high-stakes testing is the incentive for teachers to alter the scores of their students. We investigate the effects of teacher cheating on subsequent student achievement, attendance, behavior and educational attainment. We find that test scores drop below expected levels in the first year post-cheating year. These effects persist for reading and ELA, but not for math. The drop in later test scores appears to be due in part to a reduction in access to remediation services. We also find some evidence that cheated middle-school students may be more likely to drop out of high school.
Citation: Jarod Apperson, Carycruz Bueno, Tim Sass (2016). Do the Cheated Ever Prosper? The Long-Run Effects of Test-Score Manipulation by Teachers on Student Outcomes. CALDER Working Paper No. 155
Full Abstract
In this paper we examine how failing to make adequate yearly progress under No Child Left Behind (NCLB), and the accountability pressure that ensues, affects various non-achievement student behaviors. Using administrative data from North Carolina and leveraging a discontinuity in the determination of school failure, we examine the causal impact of accountability pressure both on student behaviors that are incentivized by NCLB and on those that are not. We find evidence that, as NCLB intends, pressure encourages students to show up at school and to do so on time. Accountability pressure also has the unintended effect, however, of increasing the number of student misbehaviors such as suspensions, fights, and offenses reportable to law enforcement. Further, this negative response is most pronounced among minorities and low performing students, who are the most likely to be left behind. iii
Citation: John B. Holbein, Helen Ladd (2015). Accountability Pressure and Non-Achievement Student Behaviors. CALDER Working Paper No. 122
Full Abstract
Current federal policy emphasizes a focus on turning around schools that consistently fail to serve their students and communities. At first glance, the identification of chronically low-performing schools and successful turnarounds may seem straightforward. Characterizing school performance, however, requires resolving multiple dilemmas about schools that have not been previously addressed within the literature on turnaround. Furthermore, the literature lacks empirical evidence on the frequency of turnaround in these low-performing schools. This paper addresses these issues involved in identifying low performance and turnaround among schools. Additionally, it examines the long-term performance trajectories of chronically low-performing (CLP) elementary and middle schools in multiple states to identify schools that have shown rapid improvement (designated turn around [TA] schools), schools that have shown moderate improvement (MI), and schools that are persistently not improving (NI). The findings indicate school turnaround is an uncommon event in these low-performing schools, though not rare—approximately 10 to 30 percent of CLP schools, depending on the state and school level, are identified as TA schools based on improvements in performance.
Citation: Michael Hansen, Kilchan Choi (2013). Chronically Low-performing Schools and Turnaround: Evidence from Three State. CALDER Working Paper No. 60
Full Abstract
Test-based accountability has become the new norm in public education over the last decade. In many states and school districts nationwide, student performance on standardized tests plays an important role in high-stakes decisions such as grade retention. This study examines the effects of grade retention on student misbehavior in Florida, which requires students with reading skills below grade level to be retained in the 3rd grade. The regression discontinuity estimates suggest that grade retention increases the likelihood of disciplinary incidents and suspensions in the short run, yet these effects dissipate over time. The findings also suggest that these short term adverse effects are concentrated among economically disadvantaged and male students.
Citation: Umut Özek (2013). Hold Back to Move Forward? Early Grade Retention and Student Misbehavior. CALDER Working Paper No. 100
Full Abstract
This paper examines the effects of policies that increase the number of students who take the first course in algebra in 8th grade, rather than waiting until 9th grade. Extending previous research that focused on the Charlotte-Mecklenburg school system, we use data for the 10 largest districts in North Carolina. We identify the effects of accelerating the timetable for taking algebra by using data on multiple cohorts grouped by decile of prior achievement and exploiting the fact that policy-induced shifts in the timing of algebra occur at different times in different districts to different deciles of students. The expanded data make it possible to examine heterogeneity across students in the effect of taking algebra early. We find negative effects among students in the bottom 60% of the prior achievement distribution. In addition, we find other sources of heterogeneity in effect
Citation: Charles Clotfelter, Helen Ladd, Jacob Vigdor (2013). Algebra for 8th Graders: Evidence on its Effects from 10 North Carolina Districts. CALDER Working Paper No. 87
Full Abstract
We use North Carolina data to explore whether the quality of teachers in the lower elementary grades (K-2) falls short of teacher quality in the upper grades (3-5) and to examine the hypothesis that school accountability pressures contribute to such quality shortfalls. Our concern with the early grades arises from recent studies highlighting how children’s experiences in those years have lasting effects on their later outcomes. Using two credentials-based measures of teacher quality, we document within-school quality shortfalls in the lower grades, and show that the shortfalls increased with the introduction of No Child Left Behind. Consistent with that pattern, we find that schools responded to accountability pressures by moving their weaker teachers down to the lower grades and stronger teachers up to the higher grades. These findings support the view that accountability pressure induces schools to pursue actions that work to the disadvantage of children in the lower grades.
Citation: Sarah C. Fuller, Helen Ladd (2013). School Based Accountability and the Distribution of Teacher Quality Across Grades in Elementary Schools. CALDER Working Paper No. 75
Full Abstract
Using longitudinal data on spanning the 2002-03 through 2007-08 school years in Florida and North Carolina, this paper decomposes the workforce dynamics among teachers and principals in low-performing schools that significantly improved their performance. In general, I find strong, consistent evidence of human capital development (i.e., improvements in the productivity of the teachers and principals already in the school) accounting for the increased performance in turnaround schools. These findings are robust to the inclusion of school random effects, alternative categorizations of both teachers and turnaround schools, and are observed across elementary and middle school samples in both states. There is also general evidence of productive incoming teachers helping to improve these turnaround schools, but little evidence to support negative attrition specific to these schools played a role. These findings are important as they document large improvements in the joint productivity of teachers in low-performing schools, a finding which is out of step with current federal efforts to improve schools that implicitly assume teacher productivity is essentially fixed over time.
Citation: Michael Hansen (2013). Investigating the Role of Human Resources in School Turnaround: A Decomposition of Improving Schools in Two States. CALDER Working Paper No. 89
Full Abstract
How to incorporate mobile students, who enter schools/classrooms after the start of the school year, into educational performance evaluations remains to be a challenge. As mandated by the No Child Left Behind Act of 2001 (NCLB), all states currently require that a school is accountable only if the student has been enrolled in the school for a full academic year. This paper investigates the school response to this eligibility requirement in a regression-discontinuity framework. Comparing students who enter schools right before and after the eligibility cutoff, I find no evidence that schools behave strategically in response to this requirement.
Citation: Umut Özek (2012). One Day Too Late? Mobile Students in an Era of Accountability. CALDER Working Paper No. 82
Full Abstract
School closures are increasingly common among U.S. public schools, driven by both budgetary constraints and accountability pressures to turnaround low-performing schools. This paper contributes to the nascent literature on school closures by evaluating student achievement and mobility outcomes in a large-scale restructuring effort in Washington, D.C. in which 32 elementary and middle school campuses were closed or consolidated in the summer of 2008. Using longitudinal data, we investigate how student outcomes change in relation to this initiative with an instrumental variables strategy that counters the endogeneity of student assignment across schools before and after the restructuring occurred. The results show that the academic performance of students directly affected by the school restructuring experienced a temporary decline, but it rebounded by the second school year after restructuring occurred. Additionally, we find no evidence of closure adversely inducing further mobility among affected students.
Citation: Umut Özek, Michael Hansen, Thomas Gonzalez (2012). A Leg Up or a Boot Out?: Student Achievement and Mobility under School Restructuring. CALDER Working Paper No. 78
Full Abstract
Struggling schools that come under increased accountability pressure face a number of challenges, including changing instructional policies and practices to facilitate student improvement. But what effect does school accountability have on teachers’ mobility decisions? This study is the first to exploit policy variation within the same state to examine the effects of school accountability on teacher job changes. Using student-level data from Florida State the authors measure the degree to which schools and teachers were “surprised” by the change in Florida’s school grading system (A+ Plan for Education) in the summer of 2002— what they refer to as an “accountability shock.” They observed the mobility decisions of teachers in the years before and after the school grading change and found over half of all schools in the state experienced an accountability “shock” due to this grading change. Teachers were more likely to leave schools facing increased accountability pressure; even more likely to leave schools shocked downward to a grade of “F”; and less likely to leave schools facing decreased accountability pressure. Schools facing increased accountability pressure also saw a rise in the average quality of the teachers who stayed. If these schools were able to retain more of their high- quality teachers, perhaps through increased incentives to remain in the school, the performance gains associated with school accountability pressure could be greater than those already observed.
Citation: Li Feng, David Figlio, Tim Sass (2010). School Accountability and Teacher Mobility. CALDER Working Paper No. 47
Full Abstract
No Child Left Behind judges the effectiveness of schools based on their students' achievement status. However, many policy analysts argue that schools should be measured, instead, by their students' achievement growth. Using a ten-year student-level panel dataset from North Carolina, we examine how school-specific pressure related to two school accountability approaches (status and growth) affects student achievement at different points in the prior-year achievement distribution. Achievement gains for students below the proficiency cut point emerge in response to both types of accountability systems. We find little or no evidence that schools in North Carolina ignore students far below proficiency under either approach. Importantly, we find that the status, but not the growth, approach reduces the reading achievement of higher performing students, with the losses in the aggregate exceeding the gains at the bottom. The distributional effects of accountability pressure depend on the type of pressure for which schools are held accountable and the tested subject.
Citation: Helen Ladd, Douglas Lauen (2009). Status vs. Growth: The Distributional Effects of School Accountability Policies. CALDER Working Paper No. 21
Full Abstract
This paper examines the effect of accountability policy on school practices and student outcomes with remarkably comprehensive and detailed data that include a multi-wave five-year survey of the census of public schools in Florida and administrative data on individual student performance over time. The authors show that low-performing schools facing accountability pressure changed their instructional practices in meaningful ways. In addition, they present medium-run evidence school accountability promotes improved student test scores, and find that a significant portion of these test score gains can likely be attributed to the changes in school policies and practices uncovered in these surveys.
Citation: Cecilia Rouse, Jane Hannaway, Dan Goldhaber, David Figlio (2007). Feeling the Florida Heat?: How Low-Performing Schools Respond to Voucher and Accountability Pressure. CALDER Working Paper No. 13
Full Abstract
This paper is the first to explore the effects of school accountability systems on high-achieving students' long-term performance. Using data from a large state university, we relate school accountability pressure in high school to a student's university-level grades and study habits. We find that an accountability system based on a low-level test of basic skills apparently led to reduced performance by high-achieving students, while an accountability system based on a more challenging criterion-referenced exam apparently led to improved performance in college on mathematics and other technical subjects. Both types of systems are associated with increased "cramming" by students in college. The results indicate that the nature of an accountability system can influence its effectiveness.
Citation: Colleen Donovan, David Figlio, Mark Rush (2007). Cramming: The Effects of School Accountability on College-Bound Students. CALDER Working Paper No. 7