CALDER working papers have not undergone final formal review and should be cited as working papers. They are intended to encourage discussion and suggestions for revision before final publication. Any opinions, findings, and conclusions expressed in these papers are those of the authors and do not necessarily reflect the views of our funders.
Using rich Texas administrative data, we estimate the impact of middle school principals on post-secondary schooling, employment, and criminal justice outcomes. The results highlight the importance of school leadership, though striking differences emerge in the relative importance of different skill dimensions to different outcomes. The estimates reveal large and highly significant effects of principal value-added to cognitive skills on the productive activities of schooling and work but much weaker effects of value-added to noncognitive skills on these outcomes. In contrast, there is little or no evidence that middle school principals affect the probability a male is arrested and has a guilty disposition by raising cognitive skills but strong evidence that they affect these outcomes through their impacts on noncognitive skills, especially those related to the probability of an out-of-school suspension. Heterogeneity analysis reveals that the principal effect on the probability a male is arrested is strongest for males with the highest predicted risk of arrest based on information prior to middle school entry, while principal effects on the probability of attending and persisting in college span the predicted risk distributions outside of the top decile. Finally, the principal effects on the probability of engagement in the criminal justice system are much larger for Black than for non-black males.
Citation: Eric Hanushek, Andrew Morgan, Steven Rivkin, Jeffrey Schiman, Ayman Shakeel, Lauren Sartain (2024). The Lasting Impacts of Middle School Principals. CALDER Working Paper No. 311-1124
We use student-level data on elementary special education identification from Washington state to explore student identification rates in the months immediately after the onset of the COVID-19 pandemic and over 2 subsequent years. Special education identification rates dropped dramatically in March 2020 through the end of the 2019–20 school year and remained below historical norms through 2020–21 before returning to pre-pandemic levels early in 2021–22. The magnitude of these effects implies that over 8,000 fewer elementary students were identified for special education services during the pandemic in Washington than would have been expected based on prior trends.
Citation: Roddy Theobald, Dan Goldhaber, Andrew Katz (2024). Special Education Identification Throughout the COVID-19 Pandemic. CALDER Policy Brief No. 37
Education resources matter when they are allocated and used effectively. Yet, the upstream decisions school boards make about district budgets and resource allocation are understudied. In this descriptive study, we analyze data from 400 publicly available video recordings of financial deliberations in school board budget meetings between spring 2022 and spring 2023. Half of the video recordings are from school boards that received education finance training from the Edunomics Lab at Georgetown University. We find school boards discussed student outcomes in only 15-25% of board meetings focused on financial deliberations. Only about 11% of the variation in financial deliberations can be explained by district characteristics, student achievement, and community characteristics. We find no differences in the discussion of student outcomes for districts with and without the Edunomics training. However, descriptive evidence suggests a positive relationship between the Edunomics training and some summary measures of financial deliberations: the overall level of engagement in budgetary discussions; the likelihood per-unit cost and internal barriers (such as decision-making structure) were mentioned; and the likelihood that the budget was linked to outcomes. These findings underscore the variation in school board deliberations and suggest the potential value of training school board members to influence those deliberations.
Citation: Dan Goldhaber, Zeyu Xu (2024). Descriptive Evidence on the Relationship Between School Board Training and Financial Deliberations. CALDER Working Paper No. 310-1124
The COVID-19 pandemic forced widespread school closures and a shift to remote learning. A growing body of research has examined the effects of remote learning on student outcomes. But the accuracy of the school modality measures used in these studies is questionable. The most common measures—based on self-reports or district website information—are often inconsistent and lack nationwide coverage. Some studies have used cell phone mobility data to identify school modalities, but there is no consensus yet on how to translate device pings into modality measures. This paper contributes to the literature on modality measurement by examining the relationship between mobile device signals and school modality prior to the pandemic and applies those findings to the pandemic period in Michigan and Washington. We compare our results to state-provided closure data and other nationwide sources, including the Return to Learn Tracker and the COVID-19 School Data Hub. Our findings indicate that cell phone mobility data can accurately predict school modality under normal conditions, but the accuracy drops during the pandemic. These results have implications for future research on educational and health outcomes during both pandemic and non-pandemic-related school closures.
Citation: Dan Goldhaber, Nick Huntington-Klein, Nate Brown, Scott Imberman, Katharine O. Strunk (2024). The Call is Coming from Inside the School! How Well Does Cell Phone Data Predict Whether K12 School Buildings Were Open During the Pandemic?. CALDER Working Paper No. 309-1124
Math achievement plays a pivotal role in students’ success in school and beyond. Algebra, in particular, serves as a crucial gateway: it increases the chances a student takes advanced math in high school, which has implications for their college and career prospects. In 2023 and 2024, the American Mathematics Educator Study (AMES) surveyed teachers and principals nationwide to learn more about how students access and experience algebra in 8th and 9th grade. A key finding from the surveys: schools serving mostly low-income students are far less likely to offer algebra in 8th grade compared to those serving more affluent students. The surveys also identified other issues that can hinder equitable access to algebra and math education, including:
These systemic inequities—in both opportunities and experiences—suggest that making algebra more available is a necessary but insufficient step towards closing gaps in access to advanced math education.
This research brief summarizes findings from nationwide surveys of teachers and principals administered in 2023 and 2024 as part of the American Mathematics Educator Study (AMES). The brief is based on a working paper by RAND authors Lauren Covelli, Julia Kaufman, and Umut Özek, which examines how America’s students access and experience algebra in 8th and 9th grade. To go deeper, read the full working paper.
Citation: Benjamin Backes, Lauren Covelli, Michael DeArmond, Elise Dizon-Ross, Dan Goldhaber, Julia Kaufman, Umut Özek (2024). More Than Just Adding Courses: Evidence on Algebra and Equity from the American Mathematics Educator Study. CALDER Policy Brief No. 39
This CALDER research brief builds on pioneering research by the Measures of Effective Teaching Project (Kane et al., 2013) and Jackson (2018) to look at two measures of teacher quality--value-added measures (VAMs) for both test outcomes and non-test outcomes--and what they can tell education leaders about the teacher workforce. Both measures use a value-added framework to separate out a teacher’s contributions to student outcomes from other factors, like a student’s previous academic record or economic circumstances. By trying to isolate teacher contributions to student outcomes, the measures speak to the question, would students be better off with a different teacher?
Citation: Benjamin Backes, James Cowan, Michael DeArmond, Dan Goldhaber, Roddy Theobald (2024). What Can Different Measures Tell Us About the Quality of the Teacher Workforce?. CALDER Policy Brief No. 38
The effect of remote learning on student performance has been a frequent topic of research and discussion in the aftermath of the COVID-19 pandemic, yet little is known about the impact of remote instruction on the performance of teachers. This study documents how relative effectiveness of teachers changed when moving from in-person to remote instruction and analyzes the characteristics of teachers associated with greater relative effectiveness during remote instruction. Using matched student/teacher-level data from three large metro-Atlanta school districts, we estimate teacher value-added models to measure the association between teacher characteristics and a teacher’s relative contribution to test score growth before and during the period of virtual instruction in the 2020-21 school year. We find evidence of increased variation in overall teacher effectiveness during remote instruction. Results are driven by veteran teachers, who appear relatively more effective in virtual instruction than their less-experienced peers, and by the very best in-person teachers, some of which experience large declines in relative effectiveness when shifting to remote instruction.
Citation: M. Cade Lawson, Tim Sass (2024). Teacher Effectiveness in Remote Instruction. CALDER Working Paper No. 308-0924
Four-day school week (4DSW) schedules are growing rapidly across the U.S., with school districts citing perceived benefits to teacher recruitment and retention and student attendance as motivations for adopting the schedule. This study uses panel data from Colorado, one of the states with the highest prevalence of 4DSWs, to investigate the impacts of the 4DSWs on the percentage of teachers with shortage credentials, teacher attrition rates, and student attendance rates. Utilizing a synthetic control difference-in-differences research design, we find 4DSWs have small negative or statistically insignificant effects on teacher recruitment and retention outcomes and find little variation in these effects by school rurality. Examining student attendance outcomes, we estimate a meaningfully small 0.76 percentage point reduction in attendance rates associated with adopting a 4DSW in non-rural schools (equivalent to 46% of these schools’ typical yearly fluctuations in ADA) but do not detect an effect in small rural or non-rural schools. These findings suggest that these purported benefits of 4DSWs are not realized in Colorado, warranting concern about the continued use and expansion of this schedule given prior evidence of its negative average impacts on student achievement.
Citation: Emily Morton, Emma Dewil (2024). Impacts of Four-Day School Weeks on Teacher Recruitment and Retention and Student Attendance: Evidence from Colorado. CALDER Working Paper No. 307-0924
Professional references are widely used in hiring decisions, yet their effectiveness remains largely understudied. We analyze structured ratings collected from the professional references of teacher applicants and conduct an experiment to see whether the ratings influence hiring managers’ assessments of applicants and hiring decisions. We find little evidence that providing reference ratings to hiring managers influences their evaluations of candidates or hiring choices in productive ways. Importantly, we also find that reference ratings are predictive of future job performance. The result is a paradox: reference ratings offer potentially low-cost, high-value information, but hiring managers do not appear to make productive use of them.
Citation: Dan Goldhaber, Cyrus Grout (2024). Improving Hiring Decisions: Experimental Evidence on the Value of Reference Information About Teacher Applicants. CALDER Working Paper No. 306-0824
We used longitudinal staffing data from Pennsylvania to explore differences in special education personnel attrition across personnel categories, individual characteristics, and district characteristics. Special education administrators and school psychologists had the highest attrition rates among special education personnel, with special education administrators 6.4 percentage points more likely to leave their district than observably similar special educators in the same district. Black special education personnel were 2.1 percentage points more likely to leave than observably similar White special education personnel in the same district. Special education personnel in urban districts and districts serving high proportions of students of color also were more likely to leave, all else equal. These trends suggest the need for targeted retention efforts for these important categories of special education personnel.
Citation: Allison Gilmour, Equia Aniagyei-Cobbold, Roddy Theobald (2024). Special Education Personnel Attrition in Pennsylvania. CALDER Working Paper No. 305-0724
This report provides formative data about the implementation of projects associated with the Pennsylvania Bureau of Special Education’s Attract, Prepare, and Retain (APR) efforts during the 2023–24 school year. We surveyed or interviewed students and educators participating in six such projects: Developing Future Special Educators Grants, APR Mentoring Project, Networking and Learning Communities, Learning Institutes, Accelerated Programs for PK–12 Special Education Teacher Certification Grant, and American Sign Language programs. For the first three APR projects, these findings build on data from the first year of project implementation in the 2022– 23 school year. And for the latter three projects that were introduced in 2023–24, these data provide early evidence about how participants view their experiences with these projects.
Citation: Roddy Theobald, Equia Aniagyei-Cobbold, Marcy Stein (2024). The Special Education Teacher Pipeline in Pennsylvania: Year 2 Report. CALDER Working Paper No. 304-0724
The COVID-19 pandemic devastated student achievement, with declines rivaling those after Hurricane Katrina. These losses widened achievement gaps between historically marginalized students and their peers. Three years later, achievement remains behind pre-pandemic levels for many students. This paper examines 2022-23 academic recovery efforts across eight districts, including tutoring, small group instruction, after-school, extended year, double-dose, digital learning, and expert teacher interventions. Across 22 math and reading interventions, most were delivered to fewer students and for less time than planned. We find positive effects for one tutoring program on math scores and two tutoring programs on reading scores, ranging from 0.22 to 0.33 SD. Each of these programs served a very small share of the district’s students and was unlikely to play a major role in district-wide academic recovery. Finally, we find that having an “expert” teacher with high evaluation scores as opposed to a non-expert teacher significantly improves student achievement by 0.06 SD in math and 0.11 SD in reading. While highlighting the promise of intensive academic interventions, our findings underscore the challenges districts face in scaling such interventions to match their recovery needs. The field needs better evidence regarding successful implementation of large-scale interventions.
Citation: Maria V. Carbonari, Michael DeArmond, Daniel Dewey, Elise Dizon-Ross, Dan Goldhaber, Thomas J. Kane, Anna McDonald, Andrew McEachin, Emily Morton, Atsuko Muroga, Alejandra Salazar, Douglas O. Staiger (2024). Impacts of Academic Recovery Interventions on Student Achievement in 2022-23. CALDER Working Paper No. 303-0724
Teacher turnover is a perennial concern, especially in low-performing, high-poverty schools. While districts and schools may try to anticipate and mitigate turnover by surveying teachers about their future plans, existing research on whether teacher-reported intent is predictive of actual turnover behavior is mixed. Using unique survey data from teachers in 35 low-performing, high-poverty districts in Michigan linked at the teacher level to statewide administrative data, we are able to measure turnover behavior one, two, and three years following reported intent. We find that intent is a significant predictor of turnover and becomes increasingly predictive over time. We also find organizational commitment and school organizational conditions are important factors in teachers’ intent and, to a lesser degree, actual turnover behavior.
Citation: Erica Harbatkin, Tuan Nguyen, Katharine O. Strunk, Jason Burns, Alex Moran (2024). Should I Stay or Should I Go (Later)? Teacher Intentions and Turnover in Low-Performing Schools and Districts Before and During the COVID-19 Pandemic. CALDER Working Paper No. 302-0724
We estimate the effects of federal pandemic-relief funding (ESSER III) for K12 schools on district-level student achievement growth in 2023. We rely on student test achievement data from over 5,000 school districts across 30 states. Our novel identification strategy exploits variation in ESSER attributable to its allocation rules and their relationship to Title I. We find that each $1,000 increase in ESSER per pupil funds led to statistically significant increases in district math scores of 0.008 standard deviations and similar but statistically insignificant increases in ELA scores. Our heterogeneity analysis suggests impacts were not even across district pre-pandemic spending levels, student race, or urbanicity. Our estimates provide some insight into how much investment may be needed for a full academic recovery from the pandemic: to recover losses remaining after 2023, we estimate schools would need to spend $9,000 to $13,000 per pupil.
Citation: Dan Goldhaber, Grace Falken (2024). ESSER and Student Achievement: Assessing the Impacts of the Largest One-Time Federal Investment in K12 Schools. CALDER Working Paper No. 301-0624
In this paper we examine academic recovery in 12 mid- to large-sized school districts across 10 states during the 2021–22 school year. Our findings highlight the challenges that recovery efforts faced during the 2021–22 school year. Although, on average, math and reading test score gains during the school year reached the pace of pre-pandemic school years, they were not accelerated beyond that pace. This is not surprising given that we found that districts struggled to implement recovery programs at the scale they had planned. In the districts where we had detailed data on student participation in academic interventions, we found that recovery efforts often fell short of original expectations for program scale, intensity of treatment, and impact. Interviews with a subsample of district leaders revealed several implementation challenges, including difficulty engaging targeted students consistently across schools, issues with staffing and limitations to staff capacity, challenges with scheduling, and limited engagement of parents as partners in recovery initiatives. Our findings on the pace and trajectory of recovery and the challenges of implementing recovery initiatives raise important questions about the scale of district recovery efforts.
This is an updated version of CALDER Working Paper 275-1222 titled, "The Challenges of Implementing Academic COVID Recovery Interventions: Evidence from the Road to Recovery Project" originally released in December 2022.
Citation: Maria V. Carbonari, Miles Davison, Michael DeArmond, Daniel Dewey, Elise Dizon-Ross, Dan Goldhaber, Ayesha Hashim, Thomas J. Kane, Andrew McEachin, Emily Morton, Atsuko Muroga, Tyler Patterson, Douglas O. Staiger (2024). The Impact and Implementation of Academic Interventions During COVID: Evidence from the Road to Recovery Project. CALDER Working Paper No. 275-0624-2
Like most other states, Missouri uses assessments intended to measure whether students are on a pathway to “college and career readiness.” The state longitudinal data system now has the capacity to directly test that claim. We make use of 8th-grade assessment (MAP) scores in Math, Science, and Communication Arts for roughly 260,000 first-time Missouri freshmen who began high school between Fall, 2009 and Fall, 2012. These students were tracked through high school and for five years following on-time high school graduation. We find a strong positive association between MAP performance scores in 8th grade Math, Science, and Communication Arts and post-secondary college attendance and degree completion. This is true overall and for White, Black, and Hispanic students disaggregated by gender. Proficiency on all three exams matters even more. Based on a logistic forecasting model, if all students who scored below Proficient on the 8th-grade MAP raised their scores to Proficient, the number earning post-secondary degrees would increase by roughly 50 percent. Black and Hispanic students earning post-secondary degrees would increase by roughly 150 and 75 percent, respectively. We conclude that 8th-grade MAP proficiency scores are highly informative about whether students are on a pathway to college and career readiness.
Citation: Darrin DeChane, Takako Nomi, Michael Podgursky (2024). College and Career Ready: How Well Does 8th Grade MAP Performance Predict Post-Secondary Educational Attainment?. CALDER Working Paper No. 300-0524
Given math’s importance, states are adopting a range of K12 policies to improve math outcomes. As these and other policies unfold, policymakers need to monitor intended outcomes and ultimately gauge impact. They need answers to basic questions: Are more students taking and passing algebra and advanced math? Are performance gaps changing be- tween student groups or schools? What are the characteristics of our math teachers? Are they changing? How are math teachers with different characteristics distributed across schools and students? Policymakers also need answers to more complex questions: Did a given policy have an effect? If so, why, under what conditions, and for whom? Were there any unintended consequences? With these kinds of questions in mind, this CALDER brief singles out the value of longitudinal, individual-level state administrative data by reviewing questions that can—and can’t—be answered by the more limited data made publicly available by states. To illustrate, we focus on three states that are attempting to improve math outcomes and that enroll large numbers of students from historically marginalized student groups: California, Florida, and Texas.
Citation: Benjamin Backes, Michael DeArmond, Elise Dizon-Ross, Dan Goldhaber (2024). What We Can (and Can’t) Learn from Publicly Available Data About K12 Math Outcomes. CALDER Policy Brief No. 36
Most states responded to the onset of the pandemic by temporarily granting teachers Emergency licenses. These licenses allowed teachers to work in classrooms without passing the typical licensure exams. Since then, several states have extended their use of Emergency licenses, raising questions about how these policies impact the composition of the teacher workforce and student outcomes. In this paper, we examine the result of these policies using data on multiple cohorts of Emergency licensed teachers (ELTs) who taught in Massachusetts between 2021 and 2023. We find that ELTs were slightly more likely to remain in the same school and in the teaching workforce than teachers from other entry routes. However, ELTs’ students scored significantly lower on standardized tests in math and science than other students in the same school and same year. Our findings are at odds with earlier, more positive assessments of Emergency licensure in Massachusetts. Our updated results appear to be driven by more recent cohorts of ELTs, rather than the teachers who received Emergency licenses at the start of the pandemic. Overall, this study suggests policymakers should be cautious when drawing sweeping conclusions about the impacts of teacher licensure based solely on the earliest cohort of teachers who obtained pandemic-era Emergency licenses.
Citation: Benjamin Backes, James Cowan, Dan Goldhaber, Roddy Theobald (2024). Four Years of Pandemic-Era Emergency Licenses: Retention and Effectiveness of Emergency-Licensed Massachusetts Teachers Over Time. CALDER Working Paper No. 299-0424
Departmentalized instruction, in which teachers specialize in one or more core subjects and instruct multiple groups of students in a day, has become increasingly prominent in elementary schools. Using 8 years of data from Massachusetts and a difference-in-differences design, we estimate the effects of departmentalization on student achievement. We find that departmentalization has positive effects in English language arts (ELA) and science and mixed evidence of positive effects in math. These positive effects are not driven by teacher productivity improvements: Consistent with prior findings on teacher specialization, teachers are less effective when specializing in math and no more effective in ELA than when teaching self-contained classrooms. Rather, consistent with the theoretical underpinnings for specialization, departmentalized schools tend to assign teachers to their stronger subjects.
Citation: Benjamin Backes, James Cowan, Dan Goldhaber (2024). Departmentalized Instruction and Elementary School Effectiveness. CALDER Working Paper No. 298-0424
The Elementary and Secondary School Emergency Relief Fund (ESSER) was the largest one-time federal investment in K–12 schools in history, funneling almost $200 billion to states and school districts. We use novel data from Washington State to investigate the extent to which ESSER funding causally influenced spending on school personnel. We argue one cannot infer this directly from ESSER claims data because of the fungibility of school budgets. Thus, we rely on a more direct signal of district hiring decisions: public education job postings scraped from district hiring websites. To address endogeneity concerns, our preferred approach employs an instrumental variables strategy that exploits a formula mechanism used to determine Title I funding for 2020–21 (and thus ESSER allocations in 2022) based on the number of Title I formula-eligible children. We find strong, arguably causal, evidence that public school hiring increased in response to the availability of ESSER funding. Specifically, we estimate that each $1,000 in ESSER allocations caused districts to seek to hire $206 in additional staff, disproportionately teachers. These estimates suggest that roughly 12,000 new staff (including 5,100 teachers) were hired in Washington because of ESSER. In the absence of new funding, school staffing budgets will likely need to contract substantially following the sunset of ESSER.
Citation: Dan Goldhaber, Grace Falken, Roddy Theobald (2024). ESSER Funding and School System Jobs: Evidence from Job Posting Data. CALDER Working Paper No. 297-0424
