CALDER working papers have not undergone final formal review and should be cited as working papers. They are intended to encourage discussion and suggestions for revision before final publication. Any opinions, findings, and conclusions expressed in these papers are those of the authors and do not necessarily reflect the views of our funders.
Student-teacher relationships are at the core of student experiences in schools and, arguably, fundamental to influencing student outcomes. Using a statewide, student-level school climate survey from Massachusetts, we investigate teachers’ contributions to school climate, which we refer to as climate value added (VA), and how it varies by student race/ethnicity. We first show that climate VA contributes to student learning: Teachers whose students report positive feelings about climate also contribute more to student test scores and to an aggregate of nontest student outcomes (student absences, suspensions, and grade progression). And teachers identified by students of color as contributing to better school climate have outsize effects on learning gains for these students. Differences in teachers’ climate effects across racial/ethnic groups are largest on topics aligned with cultural competency, school participation, and comfort with faculty. Lastly, we find that Black students assigned to Black teachers report better school climate than Black students assigned to other teachers.
Citation: Benjamin Backes, James Cowan, Dan Goldhaber, Roddy Theobald (2022). Teachers and School Climate: Effects on Student Outcomes and Academic Disparities. CALDER Working Paper No. 274-1022
We describe the extent and predictors of staffing challenges faced by school districts in Washington state throughout the 2021–22 school year using data collected from job posting websites for districts representing more than 98% of students in the state. These data suggest that school districts in the state faced considerable challenges filling paraeducator and, to a lesser extent, teaching positions at the beginning of the school year. When we focus specifically on teachers, we find that teacher staffing challenges were far more pronounced for special education positions and in schools serving more students of color, less pronounced for elementary positions, and highly correlated with teacher attrition from these types of positions after the prior school year. Accounting for these relationships, districts posted more teaching positions later in the school year when they had increasing student enrollments and received more Elementary and Secondary School Emergency Relief funds than nearby districts.
Citation: Dan Goldhaber, Nate Brown, Nathaniel Marcuson, Roddy Theobald (2022). School District Job Postings and Staffing Challenges Throughout the Second School Year During the COVID-19 Pandemic. CALDER Working Paper No. 273-1022
Most research about how to improve the teacher workforce has focused on interventions designed to improve incumbent teachers, far less attention has been directed toward teacher hiring processes and whether districts can make better hiring decisions. Using data from Spokane Public Schools and Washington state, we describe the findings from a study analyzing measures of the predictive validity of teacher applicant quality measures obtained from professional references. We find that professional reference ratings of prospective teachers are significantly predictive of teacher quality as measured by inservice performance evaluations and teacher value added in math. These findings are driven by applicants with at least some teaching experience and vary by rater type (e.g., principal or university supervisor); the magnitude of the relationship between the ratings of applicants and teacher performance is much smaller and not statistically significant for applicants that do not have teaching experience. Overall, the evidence suggests that obtaining explicit ratings of teacher applicants from professional references is a low-cost way to contribute to the applicant information available to hiring officials and has potential for improving hiring outcomes.
Citation: Dan Goldhaber, Cyrus Grout, Malcolm Wolff (2022). How Well Do Professional Reference Ratings Predict Teacher Performance?. CALDER Working Paper No. 272-1022
We examine how different measures of teacher quality are related to students’ long-run trajectories. Comparing teachers’ test-based value-added to nontest value-added – based on contributions to student absences and grades – we find that test and nontest value-added have similar effects on the average quality of colleges that students attend. However, test-based teacher quality measures have more explanatory power for outcomes relevant for students at the top of the achievement distribution such as attending a more selective college, while nontest measures have more explanatory power for whether students graduate from high school and enroll in college at all.
Citation: Benjamin Backes, James Cowan, Dan Goldhaber, Roddy Theobald (2022). Teachers and Students’ Postsecondary Outcomes: Testing the Predictive Power of Test and Nontest Teacher Quality Measures. CALDER Working Paper No. 270-1022
Taking advanced courses in high school predicts a broad array of positive postsecondary and labor market outcomes. Yet students from historically disadvantaged groups and low-income backgrounds have long been underrepresented in these courses. To address this problem, more than 60 districts in Washington state implemented a policy that automatically enrolled all qualified high school students in advanced coursework. The policy relied on a simple behavioral nudge: It made advanced courses “opt out” rather than “opt in” for all qualified students. The districts implemented the policy in waves, beginning in the 2014–15 school year. In this descriptive paper, we examine enrollment patterns by comparing districts that adopted the policy at different times. We found that students in districts that implemented the policy between 2014–15 and 2016–17 were more likely to enroll in at least one advanced course in any subject relative to students in districts without the policy. This was the case for students who “qualified” for advanced courses based on their test scores and for students whose scores did not qualify them for advanced courses. We also found that the policy was associated with a higher probability of enrollment in advanced mathematics courses but only for qualified students. Qualified students across demographic groups experienced similar changes in the probability of advanced course enrollments. But among all students—regardless of qualified status—enrollments in advanced mathematics and advanced English language arts/social studies courses increased more for students from racial/ethnic groups underrepresented in advanced courses and for students who were eligible for free or reduced-price lunch (FRPL) than for non-underrepresented students and students not eligible for FRPL. These across the board increases in advanced course enrollment for students who were historically underrepresented in these courses suggests that districts may have looked beyond standardized assessment scores to identify students for automatic enrollment.
Citation: Megan Austin, Benjamin Backes, Dan Goldhaber, Dory Li, Francie Streich (2022). Leveling Up: A Behavioral Nudge to Increase Enrollment in Advanced Coursework. CALDER Working Paper No. 271-1022
Measures of student disadvantage—or risk—are critical components of equity-focused
education policies. However, the risk measures used in contemporary policies have significant limitations, and despite continued advances in data infrastructure and analytic capacity, there has been little innovation in these measures for decades. We develop a new measure of student risk for use in education policies, which we call Predicted Academic Performance (PAP). PAP is a flexible, data-rich indicator that identifies students at risk of poor academic outcomes. It blends concepts from emerging “early warning” systems with principles of incentive design to balance the competing priorities of accurate risk measurement and suitability for policy use. PAP is more effective than common alternatives at identifying students who are at risk of poor academic outcomes and can be used to target resources toward these students—and students who belong
to several other associated risk categories—more efficiently.
Citation: Ishtiaque Fazlul, Cory Koedel, Eric Parsons (2022). Using Predicted Academic Performance to Identify At-Risk Students in Public Schools. CALDER Working Paper No. 261-0922
In this descriptive study, we use longitudinal student-level administrative records from 4 cohorts of high school graduates in Kentucky to examine the extent to which students persist and attain post-secondary credentials in the CTE fields of concentration they choose in high school. To our knowledge, this is the first paper to use student-level administrative data to examine how different fields of concentration in high school CTE are related to future postsecondary outcomes. We find that concentrating in a particular CTE field in high school is associated with both continuing on with that same field in college and obtaining a postsecondary credential in that field; this relationship is especially strong in health fields and especially for women in health. The secondary-postsecondary connection is the weakest among students concentrating in occupational fields in high school, who are also the most disadvantaged socioeconomically and academically before high school. Despite the existence of secondary-postsecondary pipelines of career interests, most students enroll and obtain credentials in fields that are different from the field of concentration in high school. In addition, relative to students with similar pre-high-school achievement as measured by grades and test scores, we find that CTE concentration in high school is strongly associated with being more likely to enroll in a two-year college and less likely to enroll in a four-year college.
Citation: Zeyu Xu, Benjamin Backes (2022). Linkage Between Fields of Concentration in High School Career-Technical Education and College Majors. CALDER Working Paper No. 269-0722
The success of many students with disabilities (SWDs) depends on access to high-quality general education teachers. Yet, most measures of teacher value-added measures (VAM) fail to distinguish between a teacher’s effectiveness in educating students with and without disabilities. We create two VAM measures: one focusing on teachers’ effectiveness in improving outcomes for SWDs, and one for non-SWDs. We find top-performing teachers for non-SWDs often have relatively lower VAMs for SWDs, and that SWDs sort to teachers with lower scores in both VAMs. Overall, SWD-specific VAMs may be more suitable for identifying which teachers have a history of effectiveness with SWDs and could play a role in ensuring that students are being optimally assigned to these teachers.
Citation: W. Jesse Wood, Ijun Lai, Neil R. Filosa, Scott A. Imberman, Nathan Jones, Katharine O. Strunk (2022). Are Effective Teachers for Students with Disabilities Effective Teachers for All?. CALDER Working Paper No. 268-0622
In this paper, we use NWEA MAP test data to examine variation in students’ achievement and growth during the pandemic across multiple dimensions. Consistent with prior evidence, we find that students’ test scores in fall 2021, on average, were substantially below historic averages. Moreover, the average scores of students of color, students attending high poverty schools, and students in elementary school were more negatively impacted, and more so in math than reading. We present novel evidence on the distributions of test scores and growth in fall 2021 relative to pre-pandemic distributions, finding disproportionately larger declines for students with lower previous achievement levels across districts. However, between districts, there was considerable variation in the extent to which their fall 2021 achievement and growth distributions shifted from their historical distributions by subject, student subgroups, and baseline achievement levels. Therefore, accurately targeting students and choosing interventions for pandemic-related recovery will require careful assessment by districts of their students’ achievement and growth in the 2021-22 school year (and into the future): assuming that students in a district reflect the national trends of achievement will often lead to incorrect conclusions about the degree to which they suffered pandemic-related learning losses and the amount of support they will need to recover.
Citation: Dan Goldhaber, Thomas J. Kane, Andrew McEachin, Emily Morton (2022). A Comprehensive Picture of Achievement Across the COVID-19 Pandemic Years: Examining Variation in Test Levels and Growth Across Districts, Schools, Grades, and Students. CALDER Working Paper No. 266-0522
Using testing data from 2.1 million students in 10,000 schools in 49 states (plus D.C.), we investigate the role of remote and hybrid instruction in widening gaps in achievement by race and school poverty. We find that remote instruction was a primary driver of widening achievement gaps. Math gaps did not widen in areas that remained in-person (although there was some widening in reading gaps in those areas). We estimate that high-poverty districts that went remote in 2020-21 will need to spend nearly all of their federal aid on academic recovery to help students recover from pandemic-related achievement losses.
Citation: Dan Goldhaber, Thomas J. Kane, Andrew McEachin, Emily Morton, Tyler Patterson, Douglas O. Staiger (2022). The Consequences of Remote and Hybrid Instruction During the Pandemic. CALDER Working Paper No. 267-0522
We estimate the education and earnings returns to enrolling in technical two-year degree programs at community colleges in Missouri. A unique feature of the Missouri context is the presence of a highly-regarded, nationally-ranked technical college: State Technical College of Missouri (State Tech). Compared to enrolling in a non-technical community college program, we find that enrolling in a technical program at State Tech greatly increases students’ likelihoods of graduation and earnings. In contrast, there is no evidence that technical education programs at other Missouri community colleges increase graduation rates, and our estimates of the earnings impacts of these other programs are much smaller than for State Tech. Our findings exemplify the importance of institutional differences in driving the efficacy of technical education and suggest great potential for high-quality programs to improve student
outcomes.
Citation: Maxwell J. Cook, Cory Koedel, Michael Reda (2022). The Education and Earnings Returns to Postsecondary Technical Education: Evidence from Missouri. CALDER Working Paper No. 265-0422
We present new estimates of the importance of teachers in early grades for later grade outcomes, but unlike the existing literature that examines teacher “fade-out,” we directly compare the contribution of early-grade teachers to later year outcomes against the contributions of later year teachers to the same later year outcomes. Where the prior literature finds that much of the contribution of early teachers fades away, we find that the contributions of early-year teachers remain important in later grades. The difference in contributions to eighth-grade outcomes between an effective and ineffective fourth-grade teacher is about half the difference among eighth-grade teachers. The effect on eighth-grade outcomes of replacing a fourth-grade teacher who is below the 5th percentile with a median teacher is about half the underrepresented minority (URM)/non-URM achievement gap. Our results reinforce earlier conclusions in the literature that teachers in all grades are important for student achievement.
Citation: Dan Goldhaber, Zeyu Jin, Richard Startz (2022). How Much Do Early Teachers Matter?. CALDER Working Paper No. 264-0422
Prior work on teacher candidates in Washington State has shown that about two thirds of individuals who trained to become teachers between 2005 and 2015 and received a teaching credential did not enter the state’s public teaching workforce immediately after graduation, while about one third never entered a public teaching job in the state at all. In this analysis, we link data on these teacher candidates to unemployment insurance data in the state to provide a descriptive portrait of the future earnings and wages of these individuals inside and outside of public schools. Candidates who initially became public school teachers earned considerably more, on average, than candidates who were initially employed either in other education positions or in other sectors of the state’s workforce. These differences persisted at least 10 years into the average career and across transitions into and out of teaching. There is therefore little evidence that teacher candidates who did not become teachers were lured into other professions by higher compensation. Instead, the patterns are consistent with demand-side constraints on teacher hiring during this time period that resulted in individuals who wanted to become teachers taking positions that offered lower wages but could lead to future teaching positions.
Citation: Dan Goldhaber, John Krieg, Stephanie Liddle, Roddy Theobald (2022). Out of the Gate, but Not Necessarily Teaching: A Descriptive Portrait of Early-Career Earnings for Those Who Are Credentialed to Teach. CALDER Working Paper No. 263-0422
Over the last two decades, twenty-two states have moved away from traditional defined benefit (DB) pension systems and toward pension plan structures like the defined contribution (DC) plans now prevalent in the private sector. Others are considering such a reform as it is seen as a means of limiting future pension funding risk. It is important to understand the implications of such reforms for end-of-career exit patterns and workforce composition. Empirical evidence on the relationship between pension plan structure and retirement timing is currently limited, primarily because most state pension reforms are so new that few employees enrolled in those alternative plans have reached retirement age. An exception, and the subject of our analysis, is the teacher retirement system in Washington State, which introduced a hybrid DB-DC plan in 1996 and allowed employees in its traditional DB plan to transfer into the new plan. Our analysis focuses on a years-of-service threshold, the crossing of which grants employees early retirement eligibility and, in many cases, a large upward shift in retirement wealth. The financial implications of crossing this threshold are far greater under the state’s traditional DB plan than under the hybrid plan. We find that employees are responsive to crossing the years-of-service threshold, but we fail to find significant evidence that the propensity to exit the workforce varies according to plan enrollment.
Citation: Dan Goldhaber, Cyrus Grout, Kristian Holden, Josh McGee (2022). Should I Stay or Should I Go Now? An Analysis of Pension Structure and Retirement Timing. CALDER Working Paper No. 262-0322
Two recent CALDER studies published in Exceptional Children provide new evidence about special education teacher preparation and its implications for students with disabilities. The first study (Theobald et al., 2021) shows that special educators who received dual endorsements in special education and another subject had lower rates of workforce entry and retention in special education classrooms. The second study (Theobald et al., 2022) demonstrates that students with disabilities experienced greater reading gains when their district and their special education teacher’s preparation program both used/emphasized evidence-based literacy practices. Together, these papers suggest caution around state-level policies that seek to use dual licensure to address special education teacher shortages, but also suggest potential promise around better aligning special educator literacy preparation and practice as a policy lever for improving reading outcomes for students with disabilities. Future research could study specific policy interventions to design dual-license programs, address special educator shortages, and better align special educator preparation and practice.
Citation: Roddy Theobald (2022). New Evidence on Special Education Teacher Preparation. CALDER Policy Brief No. 31
In this flash brief, we frame the magnitude of teacher attrition during the pandemic, including from the 2020–2021 school year to the 2021–22 school year, using publicly available data on the public teaching workforce in Washington. Specifically, we compare attrition rates during the pandemic to attrition during pre-pandemic years, spanning the 1984–1985 school year to the 2018–2019 school year. We also report the relationship between attrition rates and district vacancies and compare changes in teacher turnover rates to differences in these rates between different kinds of schools.
Citation: Dan Goldhaber, Roddy Theobald (2022). Teacher Attrition and Mobility in the Pandemic. CALDER Policy Brief No. 30
The COVID-19 pandemic initially resulted in an unanticipated and near-universal shift from in-person to virtual instruction in spring 2020. During the 2020-21 school year, schools began to re-open and families were faced with decisions regarding the instructional mode for their children. We leverage administrative, survey, and virtual-learning data to examine the determinants of family learning-mode choice and the effects of virtual education on student engagement and academic achievement. Family preference for virtual (versus face-to-face) instruction was most highly associated with school-level infection rates and appeared relatively uniform within schools. We find that students who were assigned a higher proportion of instructional days in virtual mode experienced higher rates of attendance, but also negative student achievement growth compared to students who were assigned a higher proportion of instructional days in face-to-face mode. Students belonging to marginalized groups experienced more positive associations with attendance but were also more likely to experience lower student achievement growth when assigned a greater proportion of instructional days in virtual mode. Insights from this study can be used to better understand family preference as well as to target and refine virtual learning in a post-COVID-19 society.
Citation: Jennifer Darling-Aduana, Henry T. Woodyard, Tim Sass, Sarah S. Barry (2022). Learning-Mode Choice, Student Engagement, and Achievement Growth During the COVID-19 Pandemic. CALDER Working Paper No. 260-0122
CALDER Policy Brief No. 29-0122
In this CALDER Flash brief, we describe what we have learned about the staffing challenges faced by various kinds of school districts endeavoring to hire different school personnel in Fall of 2021.
Citation: Dan Goldhaber, Trevor Gratz (2022). School District Staffing Challenges in a Rapidly Recovering Economy. CALDER Policy Brief No. 29
We use publicly available, longitudinal data from Washington state to study the extent to which three interrelated processes—teacher attrition from the state teaching workforce, teacher mobility between teaching positions, and teacher hiring for open positions—contribute to “teacher quality gaps” (TQGs) between students of color and other students in K–12 public schools. Specifically, we develop and implement an agent-based model simulation of decisions about attrition, mobility, and hiring to assess the extent to which each process contributes to observed TQGs. We find that eliminating inequities in teacher mobility and hiring across different schools would close TQGs within 5 years, while just eliminating inequities in teacher hiring would close gaps within 10 years. On the other hand, eliminating inequities in teacher attrition without addressing mobility and hiring does little to close gaps.
Citation: Dan Goldhaber, Matt Kasman, Vanessa Quince, Roddy Theobald, Malcolm Wolff (2022). How Did It Get This Way? Disentangling the Sources of Teacher Quality Gaps Through Agent-Based Modeling. CALDER Working Paper No. 259-0122
CALDER Policy Brief No. 28-1121
Citation: Dan Goldhaber, Etai Mizrav (2021). The Prospective Teacher Pipeline: Simulation Evidence on Levers to Influence Teacher Diversity. CALDER Policy Brief No. 28
