Estimating Causal Relationships in NAEP

Duncan Chaplin, Urban Institute
Sheila Isanaka, Urban Institute
Austin Nichols, Urban Institute

Controlling for statistical bias is critically important, yet often difficult. Without proper controls we could easily come to faulty conclusions about the directions and magnitudes of estimated effects. When analyzing data from the National Assessment of Educational Progress (NAEP), researchers have used a variety of methods to control for such bias. These include multivariate regression to control for observable student, school, district, and state characteristics, as well as analyzes using variation over time, across cohorts, and within students. In this paper we compare three additional methods to reduce or eliminate bias when estimating casual impacts--quasi-experimental methods (such as Regression Discontinuity or Instrumental Variables methods), using variation within schools, and using variation within classrooms. These methods are used to estimate the impacts of education technology, teacher qualifications, and peer effects on student performance and other classroom outcomes.