Randomly Accountable: Failing to Account for Natural Fluctuations in Test Scores Could Undermine the Very Idea of Holding Schools Accountable for Their Efforts-Or Lack Thereof. (Research)

Abstract

THE ACCOUNTABILITY DEBATE TENDS TO DEVOLVE INTO A BATTLE between the pro-testing and anti-testing crowds. But when it comes to the design of a school accountability system, the devil is truly in the details. A well-designed accountability plan may go a long way toward giving school personnel the kinds of signals they need to improve performance. However, a poorly designed scheme, which ignores the statistical properties of schools' average test scores, may do more harm than good. The recent debate over the reauthorization of the federal Elementary and Secondary Education Act (ESEA) is a case in point. From his first days in office, President Bush promised to make education reform a centerpiece of his administration, using the reauthorization of the ESEA as an opportunity to give the state-led accountability movement a dramatic shove forward. Within six months of his raking office, both houses of Congress had passed bills that imposed new federal standards for the states' accountability efforts. However, both bills were seriously flawed. They created standards that, over time, would have identified nearly every school in the nation aslow performing, forcing them to spend precious resources developing unnecessary school-improvement plans. A tide of paperwork would have crowded out time for learning. This almost turned the most significant federal foray into education policy in decades into an embarrassment. Changes were made by a House-Senate conference committee, so the law, as enacted, remedied the most glaring problems, but created others. The saga illustrates the difficulties of designing an effective accountability system. The House and Senate Bills At the heart of both bills was a detailed formula for determining when a school is making adequate yearly progress. The consequences for schools that failed to meet their performance targets were progressively severe-after one year, districts would be required to offer public school choice to all the students in a school; after several years, districts would be required to replace school staff, convert the school into a public charter school, or hand the school over to a private contractor. The problem is that such consequences place too much weight on single-year changes in test scores at the school level, Either bill would have required an increase in the proportion of students scoring above the proficient level in both math and reading, each and every year. However, test scores at the school level often fluctuate for reasons other than any underlying change in a school's performance. Such volatility arises from two sources, The first is variation due to differences in the groups of students being tested each year. Even if the students are being drawn from the same families and the same neighborhoods, the average performance of a school can fluctuate from year to year depending on the attitudes and abilities of the students in each cohort, The average elementary school contains only 68 students per grade level. With a sample this small, having five particularly bright students (or a few students with undiagnosed learning disabilities) in any one year can lead to large fluctuations in a school 's test scores from one year to the next. The Department of Labor measures the monthly unemployment rate with a sample of nearly 60,000 households, Congress was proposing that the Department of Education measure the performance of the typical elementary school grade with a sample neatly 1/1000 the size. The second source of variation is one-time factors that lead to temporary fluctuations in test performance. Some of these factors are likely to be unrelated to the educational practices of a school. For instance, a dog barking on the day of the test, a severe flu season, or one particularly disruptive student in class could cause scores to fluctuate, There may be other sources of volatility that are more related to the educational mission of a school, such as the favorable chemistry between a teacher and a particular group of students or teacher turnover, Whatever the source of variation, single-year changes in test performance are very unreliable indicators of where a school is headed over the long term. …

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 20 citations exist upstream.