Ohio State study: Many "failing" schools aren't failing when measured on impact rather than achievement
Up to three-quarters of U.S. schools deemed failing based on achievement test scores would receive passing grades if evaluated using a less biased measure, a new study suggests.
Ohio State University researchers developed a new method of measuring school quality based on schools' actual impact on learning – how much faster students learned during the academic year than during summer vacation when they weren't in class.
Using this impact measure, about three-quarters of the schools now rated as "failing" because of low test scores no longer would be considered substandard.
That means that in these schools mislabeled as failing, students may have low achievement scores, but they are learning at a reasonable rate and they are learning substantially faster during the school year than they are during summer vacation.
"Our impact measure more accurately gauges what is going on in the classroom, which is the way schools really should be evaluated if we're trying to determine their effectiveness," said Douglas Downey, co-author of the study and professor of sociology at Ohio State University.
Downey conducted the study with Paul von Hippel, a research statistician, and Melanie Hughes, a doctoral student, both in sociology at Ohio State.
Their findings appear in the current issue of the journal Sociology of Education.
Currently, most people believe that it is obvious which schools are the best – the ones with the highest achievement scores. But using achievement scores to measure school quality assumes that all schools have students with equivalent backgrounds and opportunities that will give them equal opportunities to succeed in school. And that's obviously not true, von Hippel said.
Students from disadvantaged backgrounds often face a variety of problems at home; for example, their parents often talk and read to them less, and they are less likely to get eyeglasses for nearsightedness. The result is that they are already behind other children before they even begin school.
"The way most states rank schools is extremely distorted," von Hippel said.
"We can't evaluate schools assuming that they all serve similar kinds of children."
The results suggest that states may have to reconsider how they evaluate schools under the federal No Child Left Behind Act, which emphasizes holding schools accountable for student achievement.
The study used data from the Early Childhood Longitudinal Study, a national survey conducted by the U.S. Department of Education. The analysis focused on 4,217 children in 287 schools.
The survey measured children's math and reading scores on four occasions: the beginning and end of their kindergarten year, and the beginning and end of first grade.
By comparing test scores at the end of kindergarten and the beginning of first grade, the researchers could measure learning rates during summer vacation.
Comparing test scores from the beginning and end of first grade allowed the researchers to see how much children learn during the school year.
They then were able to calculate how much faster students learned during the first-grade school year compared to when they were on summer vacation. This was the "impact" score that showed how much schools were actually helping students learn.
"If we evaluate schools that way, things change quite a bit as far as which ones we would identify as failing," Downey said.
If failing schools are defined as those in the bottom 20 percent of achievement scores, about three-quarters of these schools are no longer failing when ranked on the impact measure.
"It suggests that many schools serving disadvantaged kids are doing a good job with children who face a lot of challenges," Downey said. It also means that many teachers in these schools should be lauded for the impact they are having - and not criticized because their students are not passing the achievement tests.
The study also found that about 17 percent of schools that are not failing when rated by achievement test scores turn out to be failing when ranked on impact.
"These schools may be serving children from advantaged backgrounds who do well on achievement tests, but the learning rate for their students isn't dramatically faster when they are in school versus when they are not. In other words, these schools are not having much positive impact," according to Downey.
The bottom line is that, under the current system, "we are not pressuring the schools that need to be pressured," Downey said.
Another way to measure school effectiveness is what has been called the learning approach – simply measuring how much students learn in a year, rather than where they end up on an achievement scale. However, a major limitation to this approach is that the amount learned in a year is still not entirely under schools' control, von Hippel said.
Students spend three months of the year on summer vacation. Even if you look at only the academic year, children spend most of their time in the home environment outside of school.
The advantage of the impact model is that it measures the different rates of learning between summer and the academic year, giving a more accurate picture of the role of schools, according to the researchers.
The profile of failing schools changes substantially when you use the impact measure rather than achievement scores, von Hippel noted.
Based on achievement scores, failing schools tend to be in urban areas, serve a higher percentage of children who qualify for a free lunch, and have a high minority population.
But if you look at impact scores, failing schools are not as concentrated in poor, urban areas with high minority populations.
"When you shift the focus from achievement to impact, there are still schools that do very well and some that do poorly," von Hippel said. "But they are not necessarily where you think they are. There are high-impact schools in every kind of neighborhood, serving every kind of child. The same is true of low-impact schools."
Von Hippel says the results of this study also suggest new ways to elevate achievement in students.
"If there's a school that rates high in educational impact but low on achievement, maybe the school should have a summer program with those same teachers who are having such a positive impact," von Hippel said. "That's certainly more appropriate than saying something must be wrong with this school because of the low achievement scores."
Downey said it is possible to use the impact model to evaluate schools without increasing the number of tests students have to take and schools have to administer.
Right now, schools test students six times – once each year between 3rd grade and 8th grade.
"Rather than use those six tests for low quality information, let's redistribute them differently to get quality information about the schools," he said.
Tests could be given at the end of 3rd grade, the beginning of 4th grade and the end of 4th grade. That way student learning rates could be compared in the summer after third grade with the 4th grade school year. Another set of three tests could be given at the end of 7th grade, and the beginning and end of 8th grade.
Contact: Doug Downey, (740) 427-2303,