League tables and school performance

The call to publish test scores for school students has been more shrill recently. Prime Minister Kevin Rudd and Education Minister Julia Gillard have linked performance statistics to their “education revolution”. Accountability and transparency are common themes in Minister Gillard’s speeches: parents should have access to the information departments of education already get so they can make better choices about where to send their children.

Knowing what differences there are between schools, taking account of levels of resourcing and socio-economic and related factors of the “catchment” from which the children are drawn, will assist in implementation of strategies to improve performance in the Government’s scheme.

At least we can note that what is advocated here involves comparing like with like. Unfortunately Prime Minister Rudd did talk at the National Press Club in August last year of moving out principals and teachers who failed to deliver improved performance after they had been given extra help. Predictably, teachers’ groups objected strongly.

Australian National University economist Andrew Leigh, in “Schools need a Report Card too” (first published in the Australian Financial Review December 20, 2007 and reprinted in On Line Opinion February 4, 2008) reminded us:

Breaking the ideological deadlock requires attention to the new productivity agenda in Australia: making public services work better. … Despite a significant increase in funding [of schools], literacy and numeracy scores of Australian teenagers have failed to rise over recent decades. On average, new teachers are less academically talented today than they were two decades ago.

Boosting the performance of Australian schools is far from straightforward, but one sensible reform would be to begin reporting on the performance of individual schools, so that parents can better choose between their local schools. Such a reform would bring us into line with Britain and the United States, where policymakers across the board take the view that a school’s test scores are quintessentially public information.

The assertions about the qualifications of teachers can be seriously challenged and those about the performance of Australian students are far too general to be of any value whatsoever: performance has risen in some tests and not others. The proposition that Australian practice should be brought into line with Britain and the United States, countries whose students almost always perform to a lower standard than Australian students, is completely unhelpful (to put it mildly) and ignores evidence from well conducted and verified countrywide and international studies.

I would have expected Leigh to be aware of the statement made by celebrated (and reviled) economist F.A. von Hayek in his Nobel Prize Lecture in 1974. He observed, “the aspects of the events to be accounted for about which we can get quantitative data are necessarily limited and may not include the important ones”. In other words simply choosing factors which are susceptible to quantitative analysis does not mean that those factors are important!

The Australian also took up the trumpet on December 2, 2008 when it praised Kevin Rudd's commitment to report on performance:

More fundamental is the right of parents in a democracy to know how their children are going and how their child's school - and other schools they are considering for them in future - are performing … Secrecy about education performance is a prime example of how taxpayers are denied information about how well the services they pay for are delivered.

And:

Anything less than [full and transparent reporting of individual school outcomes] by way of caving into union pressures, will compromise the process.

One respondent (“Chris C”) to Leigh’s article observed:

Recently, the ABS [Australian Bureau of Statistics] released its Adult Literacy and Life Skills Survey. Contrary to the widespread claims of dumbing down over the past few decades, the survey showed that, with the exception of teenagers, older people have lower levels of literacy than younger people. The patterns for numeracy and problem solving were generally similar.

These comments came after Rupert Murdoch told us in his fourth Boyer Lecture, “The unvarnished truth is that in countries such as Australia, Britain, and particularly the United States, our public education systems are a disgrace” and “Australia needs in its schools a culture of performance and accountability”.

In November also New York City Schools Commissioner Joel Klein, brought to Australia by Minister Gillard, asserted that the culture of the education system in New York City was typified by a culture of excuse which had to be transformed to a culture of performance. Unfortunately Klein’s claims for success are very hotly disputed, particularly by Daniel Koretz, Professor of Education at Harvard University, who has described the methodology behind the New York City reporting system as “baroque” (in “More Questions Raised About the New York School Reporting Model” September 26, 2008).

I do wonder what the point is of all this ongoing criticism, as if that is the way to achieve desirable change. At what point do we recognise outstanding achievements by young people in so many fields? Where and when do we acknowledge dedication of teachers? (They are not people rewarded with huge salaries for leading the enterprise into the garbage can of history!)

The many problems with publication of test scores as “league tables” are not acknowledged by advocates. How valid are the data? How will they be used? And, especially, do they actually tell us anything useful about schools and teaching?

Unfortunately, many economists and politicians who venture into this area seem to regard research in education as inferior. Is this because its results do not agree with their preconceived views? Small bits of “experimental” economics and generalisations about business practice are quoted as authoritative. Getting hold of any data is considered worthwhile, no matter its validity.

The statistical validity of the information from school test scores is extremely limited: therefore the value of the information is also extremely limited. This is not because unions or teachers or anyone else arranged that to be so.

The late Kenneth J. Rowe (who died tragically in the recent Victorian bushfires) of the Australian Council for Educational Research (ACER), drew out these profound defaulters in numerous papers. His views are captured by this comment. “Australian politicians and senior bureaucrats currently advocating the publication of such performance information in the form of ‘league tables’ are naively, and in typical fashion, stomping around in an uninformed epistemopathological fog.”

Among the points he made (sometimes quoting other researchers) in his article in the Journal of Journal of Educational Enquiry (PDF 314KB) in 2000 are the following:

Whereas the long-term goals of school education may be expressed as the enhancement of young peoples’ access to and participation in society, as well as preparation for meeting the constantly changing demands of the modern workplace, the most direct and readily accessible measures of schooling outcomes are obtained from assessments of students’ academic attainments …

… standardised achievement tests or public examinations [which] have inherent risks as instruments of evaluation for accountability since they seldom cover more than the common core or very basic curriculum units [and] may be highly deceptive because of lacking content validity. ...

In high stakes testing environments, educational practitioners are likely to distort their behaviour in order to meet the demands of the indicator, usually to the detriment of their real job. …

Where examination scores have been used as outcome measures, differences between classes and faculties within schools are typically large and substantially greater than differences among schools, although effects are not especially consistent across faculties or from year to year … there seems to be little awareness that ... the majority of such tests assess skills in terms of generalised academic abilities and enduring cognitive “traits” rather than specific learning outcomes arising from classroom instruction.

It is not possible to provide simple summaries that capture all of the important features of schools… the historical nature of school effectiveness judgements is an acute problem… Above all, even when suitable adjustments for students’ intake characteristics and prior achievement have been taken into account, the resulting value-added estimates have too much uncertainty attached to them to provide reliable rankings.

As Rowe shows, charting the “residuals” (variations from the mean or average of all the data) of test scores together with their 95 per cent confidence intervals for a large number of schools reveals that “the 95 per cent confidence intervals surrounding the mean point-estimates for each school cover a large part of the total range of estimates, with approximately 80 per cent of the intervals overlapping the population mean (zero)”. Attempts to separate or rank schools on the basis of this data are subject to considerable uncertainty: it really is only possible to claim that the “outliers”, the 10 per cent of schools at the each end of the graph are different from each other. One cannot say anything statistically valid about the 80 per cent of schools in the middle! At this point we can see why, in the best school systems, publication of such league tables is not done!

League tables are pounced on by tabloid media and many politicians, often in a nonsensical manner, some headlines proclaiming schools were “failing half their pupils”. As Rowe said, “… it is difficult to find ways of helping [schools] in a prevailing social and political atmosphere of blame, recrimination and retribution”.

In New Zealand, the “marketisation” flowing from publication of “league tables” was found (by Hugh Lauder & David Hughes et al, Trading in Futures Why Markets in education don’t Work, Open University Press, Buckingham, 1999) to lead to no significant improvement of student performance but to simply polarise schools by social class and ethnicity. The same is true of Australia: that children from lower SES environments tend to be concentrated in public schools fallaciously suggests to some that public schooling is inherently inferior!

It is of interest that the Good Childhood Inquiry in Britain which reported February 5, 2009 recommended that “Government should:

• replace all SATS tests with an annual assessment designed mainly to guide a child’s learning; and
• stop publishing data on individual schools from which league tables are constructed by the media.”

The Children’s Society’s statement on schooling says, “… exam grades and qualifications must not be seen as the primary objective of children’s education, rather one of the markers of children’s growth, learning and achievements among many others”.

In study after study, good teaching has emerged as the principal determinant of high performance. But this is not achieved simply by financially rewarding teachers whose students achieve relatively high scores in tests. It is achieved by attention to recruitment and training, by valuing teachers - not least by setting good starting salaries - and by “relational trust”, building relationships between community, school, teachers and students.

These are results from studies of schools in Finland and other European countries whose students consistently achieve the highest scores in international tests (where “league tables” are not used), and in Chicago where significant improvements have taken place as revealed by longitudinal studies. These are also conclusions from studies by the OECD of school leadership and an analysis of the results of international tests by McKinsey and company in “How the Best Performing Schools come out on Top” (PDF 9.51MB) and of schools around the world by Professor Brian Caldwell and Dr Jessica Harris published late last year as “Why not the Best Schools?”.

Test scores are useful in tracking the performance of individual students over time: studies show that formative evaluation, focusing on achievement as it occurs, achieves results while summative evaluation such as test scores does not. The best performing schools use tests in this way.

“Chris C” noted in response to Leigh’s article, “Australia consistently performs in the top ten countries in the world in reading, mathematics and science. The most recent OECD Program for International Student Assessment put only five countries statistically significantly ahead of us in reading, only eight countries in mathematics and only three countries in science. The UK is behind us in all three areas, and the US in two areas, despite their more detailed school reporting”.

In commenting on Australian students’ performance in international tests, ACER’s CEO Professor Geoff Masters has consistently observed that while the results are good there is room for improvement. ACER commentary has also observed that in some areas other countries are improving their performance while Australia is not. A major feature of international tests is the number of students reaching the highest levels: far fewer in Australia than in leading countries! A feature of the last 12 years has been a decline in Commonwealth government funding of public schools and of universities.

By no means the least important realisation to emerge in the last few decades has been the recognition of the profound significance of early childhood intervention in education, especially where socially less advantaged children are concerned. In Australia as in other countries, particularly the USA, areas of disadvantage including remoteness from major centres and high proportions of ethnic minorities, are factors contributing to lower educational outcomes. The economic and social consequences are profound and recognised by Nobel prizewinning economists and others!

If we genuinely want to improve educational levels we will pay attention to sound scholarship, and to what we can learn from best practice.