The launch, the crash and the recovery of My School

By now everyone will know of the erratic launch of My School, www.myschool.edu.au the effort by the Rudd Government to release standardised and test-based reports for every school in Australia. The launch, crash and subsequent political (rather than aeronautical) spin was fun to watch and made good media - but it was a sideshow that tended to detract from so many of the real issues.

Those able to successfully log on found a site with authoritative statements about the accuracy and fairness of the school data being used. Armed with this assurance they could then cast their eye over the figures and little coloured boxes to find out how any school compares with similar others, green for substantially above in reading, writing, spelling, grammar and numeracy, red for substantially below and various shades in-between.

The website, developed by the Australian Curriculum, Assessment and Reporting Authority (ACARA), doesn’t directly compare schools that are not similar - but it helpfully provides the names of 20 local schools so that you can do it yourself. But why bother? The league tables weren’t far behind and will be around for years to come.

The people at ACARA know very well that their data shouldn’t be used in this way and say on their website that any resulting simple comparisons of schools will be misleading. Almost in chorus the various education ministers have also denounced “simplistic league tables” - while still fully ensuring that anyone can easily construct them.

The debate about school league tables has raged backwards and forwards for some time: whether or not they distort the curriculum, encourage cramming and even cheating, improve schools and systems of schools, the problems in using student test scores to say anything useful about schools.

But what about the actual school reports on the My School website?

My School lists up to 60 statistically similar schools against which any school can be compared. The use of “statistically similar” and “like school” conveys all the assurances that any comparisons created from the data are going to be fair and reasonable.

Leaving aside all the statistical juggling, ACARA considers 16 factors which apparently have a strong correlation with student performance. All schools are given an index number and are then grouped according to their level of relative socio-educational advantage - which by itself is a league table - and this is supposed to enable meaningful and fair comparisons.

To create this Index of Community Socio-Educational Advantage (ICSEA) they use the socio-educational characteristics, not of each student’s family, but of their census collection district. The funding formula for private schools, hardly held up as a model of efficacy, is constructed in a similar way.

So in trying to account for the differences between schools ACARA is already off to a shaky start. But it needs to get everything right: when you get into the business of comparing schools, with all that this entails, there can be little margin for error - too much is at stake.

The biggest problem is that 70 per cent of the differences between schools are due to which students each school enrols, not what they actually do as schools. No less than Professor Barry McGaw, the head of ACARA has repeatedly stated this.

This means that any comparative index used by ACARA must take account of everything and anything that kids bring with them when they come to school and impact on their test scores. This includes such things as prior learning (from schools and home), their family income (not the average for their suburb), gender (yes, boys and girls are different), how recently they arrived in Australia and specifically where they came from.

Any index must consider how they were selected into the school, the size and location of the school, especially the proximity and nature of competing schools. We are supposed to be told about the resources of each school, something now promised for the future.

In addition to factors considered by ACARA, differences between schools are significantly created because some schools can largely choose who does walk in through the gates each day and others can’t. The dividend for those who can is quickly realised in school results and the boosted reputation of those schools which can import much of their success and, when push comes to shove, export their failures.

In walking away from trying to include such factors ACARA is not even get close to getting it right. The creation of school comparisons in spite of such differences between schools makes a nonsense out of the precise numbers, graphs and colours that will absorb our attention over the next few weeks.

This has reached farcical levels when you consider some of the schools that have emerged as “statistically similar”. Asquith Boys High School in Sydney apparently has much in common with MLC Burwood, until you remember that the “L” in MLC stands for ladies. One of Australia’s biggest schools, Cherrybrook High is apparently similar to Upper Sturt Primary (enrolment 37) in South Australia. But hope is at hand: the students at Malek Fahd Islamic School apparently have a lot in common with those at Immanuel Lutheran School at Gawler East, South Australia.

The serious point is this: it’s bad enough in this country that we have allowed our schools to increasingly form a social and academic hierarchy, one which has little to do with school merit. To then turn around and rank these schools, even against so-called similar others, amounts to a cruel joke.

Without doubt our parents and the wider community have a continuing right to know that their schools are doing their very best for all our students. But there is ample evidence to show that far better school performance and improvement is gained through independent appraisal of school progress to identify and improve areas of weakness. Far better system performance is achieved, not by ranking schools in some failed quasi-market experiment, but by co-operative development and commitment to quality teaching and learning.

Professional and independent appraisal and development of schools doesn’t resonate as well as the current rhetoric about “transparency” and “openness”. But the Rudd Government came to power extolling the virtue of evidence-based policies. It must take heed of the evidence and properly resource ongoing independent and thorough appraisal of our schools.

Trying to achieve quality schools and give assurance to the public by using student test scores to rank schools is quick, cheap, invalid, dirty and damaging. I really need convincing that it amounts to little more than institutionalised fraud.