Good old Collingwood forever

We should certainly pay far more attention to independent validation of our knowledge of what works. Indeed it's somewhat shocking that, for a country which is or at least was one of the best policy reformers in the world, we've always been a laggard when it comes to RCTs.

But I'm in good company when I tell you that RCTs are one among many tools but not quite the panacea they're being made out to be. 2015 Nobel Laureate Angus Deaton agrees. 2000 laureate and one of the great econometricians of the last century James Heckman describes RCTs as "a metaphor and not a gold standard".

The thing about RCTs is that they assure us of just one thing. To be precise; they give us a known degree of confidence that, at a particular time and place, a particular treatment had a particular effect.

The idea that RCTs are a gold standard seems appealing. But it also has its downsides. It collapses the difficult task of evidence-based policy into single, discrete routines, tips and tricks. For the knowledge from an RCT to be useful these routines, tips and tricks must work independent of context – or with some additional work to test their applicability.

Note two things about RCTs. Firstly it's the view from the top. It's certainly a major problem of social research and social policy that those working in the field can talk a good game about how their intervention is fundamental to addressing social harm and injustice. And there's plenty of confused and wishful thinking amongst those in the field about the efficacy of the programs they run.

In this context an independent RCT is a very useful means by which those in senior policy positions can keep those delivering programs under surveillance – and force them into a more evidence based discourse for justifying their program.

So far so good.

But the second thing an RCT does is that, to be effective it must tame and confine the knowledge we're after – of what works in the field – into a simplified, discrete question. This is an example of one of the pathologies of economics – as distinctive to our profession as wigs and gowns are to the fanciest lawyers. Instead of careful adaptation of our methods to the kind of knowledge that would be most useful, we presuppose that methods that resemble those used in science must give us the 'gold standard' knowledge. This is the intellectual vice that Friedrich Hayek anatomised and anathematised as "scientism".

The high watermark of scientism is usually taken to be the words of Lord Kelvin in 1883 in which he argued that "when you cannot measure [something], when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind". I guess it's too bad for him that he chose to express this truth in words, not numbers. Indeed it might give us pause to realise that he couldn't possibly express it in numbers.

Be that as it may, prestigious academic journals are happy to snap up good studies of such discrete questions, especially if a well designed and funded RCT is involved. But the risk is that the knowledge will be crude and decontextualised. There's a deep academic literature on questions like "does performance pay for teachers or school vouchers, or charter schools improve student outcomes". But the answer to these kinds of questions is usually that "it depends". As Deborah Johnston puts it in discussing aid to Africa "It is an over-simplified and erroneous question to ask 'do cash transfers work?'".

A more productive question relating to the same subjects might be this: "in what kinds of circumstances might performance pay or school vouchers improve performance and what structures will help optimise outcomes". One might be able to go back into the data collected for RCTs, and it may shed some light on those questions, but it will be hard work because the whole architecture of the study is focused on a singular question, not on helping to steer our way through a specific situation.

And a great deal of the policy and delivery knowhow we desperately need can't be simplified discrete, context independent nuggets of knowledge. How does one improve mental health or domestic violence in outback communities, in the exasperated outer suburbs of our sprawling cities, or our regional towns? How do we do the best we can for children whose parents cannot or will not look after them properly.