Good old Collingwood forever

Evaluation must also generate disciplined, transparent knowledge of why things work. And that kind of knowledge is a gateway both to greater insight as to what kinds of changes – in the program or in the context it operates – might affect the program's efficacy. This is Collingwood's idea that knowledge comes from, and can only come from, asking the right questions in the right order.

I'll let him elaborate:

For example, if my car will not go, I may spend an hour searching for the cause of its failure. If, during this hour, I take out number one plug, lay it on the engine, turn the starting-handle, and wait for a spark, my observation "number one plug is all right" is an answer not to the question, "Why won't my car go?" but to the question, "Is it because number one plug is not sparking that my car won't go?" Any one of the various experiments I make during the hour will be the finding of an answer to some such detailed and particularized question. The question, "Why won't my car go?" is only a kind of summary of all these taken together.

So there you have it – program evaluation a la R.G. Collingwood a few decades before the ideas were formalised into program evaluation. I commend him to you as an antidote to a lot of confused and wishful thinking at the top of our hierarchies of organisation and knowledge, in which the tale of a certain method and its apparent rigour wags the dog of what we need to know.

As I like to say: Good old Collingwood forever.

If you agree with me that these are some of the things we need to do, and particularly some of the things I'm hoping policy makers have in their mind as they work out how to spend that $40 million to build the evidence base in for indigenous programs then we need to:

• Build the status of delivery alongside policy;
• Build the status of program evaluation over the blithe context independent presumptions of those arguing for the dominance of RCTs; and
• Deliver evaluation which directly helps those in the field improve their efficacy; whilst at the same time
• Generating transparency for those outside the program as to how it's going
• Find ways to generate knowledge that is transferrable, that helps us learn how to deliver more successful programs.

Is this possible, or is it a pipe dream?