Formal RCTs will be a small part of the progress we make on these questions. If we look at the way successful innovation works in most circumstances, most of it doesn't get down to the implementation of single ideas that work largely irrespective of context. It usually requires considerable investigation, experimentation and coordination between different parts of systems with trade-offs carefully and collaboratively explored.
Of course this must be done as rigorously and as transparently as possible with assumptions behind a program – the program logic – tested along the way. In this context it's possible to do all manner of mini-experiments which may take the form of a RCT, though it may be no more than A/B testing two ways to present a choice to a user, or various ways of wording a letter to program participants. Great innovators like Google and Amazon perform literally tens of thousands of such experiments every year, and 'nudge units' around the world are slowly taking these experiments closer to business-as-usual in government.
Given how capacious our ignorance is, and will always be, being humble and prepared to adapt one's methods to the problem at hand is a good starting point for a discipline.
To explain why let me offer a confession. I'm a Collingwood supporter.
Whn Collingwood won the Premiership in 2010 I rather feared that the high it would induce would give way within a day or so to remorse at the utter triviality of all those decades of yearning. I'm here to tell you that, to my shock and shamed surprise, I'm evidently a trivial person because it put a spring in my step for pretty much the next year!
But I digress, because the Collingwood I'm referring to isn't my beleaguered football team – which right now is a pretty clear counterexample to the first axiom of economics – that everyone always acts in their own self-interest.
I'm talking about a philosopher I want to recommend to evaluators everywhere: R. G. Collingwood, whom I ran into when studying history at uni. History, you see is like evaluation in that it has no overarching theories to impose on its material. Unlike economics, it puts great store in attending to the material before it on its merits.
In any event, if you read R. G. Collinwood's terrific little autobiography, which sketches his intellectual development, you'll come across a story which he uses to explain where his philosophy starts. It starts with questions.
Every day I walked across Kensington Gardens and past the Albert Memorial [which] began by degrees to obsess me .… Everything about it was visibly mis-shapen, corrupt, crawling, verminous; for a time I could not bear to look at it, and passed with averted eyes; recovering from this weakness, I forced myself to look, and to face .… the question: a thing so obviously, so incontrovertibly, so indefensibly bad, why had [the architect Gilbert] Scott done it? .… What relation was there, I began to ask myself, between what he had done and what he had tried to do? If I found the monument merely loathsome, was that perhaps my fault? Was I looking in it for qualities it did not possess, and either ignoring or despising those it did?
For Collingwood, this slowly produced a revolution in his thinking. He came to believe that knowledge wasn't captured in assertive propositions like this one "demand falls as price rises" or "increasing penalties for breach lowers tax evasion and dole cheating". As he put it "knowledge comes only by answering questions". And, in order to get anywhere, "these questions must be the right questions and asked in the right order".
Just as natural science is the painstaking process of proposing hypotheses – or to use Collingwood's terminology, asking questions which make specific phenomena examples of deeper patterns in nature, so program, developmental and other forms of evaluation unpick a program into its many moving parts, each having a role in the program logic so that each element of the logic can be investigated, validated, invalidated and/or optimised.
Thus evaluation becomes not just the investigation of what has worked. For knowledge of what has worked cannot, of itself help show the extent to which it will still work as circumstances change. With apologies to Lord Kelvin, this is "knowledge of a meagre and unsatisfactory kind".
Discuss in our Forums
See what other readers are saying about this article!
Click here to read & post comments.
2 posts so far.