The Map and the Territory: the problem with models

This essay is based on a peer-reviewed article by a statistician and a mathematician (who call themselves 'physicists' at the end) in an economics journal. Its title is 'Escape from model-land'. It's been about for a couple of months, but there'll be many readers for whom the article is new and important, so what follows is my summary, which is informed by my own experience. .

The authors start their abstract with what seems to me a great and often unrecognised truth.

Both mathematical modelling and simulation methods in general have contributed greatly to understanding, insight and forecasting in many fields including macroeconomics. Nevertheless, we must remain careful to distinguish model-land and model-land quantities from the real world.

Let me give an example from my own work. I was interested in the extent to which local issues and candidates affected election results. Votes are counted first at polling booths, then they are aggregated at the sub-division level, then at the divisional level (the electorate), then at the State or Territory level, and finally for the whole country. I could use a simultaneous-equations model to estimate the average effects at each level, and I did so. That process gave me an estimate of the various 'effects', really local ones, wider local ones, divisional ones, and state ones. So when you said that Labor had won, say 47 per cent of the vote, you were looking at a lot of separate contributions, both positive and negative, to that outcome.

And although I was able to give numbers to these 'effects' those numbers were estimates only. More, these effects were highly simplified versions of the real world, where people voted as they did for all sorts of reasons. Not one of them was interested in these 'effects' of mine, but they were affected by how rusted-on their party loyalty was, whether they knew any of the candidates, what they thought of them, what issues in the campaign had resonated with them, if any, how grumpy they felt that day, and so on. My model was the 'map', while the all-too-human reality of election-day was the 'territory'. I found the map/territory analogy in the article and think it is a most useful one.

The authors offer what they call 'a short guide to some of the temptations and pitfalls of model-land', a map of which they provide. It has some amusing descriptors. They are concerned that simulations and models are too frequently used to inform policy, when the modellers do not properly explain the limitations of their work, and the policymakers do not understand the limitations anyway. What we then get is policy-based evidence, rather than evidence-based policy. 'Climate change' is one of the areas the authors single out for attention. In their view (and it is mine also) whether or not models are useful for policymaking has to be determined by looking at whether or not the models can explain the past properly, and whether their predictions about the future prove to be correct, 'never based solely on the plausibility of their underlying principles or on the visual "realism" of outputs'.

In model-land models are tested against one another, simulations against other simulations. This process:

promotes a seductive, fairy-tale state of mind in which optimising a simulation invariably reflects desirable pathways in the real world. Decision-support in model-land implies taking the output of model simulations at face value (perhaps using some form of statistical processing to account for blatant inconsistencies), and then interpreting frequencies in model-land to represent probabilities in the real-world.

One of the problems in so doing is the zero probability of what they call 'the Big Surprise' - an event which often occurs in the real world but not in model-land.

The authors are scathing about something that we see again and again in climate science.

For what we term 'climate-like' tasks, the realms of sophisticated statistical processing which variously 'identify the best model', 'calibrate the parameters of the model', 'form a probability ensemble from the ensemble', 'calculate the size of the discrepancy' etc are castles in the air built on a singe assumption which is known to be incorrect: that the model is perfect.

Mathematicians thrive in model-land, they say, and they can show also that interesting solutions will not hold in the real world. Mathematicians also ask for greater computational power, which has become increasingly available.