Keeping score: in search of an accurate Environmental Sustainability Index

Last January, an organisation calling itself the 'Global Leaders of Tomorrow Environment Task Force' published what it described as an ESI – '2001 Environmental Sustainability Index'. The task force was appointed by the global economic top dogs who make up the World Economics Forum. The index was, said the Task Force, an attempt to measure and compare the 'environmental sustainability' of all the world's nations. Its conclusion was undoubtedly pleasing to the economic top dogs gathered at Davos: the most 'ecofriendly' nations were the world's most industrialised. The 'ecooffenders' were the poor. 'The good news,' wrote the chairman of the taskforce in Newsweek magazine, 'is that a clean environment may not have to come at the expense of economic competitiveness.'

It is certainly good news that such tables are beginning to be compiled; properly done, they will indeed give a useful picture of which nations need to do most. But the ESI is misleading in the extreme, and represents some of the worst eco-villains as the world's good guys; to the benefit of the powerful nations. This implies not only that modern industrialised nations are getting it right on the environment, but also that to be 'environmentally sustainable', poorer countries need to go down the same development path as richer countries. In fact, achieving genuine sustainability will require far more changes from richer countries.

The Ecologist and Friends of the Earth, in an attempt to rectify this, have reformed and recalculated the methodology by which the ESI was produced. The results – as the maps and tables accompanying this article demonstrate – tell a very different story.

The ESI comprises 22 equally weighted indicators, in five categories. These 22 indicators, in turn, are made up of differing numbers of 'variables' – there are 67 variables in total. It is this choice of categories, and the number of indicators within them, which is at the heart of the problem.

The choice of categories is poorly justified, as are the numbers and types of indicators and variables within them. Yet these choices have great bearing on the final ESI ranking. This leads to a flawed categorisation, on a number of levels.

What's wrong with the ESI?

Too many socioeconomic indicators.

There are two separate conditions that need to be met for genuine sustainability. Firstly, a 'socioeconomic imperative' – addressing people's quality of life, including their health, standard of living, economic security and social justice. Secondly, an 'ecological imperative' – that humanity as a whole does not use more ecological services than nature can regenerate.

‘Environmental sustainability' could be defined specifically as meeting this second imperative. A genuine ESI should therefore measure the extent to which this condition is met. Although issues such as infant mortality rate, and the percentage of the population with access to safe drinking water (which are included as part of the ESI's 'Reducing Human Vulnerability' category, above) are critical aspects of sustainability, they are socioeconomic, not environmental, aspects of it. Including them in an index explicitly measuring environmental sustainability is thus an error which introduces bias in favour of countries able to provide these socioeconomic services – in other words, rich countries.

Too many 'capacity' indicators

A category called 'Social and Institutional Capacity' swamps the ESI. This one category (of five) supplies seven out of 22 indicators. Also, having the 'capacity' to deal with environmental problems is not a measure of whether these problems actually get resolved. There are a number of issues here:

• What does 'capacity' mean?

The 'private sector responsiveness' indicator includes as one variable – and a positive one – a country's number of members on the World Business Council for Sustainable Development (WBCSD). The logic is that the WBCSD is part of the solution to the world's environmental problems, because the WBCSD embodies the capacity to 'get things done'. Yet these same WBCSD members also have capacity to destroy environmental assets. It would be difficult to argue that member companies such as Rio Tinto or Texaco were using their 'capacity' as a whole to improve conditions of 'environmental sustainability', rather than destroying it in large swathes.

• Duplication

Some of the capacity indicators duplicate measurements already used elsewhere in the ESI. For example, there is a 'price of premium gasoline' variable within the 'capacity' section – but there is already an indicator of the end result – CO₂ emissions, and air pollutants. The price of gasoline has a greater effect on the overall ESI than CO₂ emissions.

• Poor proxies for positive action

Some indicators are not directly related to environmental sustainability at all. An example is the 'number of scientists per head of population'. The effect of science on environmental sustainability is, of course, dependent on how it is deployed. Having lots of scientists around is not much good for environmental sustainability if their efforts result in nuclear wastes, endocrine-disrupting chemicals and ozone depleters, even if some short-term social or economic gains are made. Again, the potential for capacity to be used in a negative way is not recognised.

• Not enough on global indicators

The indicator set is very focused on environmental effects or institutions within countries. However, many countries escape environmental constraints in their own country by drawing on environmental capital either from other countries or from global commons. The biggest overexploitations of nature's carrying capacity are occurring at a global level – fish stocks, climate change, ozonelayer depletion. There are variables in some indicators which measure global impact – but these amount to just over one indicator out of 22 – around 5 per cent of the index.

• The weightings

The ESI team decided – after exploring various angles – not to assign different weightings to any of the indicators. However, by getting to 22 'equally weighted' indicators, substantial weighting has already been done – in choosing the 22 and what they comprise. The category of 'Capacity' has seven indicators, for example, while the category 'Global Stewardship' has just two. There is a whole indicator measuring countries' economies' carbon intensity – yet the end result this affects, carbon dioxide emissions, the cause of worldwide climate change, gets only two fifths of an indicator.

Moreover, the ESI team did in fact conduct a weighting survey – whose results put five of the seven capacity indicators in the bottom six of the 22 indicators, in order of importance. There is an overwhelmingly disproportionate bias towards the indicators of 'Capacity', which is a major cause of the index's high scores for rich countries.

• Individual problems

There is also a wide range of other problems with many of the specific indicators themselves. For a start, indicators are used in one country when the problem may be caused by another: incidences of acid rain being an obvious example. On top of this, major problems are omitted, leading to bias against particular countries.

The environmental health indicators, for example, have variables of 'child death rate from respiratory diseases' and 'death rate from intestinal infectious diseases'. These could be caused by a number of factors, not necessarily due to environmental problems (lack of public health service, for example – a socioeconomic issue), and/or by factors already covered (such as access to safe drinking water). In only including infectious and respiratory diseases, major categories of illness are missed out – such as environmental ill health resulting from Chernobyl, Bhopal etc.

On top of all this, what might be called 'timing issues' are neglected. The population indicator comprises 'fertility rate' and 'change in population between 2000 and 2050'. The logic is that high population is an indicator of environmental damage. But if they believe this, then these indicators need to be weighted with another which looks at current population (density). Using their current measure, a country with low population but high population growth will score much worse than an already densely populated country with a static population. Yet the latter country could easily have far greater population pressure on the environment. This indicator category is the only one which measures future trends – and this bias is against poor countries. A better indicator might have been current population per acre of arable land.

Then there are the methodological inconsistencies. Three variables measuring the effects of one country on global systems use the measure of impact per capita. However, the CO₂ emissions variable is measured in terms of 'tonnes of carbon times tonnes of carbon per capita' – apparently doing this 'reflects two ways to measure global responsibility'. Yet this implies that simple 'tonnes of carbon' is a way of comparing countries' responsibility. If this were true, then the implication is that all countries should emit equal amounts of carbon – Luxembourg the same as China, Chad the same as the USA. The effect of doing this is to have an indicator of emissions squared divided by population – which creates wild extremes between countries, and has the effect of reducing the score for heavily polluting countries. The ESI uses a methodology for wildly skewed indices which in this case makes higher carbon per capita countries score better than if a simple carbon per capita figure were used.

In addition, inappropriate units are occasionally used in the index. For example, one variable is Forest Stewardship Council-accredited forest area as a percentage of total forest area. This penalises countries with large tracts of forest untouched by forestry. A better indicator would be FSC production as a percentage of total wood production – a better measure of the relative unsustainability of a country's forestry practices.

Finally, there is a bizarre bias against countries from whom no data at all on certain subjects has been collected. The ESI team uses a methodology to deal with nonreporting countries, which we feel can heavily penalise those countries. For example, there are three variables in the 'Urban Air Pollution' indicators. There is no data for these three variables for Haiti, yet Haiti is put in 98th place in this indicator. Similarly, the nuclear waste indicator has data missing for 77 countries and it appears that some, if not many, of these 77 countries have not been given the obvious '0' for nuclear waste (as they have no nuclear facilities), but some oddly extrapolated, larger value.

Summarising the mistakes

The ESI team says "unlike many efforts to think about indicators of 'sustainable development', we have focused on environmental sustainability, which is a more narrow formulation. This choice was made deliberately, based on a conclusion that one reason efforts to measure sustainability fail is that they seek to fold too many disparate phenomena under the same conceptual umbrella. While we accept the premise that politics, economics and social values are important factors worthy of being sustained, we do not think there is a sufficient scientific, empirical or political basis for constructing metrics that combine all of them along with the environment. Moreover, the environment often gets overshadowed in 'triple bottom line' analyses and other sweeping sustainability efforts".

While this may be true, the fact is that the ESI, with this survey, falls into its own trap. The inclusion of social and economic indicators allows it, conveniently, to conclude that the world's richest, highest consumption, highest-impact nations are the true protectors of the environment – and that the poor need to follow their lead.

How it should be done?

Some of the data in the ESI can be used to create a far better measure of 'environmental sustainability' – and this we have tried to do here. Our approach has been first to cull all the non-environmental indicators, then all the ones with methodological problems. This left us with indicators in seven areas. Four relate to how a country uses its own environmental resources – with effects on land, air, water and biodiversity. The other three relate to a country's use of global environmental resources – the categories being pollution to land, pollution to air, and resources extracted.

We used the same methodology and data as the ESI to calculate rankings for these (sometimes different) indicators. Using the ESI's methodology, values for the variables for each indicator were averaged to get indicator scores; so the revised index totals are the average of the seven indicators we used. The one change made was to calculate CO2 emission scores based on per capita emissions. We believe this simplified index (comprising seven indicators and 15 variables, rather than 22 indicators and 67 variables) captures true 'environmental sustainability' much better than the original.

The real story

When these more genuine indicators are factored in, and some of the more bogus ones removed, the final results tell a very different story about which countries are really environmentally sustainable – and which aren't.

One of the most obvious examples is the very unsustainable US which, under these new calculations plunges from 11th in the rankings to 112th. At the other end of the scale, the Central African Republic, Bolivia and Mongolia are elevated to the top three. This reflects the good environmental conditions in their own country, and the small effect their development has on global ecosystems.

In all, richer countries do worse – for although they often have good environmental conditions at home, and manage to protect what's left of their biodiversity, they have a large negative impact on global ecosystems. Some countries, of course, score badly whichever way you look at it – the bottom three are South Korea, Kuwait and the Lebanon.

What conclusions should we draw from this? Simply, if we are going to label nations 'good' or 'bad' in environmental terms, we must get our measurements right. Studies like the ESI, based on misleading data, which fail to take into account the true environmental costs that rich countries impose on the world, are designed to make dirty nations look clean.

When the 'Global Leaders for Tomorrow' next sit down at their calculators, their results should tell a very different story.