Funding scientific research: money can't buy love

Australia has recently become concerned about its world standing in terms of scientific research. There are many ways to assess a country's position, but the issue comes down to three basic questions. How much science does Australia produce for the amount of money it spends; what is the quality of its science and what makes for high quality anyway? This is a brief analysis of these three questions for both policy makers and the general public.

When Australia compares itself with other countries, it is often selective, emphasising the 34 OECD countries or a subset of them (Pettigrew, 2012; McKeon et al., 2013; West, 2013; Chubb, 2014).

The omission of countries for which there are data, however, creates a distortion.

For example, the Office of the Chief Scientist recently selected 11 of the 34 OECD countries against which to compare Australia's placement in the percentage of papers in the top one percent of papers cited. The Office concluded, without giving details, that "our best are very good" and we "do well" on this metric. However, 39 countries are actually available for the comparison, including 27 from the OECD, and overall Australia comes 13^th in this expanded comparison (pers. obs. on data in van Noorden, 2012). Is this "very good" and "doing well?" In the analysis here, no conscious omission of countries is made.

Amount of Science Produced

It seems intuitively obvious that the amount of money a country spends on scientific research would be an important determinant of the amount of science it produces. But if this is the case is the relationship linear or curvilinear and how tight is it? Strangely, this relationship is rarely examined.

The most complete recent data to assess this relationship comes from 2011 when Nature magazine totalled the number of scientific papers published per country for that year (van Noorden, 2011). This is the dependent variable and can be taken as a proxy for the amount of science, or knowledge, produced. The total amount of each country's GDP spent on R&D in the preceding two years, 2009 + 2010, is a reasonable independent variable. This latter variable can be calculated from data for each country's GDP and the percentage of GDP spent on R&D (World Bank data). Data are available for 38 countries including 29 from the OECD. There are more recent data (van Noorden, 2012), but these include one less OECD country. The results, however, are similar.

The plot is shown in Fig. 1. The best-fit line for the data is a quadratic polynomial (but better than a linear only in the third decimal place) and shows that the amount spent on research accounts for 87 percent of the variance in the number of papers produced. When the three strongest outliers (China, Japan and the USA) are removed, the best-fit trendline, a polynomial, explains 82 percent of the variance.

Fig. 1. Number of papers published in 2011 (thousands) compared to the amount spent on R&D in the two preceding years (2009 + 2010) combined for 38 countries, including 29 of the 34 OECD countries (Chile, Estonia, Hungary, Luxembourg, and Slovenia excluded for lack of data on papers published). Australia is shown in red.

This plot suggests that if a country wishes to produce more science in general, the most effective method would be to allocate more money to R&D. This is an eminently amenable policy initiative. Realistically, however, the maximum expenditure on R&D as a proportion of GDP in the period 2009-2013 by any country was 4.2 percent (Israel in 2009; World Bank data). This number may represent the upper limit for any country under current conditions. Australian allocated 2.21, 2.20, 2.20 and 2.20 percent of its GDP to R&D in 2009, 2010, 2011 and 2012, respectively, and in 2012 the average for 35 countries was 2.06 (pers. obs.).

As to countries' positions in this plot, Australia is just above the trendline. Seven countries, however, are even further above it: India, Spain, Italy, Canada, Great Britain, China and the USA. Policy makers may wish to investigate why these countries, especially Great Britain and China get more science for their dollar than does Australia.

Australia often decries its placement among the OECD countries, especially as being the only one without a national science and technology strategy (Office of the Chief Scientist, 2014). On the metric of the amount of science produced per dollar spent, Australia exceeds all but five of the 29 OECD countries for which there are data. Is this a case for or against a national science strategy?

It's worth noting that some assessments of Australia's comparative R&D output use R&D as a percentage of GDP, instead of the absolute amount of money. However, this is a derivative variable and does not reveal, as absolute expenditure does, the specific dollar input spent to achieve a specific amount of science ouput. For example, the best-fit trendline for papers published in 2011 versus R&D as a percent of GDP for 2009 (N = 37) is a power function that explains only 10 percent of the variance.

Impact of the Science

Quantity of science is one thing. Quality is another. Measures of quality, for better or worse, are usually based on the number of citations a country's papers receive. The Office of the Chief Scientist in Australia, for example, accepts that a measure of citation rate is a proxy for the quality of science (Pettigrew, 2012; Office of Chief Scientist media release, "Benchmarking Australian Science Performance," 22 February 2013; West, 2013). Here, the OECD's "normalized impact" is used (see OECD, 2013:60 data and definition). Parenthetically, the normalized impact (years 2003-2011; OECD, 2013 data) is highly correlated with the average number of citations per paper (years 2003-2013; SCImago data; R = 0.95, N = 40, P < 0.01, pers. obs).

Does the total amount of money spent on R&D money affect the "impact" of a country's science as well as its quantity?

When countries' impact factor is plotted against the amount spent on R&D (years 2009 + 2010), the best-fit trendline is a polynomial, which accounts for only three percent of the variance of the impact measure (Fig. 2). So the money spent on R&D doesn't seem to affect the quality of a country's science.

Fig. 2. The normalised impact ratio compared with R&D expenditure in 2009 + 2010 for 40 countries, including all 34 OECD countries. Australia is shown in red.

What other factors might affect the quality of a country's science? Recently, the OECD found an association between the percent of international collaboration among institutions (years 2003-2011; OECD, 2013: 60 for data and definition) and normalized impact (above). While it identified a "positive relationship between measures of scientific research collaboration and impact (OECD, 2013: 60)," it did not analyse the shape or goodness of fit of the relationship.

When this analysis is done for 39 countries, including all but one OECD country (Greece), the best-fit trendline with collaboration as the dependent variable is a power function that accounts for 41 percent of the variance of impact (Fig. 3). In other words, the amount spent on R&D is not a good predictor (determinant?) of a country's research impact but the degree of collaboration is.

Fig. 3. The normalized impact ratio compared with the percentage of international institutional collaboration for 39 countries, including 33 of the 34 OECD countries (Greece omitted). Australia is shown in red.

In this plot, Australia is respectably above the trendline, but it is surpassed by six countries: United States, Great Britain, Netherlands, Denmark, Switzerland and Iceland, all in the OECD. If policy makers are interested in increasing the impact of their country's research beyond what increased collaboration can achieve, then scrutiny of the policy settings in these countries, especially the smaller ones such as Denmark, Iceland, Netherlands and Switzerland would be informative. Like R&D money allocated (above), the percent of international collaboration among institutions is amenable to policy changes.

And just to note, the amount spent on R&D (years 2009 + 2010) was inversely correlated with the percent of international collaboration among institutions (years 2003-2011; R = -37.6, N = 40, P< 0.02). This suggests actual policies affecting collaboration are more important that the sheer size of the R&D budget.

Finally, how much of the crème de la crème of impact does collaboration gain a country? When countries' percent of the top one percent of cited papers (van Noorden, 2012 for data.) is compared to their degree of collaboration (above) using the 32 available countries including 27 from the OECD, the best-fit trendline is a power function (Fig. 4) that accounts for 69 percent of the variance (a linear function differs only in the third decimal place).

Fig. 4. The percent of a county's papers in 2012 among the top one percent of papers cited for 32 countries, including 27 of the 34 OECD countries (Chile, Estonia, Hungary, Iceland, Luxembourg, Slovenia and Slovakia omitted). Australia is shown in red.

Australia is above the trendline, but ten other countries clearly exceed it: USA, Czech Republic, Great Britain, Israel, South Africa, Netherlands, Denmark, Austria, Belgium and Switzerland, all but one of which is in the OECD.

In policy terms, it looks as if collaboration helps the cream rise to the top.

In summary, in scientific research, money doesn't buy impact but collaboration does.