Let users decide if research is relevant

The debate about the effect of research, the relationship between effect and excellence and whether end users should be included in the peer review process is an interesting one.

It is a reminder of an incident last year when a colleague from the US, Jeff Ullman, was visiting Sydney for the scientific advisory group meeting of National ICT Australia. The meeting was in NSW government offices on the 44th floor of Sydney's Grosvenor Tower. I was escorting my colleague to the venue in a crowded lift when I spotted someone I knew.

I introduced my visitor as a professor of computer science at Stanford University. My Australian colleague appeared impressed, but there was not a flicker of interest from any of the other people in the lift. I then mentioned that Ullman was one of the world's leading database theoreticians. A couple of people in the lift - possibly IT consultants - took more notice and glanced towards Ullman.

I then added that Ullman was also the second most cited computer scientist in the world. This attracted some more attention. I finally added that one of Ullman's PhD students had dropped out and co-founded Google.

Suddenly it registered and everyone in the lift was looking at Ullman. He is a rare example of a researcher who has made a foundational contribution, whose work has received exceptional peer recognition and whose ideas have also had influence on the scale of Google.

In most cases, the link between peer recognition and wider impact is far less direct. As a hypothetical scenario, let us take the example of a group of theoretical physicists and a group of cultural theorists. Both groups do work that is abstract, elegant, intellectually satisfying, contributes to our understanding and is in all likelihood as far removed from everyday reality as possible.

But somehow the work of theoretical physicists, at least as perceived by the cultural theorists, has broader support from funding agencies and university administrators. If one digs deeper, one may find that the group of theoretical physicists is respected by a group, that is respected by another group, that is respected by people who make bombs and missiles.

The cultural theorists are also respected by a peer group, but there is perhaps no link to a wider group that is the equivalent of the makers of bombs and missiles.

Recognition by peers has served the academic research community well and it remains an essential component of the research evaluation process. But we are also living in changing times. The increasing cost of healthcare and other competing demands on the public purse mean that research, especially that funded by taxpayers, has increasingly to be relevant.

Hence peer review, a necessary criterion of excellence, is no longer a sufficient criterion on its own. The relevance of the peer review group must also enter the equation. Research groups have a responsibility to make that link to wider impact; the equivalent of the makers of bombs and missiles.

It is no longer sufficient in today's climate - at least until the baby boomers have made their graceful exit through the system and as long as no other pressing need takes their place for the attention of the public purse - to say that one's research has impact purely because it is influencing the research of other researchers.

This can be important only if these other researchers have a link to end users with wider influence.

The traditional reluctance to include end-users in the peer review process does have the potential to make us - the academic research community - appear to the wider community as yet another self-indulgent interest group, but there is a more compelling need to involve end-users in the evaluation process.

Society is increasingly looking to the research community to create economic opportunities or to solve its problems. Solutions to society's problems and ideas that create economic opportunities seldom come packaged in a single discipline, but more frequently have interdisciplinary angles.

Much innovation occurs at the intersection of disciplines, as the context of one discipline provides the right stimulus for the development of another.

Good interdisciplinary research is inherently difficult and can be made even more difficult if suitable incentives are not in place. As researchers, we tend to build our credibility in a single discipline, and our track record ensures that our papers get a fair hearing from our peers. When we venture into interdisciplinary research, we are suddenly being evaluated by reviewers who do not have a pre-existing credibility assigned to us and some of our papers may have a more critical reception.

If at this time our research output - and volume sometimes can easily be confused for quality - is being evaluated, there is always a temptation to claw back to the comfort of our disciplinary burrows.

The peer review process is very effective at evaluating research in disciplinary silos, but less so when it comes to interdisciplinary work. End-users, however, tend to be outcome-focused and are less likely to display the biases of any particular discipline. They tend to be more concerned with and effective in evaluating the relevance, if not the elegance, of interdisciplinary research.

The Research Quality Framework is a necessary step in reinforcing society's support for publicly funded research. It will be successful if it leads to a dynamic, publicly funded research sector where the best institutions have to continue to work hard to maintain their status and where there is also opportunity for aspiring institutions to emerge as leaders if they are willing and able to do so.

The relegation system of sports leagues is often better than an ossified caste system: the latter may be stable and predictable but it seldom leads to global competitiveness.