Merit versus diversity in NHMRC grants

Australia punches above its weight for medical research. So when the National Health and Medical Research Council (NHMRC), the main government granting agency in this field announced on October 16^th that in future, mid-level and senior grants will be awarded to males and females in exactly equal number, people sat up and took notice.

This move followed, and was to some extent a reaction to, earlier outrage at some of the 2019-2021 statistics listed below, as well as a change.org petition of which the Council was self-consciously aware, despite the fact that "funded rates for women and men have been close to equal since 2017".

What is the background of this decision?

NHMRC grants are highly competitive and only around 14% of applications are successful. The application itself is far from a costless exercise and can take many weeks of full-time work for the lead researcher. An affirmative action element for grants already exists: the Structural Priority Funding scheme allocates a pool of money to unsuccessful female and non-white applicants until it is exhausted.

The table below is for Investigator Grants, which is an important class of grants, but not the only class. Data for some other classes, for instance, Centres for Excellence grants and Synergy grants, suggest that females have enjoyed more success than males. But let's stick with the Investigator grants, which have generated all the controversy and is the grant class that the NHMRC have targeted for intervention.

Around 60% (34.3%+24.8%) of applications are by junior researchers, who are designated Early Leaders. This kind of language speaks to the US style managerialism that has infected much of Australian research and research institutions. A minor point perhaps but culture, especially US culture, is at the heart of this controversy. Nobody wants to be a humble researcher anymore. We need leaders and you best become a leader as early as possible.

Amongst these junior applicants, females are the majority and there were 137 successful females compared to 123 males. Indeed, females make up the majority of those employed in medical research among the under 30's. As we move into the more senior researcher levels though, males dominate applications, largely reflecting their legacy dominance in the industry. And because success rates tend to be higher for more senior applicants, this results in more overall grants to male applicants (422 compared to 313). That the demographic skew is the main cause of the gender difference is not just my opinion. It is stated by the NHMRC CEO Ann Kelso last February.

Another way to look at the data is success rates– the proportion of applicants in a given pool who receive a grant, see Table 2. For junior researchers (EL1+EL2) the success rates are 13.6% for males and 11.7% for females. For senior researchers (designated Leaders 1-2) the success rates are 16.7% for males and 13.7% for females. This does appear to indicate a modest tendency for male applicants to have a higher chance of success. However, this data does not adjust for the assessed quality of the project itself, upon which the NHMRC largely base their decisions.

With the new NHMRC policy, female success rates will be close to twice as high as male success rates. This directly conflicts with the claim of the NHMRC CEO Ann Kelso that the "NHMRC is committed to ensuring that all researchers have equal opportunity to undertake health and medical research regardless of their gender".

Two issues: gender dominance versus gender bias

So there are two distinct issues. Males dominate the industry at higher levels of experience, especially L3. There are more male applicants, so there are more grants awarded to males. Second, there is a modest disparity in success rates in favour of males, which may or may not be explained by quality.

On the first issue, lack of females at senior levels, it has been rather ludicrously claimed that "the largest contributing factor to lack of retention of women in STEMM at higher levels is that women receive fewer grants … due to gender bias in the peer-review system." So a lack of grants causes females to drop out? Surely, the first order effect is that fewer women in STEMM leads to the fewer grants. Talk about chicken and egg!

This "problem of too many males" will likely dissipate over the next twenty years as those junior females become more senior researchers. If there are forces that discourage women, such as family care duties, then female quotas are not the way to fix it. Why? Because many of thefemales who receivetheaffirmative action may not even have children or may be rich enough to afford a full-time nanny. Indeed, if these women are still in the industry at mid or senior levels then they are, almost by definition, less likely to have been subject to the systemic biases that lead to female attrition. Is the NHMRC going to ask female applicants if they have kids and how much time they spend taking care of them? It would probably be illegal to do so. And what about the (likely much smaller number of) males who have put their career second because of family responsibilities. Why do they not get a free ride?

This is a pervasive issue in affirmative action. If you use a crude proxy for disadvantage like sex or race then you deliberately ignore the cause of disadvantage, for instance lack of child care or a remote geographical location. Unless the systemic bias is against all females equally, a sex bonus will advantage some females who are already advantaged, and disadvantage some already disadvantaged males. Sex-based intervention should be an absolute last resort. It is as ludicrous as giving a height-based salary bonus to shorter people because males are taller and also are paid more. You kind of miss the point if you legislate to correct the gender skew with a height bonus. Short males will be the winners and tall females will be the losers.

The second problem of gender differentials in success rates deserves some serious study, but there was nothing in the NHMRC discussion paper that got to the heart of the matter. Some throw-away lines argue that it is "difficult to see and measure the biases of those who are evaluating applicants for appointments, promotions and grants." In fact, it is not as difficult as claimed. Gender-blind versions of an application could easily be used to get a sense of this.

Projects are assessed on the quality of the researcher, based on their track record, and on the project itself. Are there any gender (or more generally identity) blind measures of researcher quality that could be used? I would be strongly in favour of this and have lobbied hard to include identity-free assessment of job applicants here at University of Melbourne, with some success.

Is it possible the quality of the female-led projects was lower? The data in the NHMRC discussion paper suggest a modest difference. If so, is it an artifact of bias? And if so, how can we remove it?

If you mention quality and merit in some places, you are liable to be shouted down, but if the NHMRC are not in the quality and merit business, then they need to be closed down. The task may not be straightforward but it is not impossible either. Journals publish papers, universities appoint and promote. And there is a market to challenge and replace journals and universities that systematically get it wrong. Granting agencies need to get it right, too, especially in medical research, where huge patents are in the offing.

Identity is a fuzzy concept and irrelevant to merit-based grants

Does it really matter how many grant-holders are female, or male, so long as the allocation is merit based? To some it does matter, but not to me – or anyone with a clear moral compass.

If 100% of successful applicants were originally from Iceland I would not care at all in principle. Certainly such an extreme skew in recipient demographics could be a possible indicator of a systemic problem. There might be some subtle Icelandic bias that I would look for and correct if I found it. But 'too many' Icelandic researchers is not a problem in itself. Alas, too many male researchers seems to be a problem.

But there is another more important point to make in relation to the entire framing of the issue by the NHMRC. Why are grants being classified according to the sex of the lead researcher? The leader is one of many who benefit from a grant, but so do the others working on that project. L3 grants, for instance, tend to be larger and employ many junior post-docs. Most of these will likely be women.

A more sensible analysis would be based on the sex of the researchers who benefit, not of the grant holder. But the gender skew would not then be so stark – and would not generate so many angry tweets. How boring.

What about diversity?

Many claim that diversity is an important driver of better science. This is based on various artificial laboratory experiments that manipulate team profiles and measure their success on well-defined 'decision' tasks. There is less evidence of a diversity dividend outside the laboratory.

Diversity, in any case, has little to do with Y-chromosomes. Where is the evidence that gender predicts different medical research approaches and frameworks that would lead to a diversity dividend? The wisdom of crowds is real, but it is not about identity-defined diversity. It is about actual diversity.

Moreover, in medical research in particular it is hard to see how diversity is a key driver of excellence. Researchers are specialists with a deep understanding of a narrow topic. They are not a random independent sample of the population brain-storming a question that a social scientist has just lobbed in front of them. Newton did not develop his three laws by consensus and committee. He thought about it alone for most of this life. There was no diversity involved at all. Research success requires you to develop a unique combination of expertise and then be single minded in pursuing your research agenda, regardless of setbacks and scepticism.

Moreover, why does the diversity argument not apply to vocations like primary school teaching, where it is becoming increasingly difficult to recruit males? For example take the profession of psychology in the US where 71% of psychologists are female and, amongst under 30s, 95% are female? It is clear where this trend is heading. And the gender of your teacher or psychologist is surely more relevant than your unseen medical researcher. It's medical research, to improve health outcomes for us all. I don't care about their identity. I only care about the research.

And the elephant in the room? Those who assess grants are far from diverse. And by this I do not mean white and male. These are the survivors of a competitive, decade long vocational process that is as much social and cultural as scientific. The assessors are flattered to be assessing the grants of others, including their betters, and they are only human so they also have their own incentives. With the best will in the world, a researcher in malaria will think that research in malaria is more important than any other research. And researchers who are part of the current consensus will tend to receive more favourable assessments than the maverick researcher who wants to disprove the prevailing norms of the assessors.

Undermining female achievement

Affirmative action inevitably leads to the perception that female achievements are not merit based. This is not just my opinion. Below is an email I received from a very accomplished female data scientist who wasscared to have her name published. She is one of several folk, most of whom were female, who thanked me for challenging this thoughtless NHMRC decision.

"As someone who has received awards designated for women, I can tell you first-hand what a terrible impact they have. Others couldn't help but assume that I'd won them because I'm a woman, not because I deserved them. Worse, I myself didn't know whether I deserved them! Early career researchers in particular do not need additional reasons to doubt themselves.

I have children of both sexes, and I fear for them. I want them to be judged on their merits alone, but that wish seems unlikely to come true in the current climate."

What should the NHMRC have done, if anything?

The NHMRC discussion paper reads like polemic. The data analysis is one-sided and incomplete. The only options considered are four different versions of affirmative action The possibility of just waiting for more females to wash through the STEMM population was not countenanced at all. There was no consideration of the possible counterarguments that I have listed above. It is hard to avoid the conclusion that this was all part of a Twitter and change.org motivated activist movement.

The role of the council is primarily to allocate tax-payer funded grants to those projects that will deliver the greatest benefits. Their primary responsibility is to ensure robust allocation of funds through robust evaluation of projects and researchers. A broader interpretation of their role might include medium term incentives to attract the best applications.

So can they assure us that their assessments are indeed robust and unbiased? No, they really can't.

They should have audited fully all their assessment processes. Are they biased towards any identity group? Are they biased towards consensus positions, whether these be traditional or more modern? How large are these biases?

It is still not too late. They should start an audit right now!

But I am not optimistic, because I have found that those who claim systemic bias in institutional systems are often loathe to investigate where it is and how strong it is. Rather, they tend to dismiss the possibility of merit-based assessment at all and fall back on crude statistical arguments that any deviations from demographic benchmarks indicate bias. So grants should be 50-50 by sex.

The NHMRC should present evidence of any bias in (a) project quality assessment, (c) researcher track record and (c) grant success rates. They have the data on the quality assessment of every project, on a 1-7 scale. To what extent does this assessment depend on the sex and age of the assessor as well as how close the project is to their own research area? Do final success rates depend on the gender or age or subject matter of the applicant, over and above their track record and project quality assessment?

NHMRC are experts in this kind of data analytic assessment.

This failure to audit and improve the assessment processes is the opportunity cost of focusing on gender counts. But fine grain policy analysis doesn't generate outrage or change.org petitions. As things stand, can we be confident that the NHMRC are doing their best to fund and encourage the best research?