Do university rankings add up?

What makes a university great?

You might think this question would be easy to answer. But you would be wrong. Not wanting to make value judgements, university insiders are reluctant to discuss the issue of quality. They have left this to outsiders to decide. Not surprisingly, many have stepped up to the challenge. Today, we have a plethora of rankings and league tables, each purporting to identify the best universities.  

It all started decades ago when the magazine U.S. News & World Report published its first annual ranking of American Universities. The editors picked a few measures such as reputation, selectivity (how hard it iwa to gain admission) and books in the library. They aggregated these measures into an overall "quality" score and ranked universities from highest to lowest. Those with the highest scores were, by definition, the best universities, and the highest-scoring institutions were "great."

U.S. News & World Report found its yearly academic ranking issue popular with parents, students, and donors. The rankings provided what appeared to be a transparent way of comparing universities and identifying quality. On the other hand, universities greeted the yearly rankings with much wringing of hands and gnashing of teeth. They particularly objected to their arbitrary nature. By selecting a particular set of indicators and assigning each one a weight, the authors impose their definition of quality on the institutions ranked.  

It took only a short time for other magazines and newspapers to follow the lead of U.S. News & World Report. Today, more than a dozen rating systems and league tables are published each year. In addition, academics have gotten into the act, producing their own league tables. Rankings are not limited to whole institutions, there are also rankings for business, law, and medical schools and even for individual subjects. Originally, rankings were national, but the global rankings carry the most weight outside the USA. The two most often cited are the Academic Ranking of World Universities from Shanghai's Jiao Tong University, first released in 2003, and the World University Rankings from the Times Higher Education Supplement of Britain (THES), first released in November 2004. The World University Rankings focus entirely on research, while the Times Higher uses a variety of factors, such as the number of overseas students and even an opinion poll. At one time or another, rankings have included student/staff ratios, the number of books in the library and other "inputs." Outputs, such as the number of first-class honours awarded or the employability of graduates, are also used by some rankings.

There is no doubt that rankings affect the behaviour of potential applicants. When universities fall in the rankings, student applications (particularly from international students) also fall. Perhaps more worrying, rankings affect the behaviour of institutions. For example, suppose high student entry scores contributed to rankings. In that case, universities may award all their scholarships to high-scoring students from wealthy families instead of students who need the money. Similarly, if awarding more first-class honours raises a university's rankings, it may decide to inflate its marks. Institutions in the middle of the rankings may divert large amounts of money from teaching and research to public relations to rise above the perceived mediocrity of the middle rank. 

According to Harvard Medical School Dean George Q. Daley,  

Rankings create perverse incentives for institutions to report misleading or inaccurate data, set policies to boost rankings rather than nobler objectives or divert financial aid from students with financial need to high-scoring students with means in order to maximize ranking criteria.

To avoid having their behaviour influenced by newspapers and magazines, Harvard and other prestigious American law and medical schools have refused to provide information to U.S. News & World Report. Some universities have dropped out of the ranking system altogether. So far, only a minority of universities have joined the rebellion, but there are already signs of change. Instead of trying to summarise a university in a single number, some rankings are multi-dimensional. Using several dimensions seems closer to reality. Some students prefer a university with good sporting facilities and extensive offerings in the fine arts. Others may be looking for night classes and low fees, while still others seek the prestige of a major research institution. Students need useful information to make an informed choice of where to study. What they need to know is not which university is the best but which is best for them.