The RAE is dead - long live metrics

On Budget day, academics were stunned to discover that the deeply unpopular Research Assessment Exercise is to be killed off. Even the DfES hadn't been told. Lucy Hodges reports

It came as a bolt out of the blue. The day before the Budget, academics were doing what they normally do - fending off e-mails from students and putting the final touches to their next research assessment exercise (RAE) submission in 2008.

The day after the Budget they were reeling at the news: the RAE as they knew it was to be abolished by Treasury diktat. A new way of doling out research block grants to universities is to be introduced, maybe for the next RAE and certainly thereafter.

Although Gordon Brown gave no inkling of the announcement to anyone, the Treasury had clearly been planning it for a while because it had prepared a 60-page accompanying document, called "Science and innovation investment framework 2004-2014: next steps". This contained a whole chapter explaining why the unpopular RAE should be killed off.

Officials at the higher education funding councils and at the Department for Education and Skills were told only a few days before. "The DfES were completely blindsided by it," said one vice-chancellor. "They were very angry. It was very sudden. Most of us were caught unawares."

The RAE may sound like an arcane funding mechanism that allocates £1.45bn of research money to the universities, but its importance is much greater than that. Academics live and breathe the RAE because it ranks their departments against one another. Being in a 5* department (the top grade) confers huge status, not to mention money. And the money is certainly useful because it is not tied to anything; it can be spent on whatever the university chooses. That contrasts with funding that comes through the research councils for which academics have to bid.

Since the first research assessment exercise in 1989, university departments have lived and died by the RAE. Vice-chancellors have restructured their universities; for example, chemistry at Exeter was closed because it scored only a grade 4. One argument was that an expensive subject such as chemistry was not worth running unless you could get a 5 and attract a sizeable chunk of money.

Now, it seems, the rules of the game are to change. The Treasury wants to drop the elaborate system whereby academics rate one another's departments and introduce a metrics-based methodology whereby the block grant for research is simply an algorithm, say, of each university's total research income.

Their argument is that this would be much simpler and less burdensome. Moreover, the correlation between universities' income from the RAE and their total research income from other sources (research councils, charities, industry, the European Union and government departments) is strong, so why make people suffer the bureaucracy of the RAE for evermore?

Many university bosses believe it is too late to change the rules for the 2008 RAE. There's even talk of the Treasury being sued if it tries to change them. "People have invested millions of pounds in restructuring and trying to focus their organisations on research quality," says Steve Smith, vice-chancellor of Exeter University and chairman of the 94 group that represents the "small and beautiful" universities, such as Exeter and Sussex. "We are very happy to look at the Treasury's proposals. We welcome the chance to examine a metrics-based approach, but we don't think you can change the rules for 2008 this late in the game."

The Russell group of research-intensive universities disagrees, and is willing to contemplate a change for 2008. "There are a number of vice-chancellors who have had a feeling for some time that the RAE is past its sell-by date," says Professor Michael Sterling, the chairman of the Russell Group and the vice-chancellor of Birmingham University. "The beneficial effects are outweighed by the disadvantages."

Professor Sterling is one of a small group of vice-chancellors who have little time for the RAE and who have been agitating for reform for some time. To this end, they have been talking to Treasury officials.

Other members of this group are Professor Eric Thomas, vice-chancellor of Bristol University, and Professor David Vandelinde, vice-chancellor of Warwick.

According to Professor Thomas, the RAE is anachronistic and grossly distorts the planning cycle for universities. "I have always opposed a further RAE after 2001 and have publicly said that," he says. "It creates financial problems because institutions invest in the belief that there will be some RAE bonanza. In fact the pot is fixed, so their ambition will not happen. That leaves them with a substantial deficit on the expenditure account."

Other criticisms are that the RAE rewards research that was done years before because of the five-or-more-year cycle on which it operates, and that it is a ridiculous administrative burden. For the 2008 exercise, 900 academics are expected to spend hundreds of hours on the panels that review research in each subject, quite apart from the work that is carried out on the RAE in each university.

There are concerns about the "transfer market" that it creates for academics as well as the planning blight for 18 months up to the exercise and for the 18 months afterwards until the results are published. "The utter nonsense is that Bristol has already had staff employed full-time to address our submission since 2005, who are working for the next two-plus years on something that will not even inform funding until 2009," Thomas says. "There should definitely be no RAE after 2008. It should be confined to the historical dustbin so that in, 20 years' time, elderly academics can 'remember the RAE' in the way that old soldiers remember the war."

Professor Vandelinde is equally dismissive, though he argues that the cost of the exercise itself is not excessive, standing at about 3 per cent of the funds distributed. The problem is that the distribution of funds has increasingly, from each university's vantage point, become secondary to the ranking of research quality. "The money is important, but it is the league table position that really focuses and bends the minds of UK academics," Vandelinde says. "The RAE is no longer just an instrument for measuring. It has become the core reason for doing much of our academic planning, hiring and for our choice of research area. We are now driven by fear of not conforming to the RAE."

Another supporter of the Treasury's plan is Michael Driscoll, vice-chancellor of Middlesex University and chairman of Campaigning for Mainstream Universities, which represents the former polytechnics. He believes the 2008 RAE should be abandoned. Speaking in a personal capacity, he says he has always been against the RAE because it reinforces existing patterns of research funding.

The universities and subjects that benefit from it are those that have always attracted the lion's share of research funding to the detriment of new universities and emerging areas of interest in science.

"It encourages conservative behaviour because people don't want to take risks," Driscoll says. "It discriminates against interdisciplinary research and it doesn't encourage collaboration. It's a winner-takes-all system. Frankly, the amount we get is so small that it's worth gambling on change."

So, the Treasury has support for its plan from universities across the spectrum. But some vice-chancellors worry that they might lose out in a metrics-based system. Universities that focus heavily on the arts and humanities, such as the LSE, Sussex and Goldsmith's, are seen as particularly vulnerable. "Universities like Sussex would lose out because they do not get a lot of research income from things like medical charities," says Professor Alastair Smith, its vice-chancellor.

The Treasury acknowledges in its document that other methods might need to be explored for the arts and humanities. A spokesman for the Treasury said that the arts and humanities would not suffer as a result of the introduction of metrics. "We entirely recognise that they may have to be treated differently, and the Budget document is clear about that," said a spokesman.

What is likely to emerge is a hybrid system to replace the RAE; part metrics and part peer-review. Professor Adrian Smith, principal of Queen Mary, University of London, believes it should be possible to gather together maths, science, engineering and medical research and devise a metrics-based system for those areas. "Personally, I would be amazed if there were not enormous scope for replacing large parts of the RAE with metrics," he says. "You'd save a lot of money."

Vice-chancellors want to give the Treasury idea a fair wind, but there is real concern that a new methodology would simply throw up different problems and new ways to play the game. "We know how the RAE works," says Professor Malcolm Grant, provost of University College London. "We have some confidence that it will continue to recognise the highest quality research and that money will flow to the highest quality research."

Professor John Hood, Oxford's vice-chancellor, is also sceptical about a metrics-based approach. Speaking at a conference in Oxford last week, he said: "My colleagues would say they have yet to be persuaded that this is a better way than peer review."

Bahram Bekhradnia, director of the Higher Education Policy Institute and the man who used to be in charge of the RAE at the Higher Education Funding council, is particularly sceptical that metrics will be a better way to allocate money; it would provide disincentives to hire young staff and the success rate for research council grants would plummet as academics piled in to make grant applications. Also, there would be unexpected winners and losers, leading to volatility in the marketplace. The Treasury refutes such suggestions.

l.hodges@independent.co.uk