Moving impact from 20% to 25% in REF

Increasing the weighting of impact.

The 2014 Research Excellence Framework (REF) had weightings based on awarding 65% for outputs, 20% for impact and 15% for environment. It was recently announced that in the 2021 REF the weighting of impact, and now relating only to the case studies, would be increased in 25%. This looks like quite a substantive change. To look at possible effeccts, this analysis looks at what would have happened to the 2014 results if the proposed 2021 weightings had been in place.

This calculation does not take into account any differences created by moving the narrative element to environment, or any differences in how submissions might have been re-written - it simply changes the 2014 weights from 65/20/15 to 60/25/15 whilst keeping the same results for outputs, impact and environment. (note that we do not have that breakdown for the smallest submissions, which are thereby excluded, leaving 1895 submissions being analysed).

In this first chart we show the within-subject rankings in 2014, 1st 2nd 3rd, etc., against what they would have been with the new weights. As is clear, there is a very strong correlation between the different rankings, although with a few points standing a little further from the mass, indicating potential changes for those submissions had the weightings been different.

Fig 1: 2014 REF rankings with 2021 weights

In a second chart we show how far that the rankings chamge. A result of 0 means no change, +1 means an improvement by one place, and -1 a worse position by one place. It should be clear that most differences would have been quite small. In fact, for 46% their league table ranking would have been unchanged, whilst 32% of submissions would have been only one position higher or lower. Even so, small changes of ranking do matter, and this could be expressed as 54% of submissions would have changed league table position.

Fig 2: Changes in league ranking with 2021 weights

So, a simple change of weightings would not have made much difference to a large proportion of submissions. However, a few submissions would have seen larger changes. One submission would have moved up by around ten places (English at Kingston Universty, which scored 100% at 4* for impact compared with 14.8% for outputs at that level). Three submissions would have dropped by 7 places, instances where the score for impacts were much lower than for outputs.

It is certainly clear that increasing the importance of impact raises the stakes for having stronger and weaker case studies. Had the 2021 weights applied in 2014, most submissions would have had a different league table place (using GPAs), though usually by only a position or two.