When a Measure Becomes a Target: Reassessing the Purpose of the Highly Cited Researchers List (2001–2024)
Objective
Investigate how the Clarivate Highly Cited Researchers (HCR) list in Mathematics has evolved from a neutral indicator of scholarly impact to a behavioral target, influenced by strategic academic practices. Drawing on the theoretical insights of Goodhart’s Law—when a measure becomes a target, it ceases to be a good measure (Goodhart 1975)—we will explore how the list reflects not just academic excellence but also the adaptive behaviors of researchers and institutions in response to incentive structures.
Research Questions
How has the composition of the HCR Mathematics list changed over time, and what patterns emerge in terms of institutional affiliations, geographic distribution, and publication strategies?
What behaviors and strategies have researchers and institutions adopted in response to the HCR list’s influence?
How do these behaviors and strategies alter the list’s original intent as a measure of scholarly impact? What does the list now measure?
Data Sources
- Clarivate HCR Lists (2001–2024): Available from 2014 onwards via Clarivate’s official archive. For 2001–2013, data is limited due to changes in list compilation and availability.
- Google Scholar Profiles: Publicly accessible profiles providing citation metrics, publication lists, and co-authorship networks.
- Supplementary Databases: Scopus, Web of Science, and institutional repositories to cross-verify publication and citation data.
Methodology
Data Collection
- Compile HCR Mathematics lists from 2014 to 2024 directly from Clarivate’s archives.
- For 2001–2013, reconstruct lists using archived web pages, institutional announcements, and bibliometric databases.
- Combine with data from Google Scholar profiles of the identified HCRs using webscraping.
Data Analysis
- Perform longitudinal analysis to identify trends in HCR listings, including frequency of repeat appearances and shifts in institutional representation.
- Compare Google Scholar metrics with HCR listing years to assess alignment and discrepancies.
- Conduct case studies on select researchers to explore strategic behaviors such as co-authorship patterns, publication venues, and citation practices.
Addressing Data Limitations
Pre-2014 HCR Data Scarcity
Clarivate’s official HCR archives begin in 2014. For earlier years, we will use:
- Archived versions of the HCR website via the Internet Archive’s Wayback Machine.
- Institutional press releases and academic CVs.
- Bibliometric analyses from prior studies that reference early HCR lists.
Temporal Misalignment with Google Scholar Data
Google Scholar profiles reflect current citation metrics, which may not accurately represent a researcher’s status at the time of their HCR listing. To mitigate this:
- Use the R package
scholarto retrieve historical citation data, acknowledging its limitation to the past nine years. - Supplement with data from Scopus and Web of Science, which offer more extensive historical records.
- Incorporate qualitative data from CVs and institutional records to contextualize quantitative findings.
Literature Review
Goodhart’s Law has inspired an extensive literature on how metrics, once targeted, begin to reflect strategic adaptation rather than intrinsic quality (Campbell 1979; Manheim and Garrabrant 2018).
Theoretical Works
The principle has been foundational in policy and academia (Strathern 1997; Chrystal and Mizen 2003; Thomas and Uminsky 2022; Muller 2018).
- Goodhart introduced the principle in the context of monetary policy, showing how financial targets lose predictive power when used for regulation (Goodhart 1975).
- Campbell presented a similar insight in the social sciences, warning that any indicator used for decision-making is vulnerable to distortion (Campbell 1979).
- Strathern applied this logic to academic audits in the UK, noting that performance metrics can reshape institutional behavior (Strathern 1997).
- Chrystal and Mizen analyzed its implications in monetary theory and policymaking (Chrystal and Mizen 2003).
- Manheim and Garrabrant categorized variants of Goodhart’s Law—regressional, causal, extremal, and adversarial—relevant for understanding metric failure in AI, policy, and research evaluation (Manheim and Garrabrant 2018).
- Thomas and Uminsky highlighted the risks of specification gaming in AI, drawing directly from Goodhart’s framework (Thomas and Uminsky 2022).
- Muller provided a cross-sector critique of metric fixation, including academic publishing (Muller 2018).
Empirical Applications
Empirical evidence from academic publishing (Butler 2003; Fire and Guestrin 2019; Smaldino and McElreath 2016), university rankings (Espeland and Sauder 2007), and healthcare systems (Bevan and Hood 2006) consistently supports the distorting effects of metric-based evaluation.
Recent work on citation gaming and the HCR list itself (Baccini, De Nicolao, and Petrovich 2019; Klein and Kranke 2023; Adler, Ewing, and Taylor 2008) confirms that once these indicators become goals, their meaning shifts.
- Butler demonstrated that Australia’s metric-based funding policies led to increased publication counts but decreased citation impact (Butler 2003).
- Espeland and Sauder studied U.S. law school rankings and documented how metrics reshaped institutional priorities (Espeland and Sauder 2007).
- Bevan and Hood explored gaming of health service targets in the UK NHS (Bevan and Hood 2006).
- Fire and Guestrin analyzed large-scale bibliometric data and found evidence that academic publishing behavior increasingly reflects metric optimization rather than research quality (Fire and Guestrin 2019).
- Smaldino and McElreath used simulations to show how metric-driven incentives can favor poor scientific practices (Smaldino and McElreath 2016).
- Baccini et al. documented how Italian academics changed their citation patterns in response to metric-based promotion rules (Baccini, De Nicolao, and Petrovich 2019).
- Klein and Kranke examined transparency concerns in the HCR list, noting evidence of gaming and Clarivate’s exclusion of some researchers suspected of unethical citation behavior (Klein and Kranke 2023).
- Adler et al. from the International Mathematical Union emphasized the limitations of citation metrics in mathematics and warned of perverse incentives created by evaluation systems (Adler, Ewing, and Taylor 2008).
Expected Contributions
- Provide insights into how the HCR list influences academic behavior and institutional strategies.
- Highlight the transformation of bibliometric indicators from passive measures to active targets in line with Goodhart’s Law.
- Offer recommendations for developing more robust and less gameable metrics for assessing scholarly impact.
- Lay the groundwork for cross-disciplinary comparisons by developing a methodological template that can be applied to other academic fields.
Conclusion
By critically examining the HCR list in Mathematics, this study aims to shed light on the broader implications of metric-driven evaluations in academia. Understanding the behavioral responses to such metrics is crucial for developing fair and effective assessment tools that truly reflect scholarly contributions. This framework also has potential applications beyond Mathematics, offering insights for other disciplines—including Economics and the natural sciences—facing similar pressures from metric-based evaluations.