Measuring research impact beyond citations

The narrow definition of impact in rankings

When university rankings talk about research impact, they almost always mean citation impact—how often a university's published research is cited by other researchers. This is a valid measure of academic influence, but it captures only one dimension of what research can achieve. Universities also produce research that shapes public policy, informs professional practice, leads to new technologies and businesses, enriches cultural life, and addresses social challenges. These broader forms of impact are almost entirely invisible to ranking methodologies.

The gap between what rankings measure and what universities contribute is growing as funders, governments, and the public increasingly demand evidence of impact beyond academia. Research councils now routinely require grant applicants to describe the potential societal and economic benefits of their work. Universities invest in knowledge transfer offices, public engagement programs, and industry partnerships. Yet none of this activity registers in most ranking systems, which remain anchored to bibliometric databases that were designed for a different purpose.

Alternative impact indicators and their challenges

Several alternative indicators attempt to capture broader research impact. Altmetrics track mentions of research in news media, policy documents, social media, and online reference tools such as Wikipedia. These indicators can reveal which research is reaching audiences beyond academia, but they also introduce new biases. Research that is controversial or sensational may attract more attention than research that is solid but unexciting. English-language research dominates altmetrics just as it dominates traditional citation counts.

Patent citations are another alternative, capturing the influence of academic research on technological innovation. A paper cited by a patent has demonstrably contributed to an invention, which is a different kind of impact from an academic citation. However, patent citation data is uneven across disciplines. Biomedical and engineering research is heavily represented, while social sciences and humanities research is not. Using patent citations as a ranking indicator would systematically disadvantage universities strong in non-patenting disciplines.

Case study approaches to impact assessment

Some national assessment systems have moved toward case study approaches, where universities submit narratives describing specific examples of research impact, supported by evidence. The United Kingdom's Research Excellence Framework includes an impact component based on such case studies, which are evaluated by expert panels that include non-academic users of research. This approach captures the richness and diversity of research impact in a way that quantitative indicators cannot.

Case study approaches are not without limitations. They are resource-intensive, difficult to scale globally, and susceptible to narrative inflation, where universities present their impact in the most favorable light. But they represent a genuine attempt to capture what quantitative indicators miss. At present, no major global ranking incorporates case study evidence, but the increasing pressure on universities to demonstrate impact beyond academia may eventually push ranking publishers in this direction.

Practical guidance for prospective students and researchers

If you are a prospective student, broad research impact metrics in rankings may be less relevant to you than you think. Your experience as an undergraduate or taught postgraduate student is shaped more by teaching quality, course design, and student support than by the societal impact of the university's research. The research environment becomes more important if you are applying for a research degree, and even then, the research culture within your specific department matters more than university-wide metrics.

If you are using rankings to evaluate research environments, supplement citation-based indicators with your own investigation. Look for evidence of industry partnerships, policy engagement, public outreach, and knowledge transfer activities. Universities that are effective at translating research into broader impact often highlight this work on their websites. Read case studies and impact stories. Contact potential supervisors and ask about the broader significance of their research. These qualitative signals can provide a richer picture than any ranking indicator of how a university's research connects to the world beyond academia.

A thoughtful approach to research impact acknowledges that different kinds of impact matter to different audiences. A prospective PhD student should care about academic impact, as measured by citations and peer recognition, because that is the currency of an academic career. A prospective undergraduate may care more about whether the university's research feeds into the curriculum, creates opportunities for student involvement, and builds connections with employers. Impact is multidimensional, and the best measure depends on what you hope to gain from your education.

The growing sophistication of impact measurement will likely reshape ranking methodologies in the years ahead. For now, the prudent approach is to treat citation-based impact scores as one narrow window into research influence, and to seek out the broader story of how a university's research matters through its own impact case studies and engagement records.