The hidden limitations of citation-based ranking metrics

Why citations are not neutral counters

Citations are often presented as the objective backbone of university rankings. Unlike reputation surveys, which rely on subjective opinions, citation counts appear to offer hard numerical evidence of research impact. A university whose faculty are cited frequently is presumed to be producing important, influential work. This logic has made bibliometric indicators central to most major ranking systems, with some assigning up to 30 or 40 percent of the total score to citation-related metrics.

The reality is more complicated. Citation practices vary enormously across academic disciplines. In molecular biology, a paper might accumulate dozens of citations within its first year of publication. In mathematics or humanities, a groundbreaking paper might take five or ten years to reach the same citation count, simply because the research community is smaller and publication cycles are longer. When a ranking normalizes citation counts without fully accounting for these field differences, it systematically favors certain disciplines over others. An institution strong in life sciences will appear far more impactful than an equally excellent institution focused on classics or philosophy.

Language and database coverage biases

Language bias compounds the field-specific problem. The major citation databases, such as Scopus and Web of Science, disproportionately index English-language journals. Research published in Chinese, Spanish, Arabic, French, or other languages is underrepresented, regardless of its quality or local impact. A university in a non-English-speaking country may produce significant research that serves its national and regional community but scores low on citation metrics simply because that research is not captured by the databases that rankings rely on.

Similarly, the type of research output matters. Citation databases are built around journal articles and conference proceedings. Books, book chapters, policy reports, creative works, and software contributions—all legitimate forms of academic output—are often excluded or undervalued. This disadvantages disciplines where book publication is the primary mode of scholarly communication, such as history, literary studies, and parts of the social sciences.

Self-citation and citation clubs

The integrity of citation metrics also faces challenges from strategic behavior. Authors can and do cite their own previous work, sometimes legitimately and sometimes to inflate their citation counts. While most ranking methodologies attempt to control for excessive self-citation, the boundaries are not always clear. A research group that cites its own members' work across multiple papers creates a citation network that looks like genuine impact but may simply reflect a closed circle of mutual citation.

More subtle is the phenomenon of citation clubs, where groups of researchers in a field routinely cite each other's work, boosting everyone's metrics. This behavior is particularly hard to detect algorithmically because it looks like normal scholarly exchange. The result is that some fields develop inflated citation cultures while others remain more restrained, further distorting cross-field comparisons. A high citation count in one discipline may reflect a dense, self-referential citation network rather than genuine influence beyond a small group of researchers.

Using citation metrics thoughtfully

The practical lesson is that citation metrics should be used with careful attention to context. When evaluating a university's research strength, first identify whether the ranking reports field-normalized citation metrics. Some major ranking systems now adjust citation counts based on the average citation rate in each field, which helps to reduce but does not eliminate disciplinary bias. Second, look at multiple citation indicators rather than a single number. Total citations, citations per faculty, and the proportion of highly cited papers each tell a different part of the story.

Third, consider the university's disciplinary mix. A university with a large medical school will naturally generate more citations than one focused on the arts, even if both are excellent in their respective fields. If you are evaluating a specific department or program, look for field-specific rankings rather than relying on institution-wide citation metrics. Finally, supplement citation data with qualitative information: read departmental profiles, look at recent faculty publications, and, if possible, speak to researchers in the field about which institutions are producing the most exciting work. Citations are a useful signal, but they are not a substitute for informed judgment.

Ultimately, the best way to use citation data is as a directional indicator rather than a precise measurement. A university with consistently high citation counts across multiple databases and time periods is likely producing research that resonates with the academic community. But a university with modest citation metrics may still be doing excellent work—especially in applied fields, the humanities, or areas where research impact takes forms that citation counts cannot track. Let citations inform your curiosity rather than settle your judgment.