In this blog post, we will examine the problems with scientific research evaluation methods centered on journal citation indices and consider whether they accurately reflect the true value of research.
In the scientific community, the “journal citation index” is used to evaluate the quality of researchers and the importance of their papers. However, criticism of this practice has grown increasingly vocal in recent years. In May last year, over 100 scientists and researchers gathered in San Francisco to publish the “San Francisco Declaration on Research Assessment.” Currently, tens of thousands of researchers have joined this movement. They point out several fatal flaws in the evaluation method based on journal citation indices. This article will examine the criticisms of journal citation indices in detail and argue that they cannot serve as a standard for evaluating scientific research.
A journal citation index is a numerical measure of a journal’s influence. It was originally designed as a metric for librarians. This was because libraries needed to evaluate the relative importance of various journals to determine which ones to subscribe to on a regular basis. The method for calculating the journal citation index is straightforward. For example, if there is an academic journal titled “Science and Technology,” and over the past two years, 20 papers were published in “Science and Technology,” and the total number of citations for these papers is 200, then the citation index for “Science and Technology” is 200/20, or 10.
In other words, a citation index of 10 means that the average number of citations for papers published in “Science and Technology” over the past two years is 10. In this way, the journal citation index is a numerical value that quantitatively expresses the importance of an academic journal. However, the problem is that the journal citation index is directly applied to evaluate the importance of individual papers.
The value of a journal citation index directly translates into the influence score of the papers published in that journal. For example, if there are two journals, “Science and Technology” with a citation index of 10 and “Monthly Engineering” with a citation index of 90, all papers published in “Science and Technology” are evaluated as 10 points, while all papers in “Monthly Engineering” are evaluated as 90 points. These scores are naturally applied to the authors of the papers. Therefore, researcher A, who published a paper in “Science and Technology,” receives 10 points, while researcher B, who published a paper in “Monthly Engineering,” receives 90 points. Regardless of the researchers’ qualifications or the originality of their papers, a 80-point gap emerges between A and B.
The first problem with journal citation indices is the “statistical trap.” Since a journal citation index is an average value, even papers published in the same academic journal can have vastly different citation counts. The influence of an individual paper does not simply correlate with the citation index of the journal in which it was published. For example, Paper A may have a citation index of 10, but its actual citation count could be in the tens or hundreds. However, the average citation count for other papers in “Science and Technology” may be very low, thereby lowering the overall average to 10. On the other hand, B’s paper may have only been cited a few times, but the high citation counts of other papers in “Monthly Engineering” could have boosted its citation index to 90, resulting in a windfall benefit. In such a situation, it is meaningless to consider the journal citation index when comparing the research of A and B. Instead, it is more reasonable to compare the number of citations for each paper of A and B and conclude that A’s paper has a much greater influence. According to the San Francisco Declaration, approximately 25% of papers in an academic journal account for 90% of the total citations. In other words, if journal citation indices are applied directly to individual papers, there will inevitably be papers that suffer losses and papers that enjoy windfall gains.
Second, research evaluation based on journal citation indices does not reflect the unique characteristics of each academic field. For example, in medicine and biology, when a new theory is proposed, numerous experiments are conducted to verify its validity. Since clinical experiments related to a single paper are repeatedly conducted through follow-up studies, the citation index of biology and medicine journals is inevitably higher than that of other natural science or engineering fields. On the other hand, in pure mathematics, a single paper is typically self-contained and does not require follow-up research or experiments. Therefore, mathematics papers have relatively fewer citations, and the journal citation index of pure mathematics journals is also inevitably lower. Additionally, in very specialized fields with few researchers, the number of citations is relatively low. Conversely, journals in large fields with active research and many researchers tend to have a higher number of citations. As such, the evaluation method based on journal citation indices has the limitation of not considering the fundamental differences inherent in each academic discipline.
The third issue is that the journal citation index leads to a concentration of papers in a few popular journals. Researchers who are about to publish their papers naturally want their work to be published in world-renowned journals such as “Cell,” “Nature,” and “Science.” This is because these journals have the highest journal citation indices. If such concentration on a few journals becomes excessive, publishing papers may become more important to researchers than the research itself. The practice of recognizing only papers published in top-tier journals has become a global phenomenon. If the practice of valuing visible results over diligent research processes continues, it could distort the very essence of science. Additionally, some journals encourage “self-citation,” which involves citing papers published in their own journals, to boost their own citation indices. Such evaluations based on journal citation indices foster unreasonable and unethical competition.
Of course, journal citation indices are not without their merits. Their widespread use stems from the advantage of enabling quick and convenient evaluation of researchers. Journal editors act as “expert evaluators,” swiftly categorizing important and noteworthy research amid the overwhelming volume of scholarly output. In today’s rapidly changing and expanding scientific landscape, this is an undeniable benefit.
However, as we have seen earlier, this convenience comes with serious drawbacks. The first is the statistical trap, where journal citation indices may not accurately reflect the actual number of citations a paper receives. The second is that citation indices do not account for the unique characteristics of individual disciplines. Some fields have pointed out that the number of citations is often unrelated to influence.
The final issue is that this evaluation method causes wasteful competition in the scientific community by creating a monopoly structure of a few well-known journals. To overcome the above issues and achieve proper scientific development, the scientific community is currently seeking new evaluation criteria.
The simplest solution is to reflect the number of citations received by individual researchers in the evaluation. This serves as an alternative to address the pitfalls of statistical analysis. Another approach involves using a correction index that accounts for the unique characteristics of each field. By dividing the journal index of a paper by the average number of citations received by the top 20% of journals in its field, a correction index can be derived, enabling normalization that reflects the unique characteristics of each academic field. Activating peer review to develop qualitative evaluation criteria rather than relying on quantitative metrics such as journal indices or citation counts would be a more fundamental solution. What is currently required in the scientific community is self-reflection and communication to devise reasonable evaluation methods.