Thank you for your analysis, which all makes sense to me.
Regarding..... "The argument is that the pandemic led to an increase in the number of COVID-related papers published and cited in 2020 and 2021, resulting in a temporary increase in the impact factors of most journals. Now, as the world returns to normalcy, the impact factors will return to their pre-pandemic levels. While this theory sounds plausible, it doesn't explain why journals that didn't necessarily publish a significant amount of COVID-related content also experienced a sharp decline."
The pandemic changed many of our behaviors. During the pandemic, lots of outside-the-home options were unsafe, leading to many of us spending more time at home. Some taught online for a semester or two, which reduced commute time, and although students could contact us remotely, they tended to do so less often than when everybody was on campus. Such behavior changes boosted publishing activity during the pandemic, allowing for the lead times between research creation and its eventual publication. My home office walls now have soundproofing, enabling continued productivity from home. We learned how to work efficiently from home, which is a research boost even with the pandemic behind us.
These are minor points compared to the main points of your analysis.
Original Message:
Sent: 07-12-2023 01:18
From: David Hunt
Subject: Unraveling the Mystery of Impact-Factor Declines in Most of INFORMS Journals
David and Tinglong,
Thanks for solving the impact-factor decline mystery and for sharing it with the INFORMS membership.
It is surprising that Clarivate did not publish journal impact factors (JIFs) using both the prior and new methodologies (number of decimal places and use of early access content) to allow a transition to the new method. Having both the prior and new JIFs would allow the "COVID effect" and any journal specific impact changes to be better monitored.
From the perspective of INFORMS, we need to make sure any perceived declines in JIFs don't have negative consequences on continuing to attract high-quality articles or on the ability to increase library subscription rates to keep pace with inflation. However, I'm sure Chris Tang (VP Pubs), Matt Walls (Director Pubs) and the journal editors are on top of this.
Thanks again for providing the background necessary to understand the issue.
Dave
------------------------------
David Hunt
Vice President
Oliver Wyman
Princeton NJ
Original Message:
Sent: 07-11-2023 20:31
From: Tinglong Dai
Subject: Unraveling the Mystery of Impact-Factor Declines in Most of INFORMS Journals
Unraveling the Mystery of Impact-Factor Declines in Most of INFORMS Journals
David Simchi-Levi1 and Tinglong Dai2
1Institute for Data, Systems, and Society, Massachusetts Institute of Technology, email: dslevi@mit.edu
2Carey Business School, Johns Hopkins University, email: dai@jhu.edu
On June 28, 2023, Clarivate released its highly anticipated annual Journal Citation Reports, which included the 2022 impact factors for more than 20,000 journals published in 112 countries and 254 categories. Among the most notable changes, Clarivate began using a single decimal instead of three decimals to avoid "false precision." But an even more striking change is the decline in impact factors; our analysis shows that the decline could continue for many journals next year until impact factors stabilize in 2025.
The ripple effect of these declining impact factors was immediately felt within the INFORMS community. As shown in Table 1, of the 12 INFORMS journals with at least two years of impact factor history, 10 experienced declines, with the INFORMS Journal on Computing (from 3.288 to 2.1, a 36% drop) and Operations Research (from 3.924 to 2.7, a 31% drop) bearing the brunt of the decline. On average, the impact factors of INFORMS journals fell by nearly 12%.
Table 1. Change in INFORMS journals' impact factors. Note that the 2022 impact factors were released
in June 2023, and the 2021 impact factors were released in June 2022. Data Source: Clarivate.
The INFORMS community was not alone in this downward trend. According to one estimate, at least 90% of the top 300 chemistry journals also experienced a decline in their impact factors. A similar trend was observed among the top medical and engineering journals.
So, what's behind this widespread decline?
One theory relates to the "COVID effect" that Clarivate had previously attributed to the increase in citations in previous years. The argument is that the pandemic led to an increase in the number of COVID-related papers published and cited in 2020 and 2021, resulting in a temporary increase in the impact factors of most journals. Now, as the world returns to normalcy, the impact factors will return to their pre-pandemic levels.
While this theory sounds plausible, it doesn't explain why journals that didn't necessarily publish a significant amount of COVID-related content also experienced a sharp decline.
Another theory points to changes in the calculation of "Early Access" content (i.e., articles published online ahead of print). As Wiley noted in its 2021 pre-warning, Clarivate's "phased approach" may have caused a temporary spike in 2020 impact factors (published in 2021), followed by a return to the previous trend.
To understand this pre-warning, we need to briefly discuss how a journal's impact factor is determined by Clarivate's Journal Citation Reports. Let It denote the number of citable items (i.e., published articles) in year t, and Ct1,t2 the number of citations in year t1 to the citable items published in year t2. Then, the Year-timpact factor (released in year t+1), denoted by IFt, is calculated using the following formula:
For example, in the recently published impact factors, a journal's 2022 impact factor was calculated as the ratio of (1) the total number of citations in 2022 for the journal's 2021 and 2020 citable items to (2) the journal's total number of citable items published in 2021 and 2020.
The formula above has remained the same over the years, but the definitions of its denominator and numerator underwent a major adjustment in 2021, specifically for the calculations of the 2020 impact factors. Prior to 2021, the impact factor was calculated based on data from the volume year; that is, an item that is not associated with a printed issue does not count. However, starting with the 2020 impact factor (which was released in 2021), the calculation was based on publication year data. This means that for the 2020 impact factor, Early Access content was included in the numerator of the calculation. Under the phased roll-out approach, Early Access content would be included in both the numerator and denominator of the calculation as early as 2022 (for the 2021 impact factors).
This change in methodology raises a question: If the inclusion of Early Access content in the calculation was indeed a significant factor, why didn't we see a dramatic decrease in the 2021 impact factor?
To unravel this mystery, we delved into specific data points, focusing on how the impact factor of the journal Management Science was calculated in the past four years, as illustrated in Table 2.
· Let's start with the most recent impact factor, published in June 2023. The journal's 2022 impact factor was calculated based on 4,649 citations in 2022 to articles published in 2020 and 2021. Divided by the number of articles published in Management Science (hard copy and Advance Access) in 2020 and 2021 (862), the impact factor is 5.400.
· Going back to the previous year, the impact factor for 2021 was calculated using 3,666 citations in 2021 to articles published in 2019 and 2020. With 594 citations in those years, the resulting impact factor was 6.172.
· Going back further, the 2020 impact factor was based on 2,993 citations in 2020 to articles published in 2018 and 2019. Dividing this by the number of citable articles in those years (613) yields an impact factor of 4.883.
· Finally, the 2019 impact factor was calculated using 2,170 citations in 2019 to articles published in 2017 and 2018. With 552 articles cited in those years, the resulting impact factor was 3.931.
Table 2. Data used for calculating Management Science's impact factors between 2019 and 2022.
Data Sources: Clarivate; INFORMS.
A closer look at the third column of Table 2 reveals a significant increase in the number of cited articles in 2021 (568), almost double the number in previous years. At the same time, we can see from Table 2 that the number of articles published in print in 2021 is 382, which implies that 186 (= 568 - 382) Advance Access articles were included in the number of citable items. On the other hand, the 2021 Impact Factor was only based on Management Science articles published in print; that is, the numbers of articles published in print in 2020 and 2019 are consistent with the data in the third column of Table 2 for raw 2021. This observation suggests that the phased rollout of the new methodology was the key to understanding the sudden drop in impact factors.
While the "COVID effect" may have played a role, the changes to the Early Access calculation are more fundamental. It is abundantly clear from the Management Science data that the changes in both the numerator and denominator of the calculation occurred in 2023 (for the 2022 impact factors based on citable items in 2021 and 2020), not 2022 (for the 2021 impact factors based on citable items in 2020 and 2019). We performed the same analysis for all 12 INFORMS journals listed in Table 1 and confirmed an increase in the number of citable items published in 2021 due to the inclusion of Early Access articles in the denominator for the first time. This analysis suggests that the full impact of the change in impact factor calculations will be realized in the 2023 impact factors published in 2024. This is because, at that time, the analysis will include all print and Advance Access articles published in 2022 and 2021 in the denominator of the impact factor calculations.
To be clear, our findings do not contradict Clarivate's timeline. As stated in their June 2023 announcement, journals "onboarded 2020 and earlier" would experience full implementation of the new calculation method in 2023, while journals "onboarded 2021" would experience full implementation in 2024. Unfortunately, Clarivate does not publish a list of journals for each group. However, based on our analysis, we are confident that all 12 INFORMS journals were onboarded in 2021, so the full impact of the new method will be realized in 2024.
In conclusion, the data from Management Science shows that the dip in the impact factor is largely due to the change in methodology and how the change was rolled out over time. If this trend continues, we could expect to see the full impact of the methodology change in 2024 and a stabilization of the impact factors from 2025 onward. However, it's also important to note that external factors such as the pace of journal production, global events, and shifts in research focus can also have a significant impact on these metrics. Therefore, while we can make informed predictions, the future trajectory of impact factors remains somewhat uncertain.
------------------------------
Tinglong Dai
Professor
Johns Hopkins University
Baltimore MD
Original Message:
Sent: 07-11-2023 19:29
From: David Simchi-Levi
Subject: Unraveling the Mystery of Impact-Factor Declines in Most of INFORMS Journals
Hi everyone,
Tinglong Dai and I recently analyzed data to understand the drop in the 2022 Impact Factor for most of INFORMS journals and to predict what will likely to happen next year, see attached pdf.
Comments and suggestions are welcome.
David Simchi-Levi
Editor-in-Chief
Management Science
E-mail: mseic@mit.edu
------------------------------
David Simchi-Levi
Professor of Engineering Systems
Massachusetts Institute of Technology
Cambridge MA
------------------------------