Do local citation patterns support use of the impact factor for collection development?

Impact Factors and Other Measures of a Journal's Influence

The role of impact factor in evaluating the value of scientific publications is critically reviewed.
\nOther possible criteria for evaluation are assessed and proposed.

Background:In the early 1960s, Eugene Garfield and Irving Sher created the journal impact factor to help select journals for the Science Citation Index (SCI). Today it has become a widespread subject of controversy even for Garfield, the man who created it who is quoted saying “Impact Factor is not a perfect tool to measure the quality of articles but there is nothing better and it has the advantage of already being in existence and is, therefore, a good technique for scientific evaluation”. The use of the term “impact factor” has gradually evolved, especially in Europe, to include both journal and author impact. This ambiguity often causes problems. It is one thing to use impact factors to compare journals and quite another to use them to compare authors. Journal impact factors generally involve relatively large populations of articles and citations. Individual authors, on average, produce much smaller numbers of articles.Objectives:Impact factor, an index based on the frequency with which a journal's articles are cited in scientific publications, is a putative marker of journal quality. However, empiric studies on impact factor's validity as an indicator of quality are lacking. The authors try to evaluate and highlight the validity of Impact Factors and its significance as a tool of assessment for scientific publications.Methods:Analysis of the several reports in literature and from their own point of view.Conclusion:A journal's impact factor is based on 2 elements: the numerator, which is the number of citations in the current year to any items published in a journal in the previous 2 years, and the denominator, which is the number of substantive articles (source items) published in the same 2 years. The impact factor could just as easily be based on the previous year's articles alone, which would give an even greater weight to rapidly changing fields.

We assessed the relationship between the size of the 39 Journal Citation Reports (JCR) medical categories and impact factor (IF) of journals in these categories, and the implications that it might have for emergency medicine (EM) journals. Using the 2010 JCR database, we calculated the mean IF, 5-year IF (5y-IF), Eigenfactor (EF), and Article Influence (AI) scores including all journals for each category. We also calculated a 'weighted IF' for all journals by dividing each journal IF by the mean IF of its category. We ranked EM journals according to IF and 'weighted IF' into all the journals included in the 39 categories. We assessed the relationship between category size and bibliometric scores by linear regression. Category size varied from 252 journals (Pharmacology and Pharmacy) to 14 (Primary Healthcare), EM category occupying the 36th position (23 journals). The mean IF of EM category ranked in 34th position, 5-yIF in 32nd, EF in 34th, and AI in 34th position. Category size had a direct and significant association with mean IF, 5y-IF, and AI but not with mean EF. When the EM journals were ranked among all the journals according to their IF, only two (9%) were placed into the first quartile and raised up to eight (35%) when 'weighted IF' was considered. There is a negative relationship between JCR size category and IF achieved by the journals. This places EM journals at a clear disadvantage because they represent one of the smallest clinical medical research disciplines.

Blockbuster effect of COVID-19 on the impact factor of infectious disease journals

Misuse of Journal Impact Factors in Scientific Assessment

The journal impact factor (JIF) is a measure that provides a ratio of citations to a journal in a given year to the citable items in the prior two years. The JIF has become a symbol and measure of a journal's prestige in the modern world of peer-reviewed publications. The difficulty and importance for a peer-reviewed journal to obtain an impact factor has been highlighted in numerous medical journals. Irving H. Sher and Eugene Garfield were the first to create the concept of the impact factor of a peer-reviewed journal. According to Eugene Garfield (2006), the JIF is based on two elements: the numerator, which is the number of citations in the current year to items published in the previous two years, and the denominator, which is the number of substantive articles and reviews published in the same two years. Scientists around the world may notice that as of 2024, the JIF is calculated not only by Clarivate, but also by exaly. Clarivate is known for being the company that calculates the JIF, using data from its Web of Science product family, that also includes services/applications such as Publons, etc. In this editorial, we will explain in more detail why the JIF is important both for the peer-reviewed journals published in Ukraine and for Ukrainian authors. Also, let's also explain how both the impact factor calculated by Web of Science and the impact factor calculated by exaly can be useful. The article presents the experience of the Journal of Diagnostics and Treatment of Oral and Maxillofacial Pathology in assigning it an impact factor calculated by the exaly. The advantages for Ukrainian authors of publishing articles in the journals with an impact factor are presented. According to the Order of the Ministry of Healthcare of Ukraine of February 22, 2019, No. 446, publication of an article or review in a journal with an impact factor allows an oral and maxillofacial surgeon to be awarded 20 or 30 continuing professional development (CPD) points. The CPD points obtained are necessary for doctors to the annual review of their personal educational portfolio with CPD points, certification, confirmation or obtaining a new doctor’s category. In summary, the peer-reviewed journal that has received an impact factor automatically becomes a kind of provider of CPD activities for healthcare professionals, for which CPD points are awarded.

The Institute for Scientific Information (ISI) publishes annually listings of impact factors of scientific journals, based upon data extracted from the Science Citation Index (SCI). The impact factor of a journal is defined as the average number of citations given in a specific year to documents published in that journal in the two preceding years, divided by the number of “citable” documents published in that journal in those 2 years. This article presents evidence that for a considerable number of journals the values of the impact factors published in ISI's Journal Citation Reports (JCR) are inaccurate, particularly for several journals having a high impact factor. The inaccuracies are due to an inappropriate definition of citable documents. Document types not defined by ISI as citable (particularly letters and editorials) are actually cited and do contribute to the citation counts of a journal. We present empirical data in order to assess the degree of inaccuracy due to this phenomenon. For several journals the results are striking. We propose to calculate for a journal impact factors per type of document rather than one single impact factor as given currently in the JCR. © 1995 John Wiley & Sons, Inc.

Hard Impact: Journal Impact Factor and JMT

It always begins innocently enough! In the middle of the 19th century, mining and earthmoving were increasingly important enterprises of the industrial revolution. To remove rock and to open mine shafts, an explosive was needed, but nitroglycerine was too unstable for practical use. The Swedish scientist/inventor Alfred Nobel discovered that mixing nitroglycerine with the diatomaceous earth kieselguhr produced a stable explosive product he patented as dynamite, which was quickly adopted by the mining and construction industries. In the early 20th century, the Italian physicist Enrico Fermi, while attempting to understand the structure of atomic nuclei, discovered that nuclei bombarded by neutrons would split and release large amounts of energy. As others have employed these discoveries, both dynamite and nuclear fission have had destructive effects on society that were initially unimaginable by their discoverers. It was only a quarter century after the first nuclear fission bombs that Eugene Garfield, a library scientist and structural linguist from the University of Pennsylvania, discovered a metric that could be used to select journals for inclusion in his new publication Genetics Citation Index (the forerunner of Science Citation Index, which was subsequently commercialized by Garfield’s company Institute for Scientific Information). This metric for journals was named “impact factor” and was to be calculated “based on 2 elements: the numerator, which is the number of citations in the current year to any items published in a journal in the previous 2 years, and the denominator, which is the number of substantive articles (source items) published in the same 2 years.” 1,2 Thus, although the journal impact factor was born innocently enough, just like the examples involving Nobel and Fermi, Garfield’s impact factor is now being used by others in ways that threaten to destroy scientific inquiry as we know it. 3,4 For much of human history (about 200,000 generations), scientists were few in number, often worked in relative isolation, and only communicated findings to close

The decision of which journals researchers submit their manuscripts to is based on a range of factors, including the quality of reviews, acceptance probability, turnaround time, target audience and fit, but an increasing one is the journal's reputation or perceived ‘quality’ as most commonly currently quantified by its impact factor.1 In an earlier editorial, it was argued that the impact factor, although not without flaws, provides a reasonable ball park assessment of the journal's impact or usefulness.2 It was, however, suggested that a small array of metrics would be more appropriate and could include a second citation metric that included citations to textbooks (are papers from the journal being used to provide evidence for undergraduate and postgraduate texts?), a download metric (how many people are reading the articles?) and a journal's acceptance rate.2 Because of its recognised flaws, new citation metrics are being introduced to rival the impact factor3-7 and these new metrics have been highlighted by their use in recent international Government research assessments.7 For example, Elsevier's Scopus database, which provides the citation metrics SCImago Journal Ranking Indicator and CiteScore, has been used in a number of national assessment exercises, including the UK Research Excellence Framework (2014), The Excellence in Research for Australia (ERA) assessments (2010, 2012, 2015), the Italian Abilitazione Scientifica Nazionale's researcher assessment (2012, 2013) and the Portuguese Fundação para a Ciência e aTecnologia (FCT) assessment (2013/14).7 In light of these developments, it is timely to describe the various citation metrics and consider what is the most appropriate metric(s) for a clinical journal such as Ophthalmic & Physiological Optics (OPO). The most well-known, currently used citation metric is the Thomson-Reuters (now Clarivate Analytics) Institute of Scientific Information (ISI)'s 2-year journal impact factor.2, 8-10 The 2016 impact factor was calculated by the number of citations to the journal's articles published in 2014 and 2015, divided by the number of ‘papers’ published in the journal during those years. Only citations from journals that are included in Clarivate's Journal Citation Reports (JCR) on the online subscription-based service Web of Science (previously Web of Knowledge) count. This severely limits what is counted as a citation, although it could be argued that this keeps the quality of citations high. The denominator includes research papers (original articles, review papers, etc.) and does not include editorials, letters and news items, but JCR citations to all articles count (including those to editorials and letters). Case reports appear to be judged on a journal-by-journal basis by Web of Science, sometimes they are included in the denominator (and as they are rarely cited, this would lower impact factors) and sometimes not. This lack of transparency is also a flaw in the use of the impact factor and it is difficult to predict impact factors as it is not easy to forecast which documents will be included in calculations and which will not. OPO has an impact factor of 2.30 with 290 citations in 2014 and 2015 to 126 research papers (290/126 = 2.30). Web of Science also provides several other citation metrics, including an immediacy index, a 5-year impact factor and an Eigenfactor score (see Table 1).3 The 2016 immediacy index was determined by the number of citations in 2016 from papers published in 2016 (for OPO, 57/60 = 0.95) and is heavily dependent on papers published earlier in the year (e.g. the January OPO issue) as the papers published in the later issues (e.g. the November OPO issue) are highly unlikely to be cited in that year, given the lag in time from submission to publication of any papers that could cite a November publication. The immediacy index is sometimes considered an indicator for the following year's impact factor. This also highlights the fact that the 2-year window for impact factors is less (1.5 years?) for papers published in the November and December issues due to this time lag between submission and publication of citing papers. This is another reason why the 2-year impact factor is flawed. The 2016 5-year impact factor was determined by the number of citations in 2016 from papers published in 2011, 2012, 2013, 2014 and 2015 (for OPO, 777/323 = 2.41). It is similar to the 2-year impact factor in that includes citations to editorials and letters in the numerator, but does not count them in the denominator. Web of Science also includes an indicator of the longevity of citations from journals, the journal half-life. This indicates the median age of papers cited in the given year, so that OPO's 2016 cited half-life of 9.9 years, indicates that half of the citations to OPO in 2016 were from papers 9.9 years old or younger. This citation metric is rarely discussed or used, but is a useful indicator of the longevity of papers in a journal. For example, our website includes a list of OPO's classic papers (http://onlinelibrary.wiley.com/journal/10.1111/(ISSN)1475-1313/homepage/classic_papers.htm) that remain relevant and still cited despite being up to 25 years old. The Eigenfactor is produced for Web of Science by Jevin West and Carl Bergstrom at the University of Washington, www.eigenfactor.com).3 It provides 5-year citation data and weights the prestige of citations using an algorithm similar to Google's PageRank, so that citations from high ranking journals such as Nature and Science count more than lower ranking journals. It does not include journal self-citations (e.g. Journal of Cataract & Refractive Surgery citations in Journal of Cataract & Refractive Surgery papers; author self-citations in different journals are counted), which works against highly specialised journals. However, its algorithm attempts to adjust for citation differences between different research fields.3 Unlike impact factors, eigenfactors are based on a total score of citations and are not divided by the number of papers published. This means that if two journals have the same impact factor, but one publishes 1000 papers per year and the other 500 papers, the former will have twice the eigenfactor. This may be partly because the eigenfactor is provided as a reference for libraries, in that journals can be ranked using a ‘Cost-Effectiveness Search’ based on the citation value per dollar that they provide. From the perspective of libraries, the 1000 paper volume journal provides twice as many articles for their readers compared to a 500 paper journal of equal impact factor. SCImago and CiteScore are the principal citation metrics developed within Elsevier's online subscription-based service Scopus to rival impact factors provided by Clarivate Analytics’ Web of Science.6 SCImago is the most well-established of the Scopus metrics. It is based on 3-year citation data, so that the 2016 SCImago Journal Rank Indicator (JRI) is based on citations in 2016 from papers published in 2013, 2014 and 2015.11 Like Eigenfactors, it weights the prestige of citations using an algorithm similar to Google's PageRank. It also differs from impact factors in that editorials and letters are included in the denominator. This means that journals which include a large number of editorials and letters, such as the Nature and Lancet series of journals, tend to have significantly lower SCImago JRIs (and CiteScores) than would be expected from their impact factors.5 CiteScore is a more recently introduced metric (December 2016) and is similar to SCImago, but differs in that it doesn't weight citations (Table 1). Both SCImago and CiteScore consider ‘documents’ beyond just journal articles, such as books and conference proceedings. Indeed, the Scopus database includes far more (they claim ‘double’) journals than Web of Science. For example, Web of Science includes four optometry journals: Ophthalmic & Physiological Optics, Optometry & Vision Science, Contract Lens & Anterior Eye and Clinical & Experimental Optometry. Scopus includes these four journals, plus the Journal of Optometry and Clinical & Refractive Optometry and Scopus includes a subcategory of optometry journals11 unlike Web of Science where Optometry journals are considered within the Ophthalmology subcategory or Google Scholar which has a joint Ophthalmology and Optometry subcategory.12 Google Scholar provides two citation metrics, the 5-year h-index and 5-year median h-index.12 These h-indices are Journal indices (h-indices can also be calculated for individual researchers and will be the topic for a future editorial). A h-index of N is calculated if N papers published in a journal have been cited N or more times. The median h-index is the median number of times that those N articles have been cited. In these ways the h-index avoids being distorted by one or a small number of very highly cited articles, which is a problem for the other metrics. Another strength is that it provides citations from all sources available on the web and can include citations in textbooks and important websites for example. This is also a potential downfall in that some of those sources are non-scholarly and some citations can be double (or more) counted if they appear in different places on the web. In addition, the h-index is typically driven by the papers in the two earliest years as these have the largest citations, so it provides potentially out-of-date information. For example, Optometry & Vision Science's 2017 h-index of 36 is based on 34 papers from 2012 and 2013, two from 2014 and none from 2015 or 2016. The 2017 h-index is therefore indicating the citation levels of the 2012 and 2013 volumes. It also provides a total score of citations and does not divide by the number of papers a journal publishes in the 5-year period, so that it is highly dependent on the number of papers. For example, although Ophthalmic & Physiological Optics (OPO) has much higher average citation metrics (Impact factor, SCImago and CiteScore) than Optometry & Vision Science, OPO's Google Scholar h-index (25) is lower than Optometry & Vision Science (36) as it published about three times fewer papers in 2012–2016 (391 vs 1207). Of the six main citation metrics, three are 5-year, two 3-year and one 2-year. Eigenfactors chose 5-years as ‘in many research areas, articles are not frequently cited until several years after publication. Therefore, measures that only look at citations in the first 2 years after publication can be misleading’.3 Scopus’ metrics use 3-years as ‘The time window that is chosen for a metric calculated for all serial titles in a multi-disciplinary database like Scopus should be a compromise that is the best fit across all subject areas……3 years is long enough to capture the citation peak of the majority of disciplines. A shorter time window, such as 2 years, is unfair to slower moving fields such as Physics and Astronomy. A longer time window, such as 5 years, is unfair to more rapidly moving fields such as Biochemistry and Molecular Biology.’6 The champions for a 2-year citation metric are, not surprisingly, those that this metric favours such as researchers in the biochemistry and molecular biology fields. Indeed, it has been suggested that the importance of the 2-year impact factor in rating research quality has led to the loss of clinical staff in UK academic departments of medicine in favour of basic scientists who are likely to provide higher impact factor papers (and thereby perhaps gain greater funding for their University) for the UK Research Assessment Exercise.10 Basic research is cited by both basic research and clinical research papers, while clinical research is cited by other clinical research papers, textbooks, professional guidelines etc. and much of the latter does not get included in impact factor calculations. Certainly, researchers in the biochemistry and molecular biology fields can readily publish in high impact factor journals and this can affect inter-discipline comparisons within Universities and with granting agencies as well as Government research assessments. Journals receiving numerous citations soon after publication might suggest that they have immediate and high impact and therefore must be high quality. However, papers in fields that cite a high volume of recent work tend to do so because older work rapidly becomes obsolete because of rapid advances in that field.13 Taking the example in Table 2, a hypothetical journal in which papers were cited highly in the first few years but then tails off, could have an impact factor of 10 but minimal impact after about 5 years. A journal in a discipline that is more of a slow burner would have citations in the other direction. Its impact factor could be 2.5, but its greatest impact is after 5 years and remains highly cited in subsequent years given the broader and sustained relevance of the research. The former journal does not appear to be providing papers that are four-times more impactful or higher quality and it could be argued that its overall impact is considerably less in the longer term. The latter example is not fanciful. For example, OPO's top cited paper, with 324 citations at an average of 25 citations per year, received 2 and 6 citations in its first 2 years after publication! The top cited papers in OPO tend to peak in terms of citations in years 3 and 4. For example, the top 10 cited OPO papers from the last 8 years that have a full 5-year history had mean citations rates of 12 and 13 in years 1 and 2, yet 15 and 17 in years 3 and 4, then drop to 13 in year 5. The lower cited papers follow this pattern even more strongly. Given that our impact factor is 2.3 currently, this highlights that impact factors tend to be greatly influenced by a relatively small number of highly cited papers rather than the breadth of papers published.2, 9 A 5-year window appears to be better, but has other problems. As discussed previously the Google Scholar h-index tends to be out-of-date and often reflects the two oldest volumes included, so that a 2017 h-index is essentially based on citations from 2012 and 2013. The issue is also present for other 5-year metrics such as the Eigenfactor and 5-year impact factor, although to a lesser extent. The 5-year impact factor score also takes many years to change in that a new editorial team would have to wait about 7 years before they could determine whether changes to the journal had influenced the 5-year citation metrics. Three or 4 year metrics may therefore be a good compromise. The impact factor's inclusion of citations from editorial and letters without including them in the denominator during its calculation seems inappropriate. I am not aware of any rationale for doing this and no other citation metric does so. The variable inclusion of case reports and other items means that there is a lack of transparency in the calculation of the impact factor and it is difficult to predict despite access to Web of Science. All journal articles other than news items should be included in both the numerator and denominator of citation metric calculations. Although the inclusion of journal self-citation is open to citation ‘game playing’ by journals, their removal as in Eigenfactors, seems harsh on specialised journals (such as Cataract & Refractive Surgery and the Journal of Refractive Surgery) which are bound to have a high level of journal self-citations. The weighting of citations in Eigenfactors and SCImago seems reasonable, but again leads to questions about transparency: what is the algorithm? Who decides what it is? I assume this is why Scopus have recently introduced the simpler CiteScore. Finally, should the main citation metric be an average or total score of citations? Is a journal of higher quality because it produces a good number of highly cited papers (but many more poorly cited ones) because it publishes a large number of papers? Or is it a journal whose average quality is high? I clearly prefer the latter, but this is likely due to the fact that OPO is a relatively small journal. The use of the total number of citations without division by the number of papers published means that Google Scholar and Eigenfactor citation metrics are heavily influenced by how many papers the journal publishes: they do not measure quantity or quality but a variable mixture of both. An earlier editorial argued that several citation metrics should be used, the standard 2-year impact factor plus another.1 That argument was predicated on the perception that the impact factor is so well established it is irreplaceable. It is not. Not too many years ago, the suggestion of Snellen visual acuity being replaced would have been laughed out of clinics. However, the Snellen chart is flawed and visual acuity using logMAR-based charts is now the standard in research studies, clinical trials and in computer-based systems and is quickly becoming the standard in everyday clinics.14 Similarly, the impact factor (i.e. the standard 2-year impact factor) is flawed, mainly because the 2-year window is inappropriate for so many journals and disciplines,2, 3, 5, 9 but also because of its illogical calculation, restriction to the relatively small range of journals included in Web of Science and lack of transparency. A range of metrics may well be ideal,2, 4, 6 but the 2-year impact factor being used as the standard surely needs to be challenged. To answer the question in the title, the best citation metric for a clinical journal (that provides an average citation score) currently available appears to be Scopus’ SCImago Journal Ranking Indicator or CIteScore. This may be best accompanied by a range of other metrics2, 4 that could include the Web of Science's 5-year impact factor and cited half-life, download metrics and acceptance rate. Thank you to Prof. Joanne Wood for helpful comments on an earlier version of this editorial. As the Editor-in-Chief of OPO, I am likely to favour the citation metrics that best portrays OPO; similarly as a researcher I am likely to favour citation metrics that favour my own clinical vision science research and highlight inadequacies in metrics that do not (such as the 2-year impact factor). OPO is listed in Web of Science, Scopus and Google Scholar.

Comprehensive decade-long analysis of point-of-care resources reveals insights into how clinical research makes its way into practice across different medical specialties and through time to enable us to identify potential translational bottlenecks in the pathway from research to practice.

Classification of periodicals in the CAPES QUALIS system: changes to the criteria are URGENTLY needed!

The distribution of forensic journals, reflections on authorship practices, peer-review and role of the impact factor

We investigated academic developmental trends in emergency medicine (EM) by analyzing the performance of EM journals. Data from the Journal Citation Reports (JCR) database for EM category journals, including journal titles, language, numbers, and impact factors (IFs) from 2000 to 2019 were collected. The aggregate IFs of EM and 11 other categories (cardiac and cardiovascular systems, clinical neurology, critical care medicine, gastroenterology and hepatology, infectious diseases, general and internal medicine, pediatrics, respiratory system, surgery, toxicology, and urology and nephrology) were collected from 2003 to 2019. The slope of the linear regression was used to evaluate the trend in EM journal IFs and the aggregate IFs of all categories. Pearson’s correlation coefficient was used to evaluate the correlation between EM journals’ IF in 2000 and their IF trend from 2000 to 2019. The EM journal number increased from 8 (all in English) in 2000 to 31 (26 in English) in 2019. In total, 28 EM journals had a positive IF trend since their initial enrollment into the JCR database, and the trend was significant for 18 journals. The correlation of the EM journals’ IF in 2000 and IF trends from 2000 to 2019 was 0.75. The increasing trend of aggregate IF for the EM category was significant. In conclusion, the increased number, language diversity, and IF trend for EM journals indicates that the development of academic EM is a continuous international trend. In the past 20 years, the IF trend increased faster for EM journals with a higher initial IF. The overall performance of EM journals was non-inferior to other medical specialties.