Determinants of impact factor and Eigenfactor score in otolaryngology journals.

The decision of which journals researchers submit their manuscripts to is based on a range of factors, including the quality of reviews, acceptance probability, turnaround time, target audience and fit, but an increasing one is the journal's reputation or perceived ‘quality’ as most commonly currently quantified by its impact factor.1 In an earlier editorial, it was argued that the impact factor, although not without flaws, provides a reasonable ball park assessment of the journal's impact or usefulness.2 It was, however, suggested that a small array of metrics would be more appropriate and could include a second citation metric that included citations to textbooks (are papers from the journal being used to provide evidence for undergraduate and postgraduate texts?), a download metric (how many people are reading the articles?) and a journal's acceptance rate.2 Because of its recognised flaws, new citation metrics are being introduced to rival the impact factor3-7 and these new metrics have been highlighted by their use in recent international Government research assessments.7 For example, Elsevier's Scopus database, which provides the citation metrics SCImago Journal Ranking Indicator and CiteScore, has been used in a number of national assessment exercises, including the UK Research Excellence Framework (2014), The Excellence in Research for Australia (ERA) assessments (2010, 2012, 2015), the Italian Abilitazione Scientifica Nazionale's researcher assessment (2012, 2013) and the Portuguese Fundação para a Ciência e aTecnologia (FCT) assessment (2013/14).7 In light of these developments, it is timely to describe the various citation metrics and consider what is the most appropriate metric(s) for a clinical journal such as Ophthalmic & Physiological Optics (OPO). The most well-known, currently used citation metric is the Thomson-Reuters (now Clarivate Analytics) Institute of Scientific Information (ISI)'s 2-year journal impact factor.2, 8-10 The 2016 impact factor was calculated by the number of citations to the journal's articles published in 2014 and 2015, divided by the number of ‘papers’ published in the journal during those years. Only citations from journals that are included in Clarivate's Journal Citation Reports (JCR) on the online subscription-based service Web of Science (previously Web of Knowledge) count. This severely limits what is counted as a citation, although it could be argued that this keeps the quality of citations high. The denominator includes research papers (original articles, review papers, etc.) and does not include editorials, letters and news items, but JCR citations to all articles count (including those to editorials and letters). Case reports appear to be judged on a journal-by-journal basis by Web of Science, sometimes they are included in the denominator (and as they are rarely cited, this would lower impact factors) and sometimes not. This lack of transparency is also a flaw in the use of the impact factor and it is difficult to predict impact factors as it is not easy to forecast which documents will be included in calculations and which will not. OPO has an impact factor of 2.30 with 290 citations in 2014 and 2015 to 126 research papers (290/126 = 2.30). Web of Science also provides several other citation metrics, including an immediacy index, a 5-year impact factor and an Eigenfactor score (see Table 1).3 The 2016 immediacy index was determined by the number of citations in 2016 from papers published in 2016 (for OPO, 57/60 = 0.95) and is heavily dependent on papers published earlier in the year (e.g. the January OPO issue) as the papers published in the later issues (e.g. the November OPO issue) are highly unlikely to be cited in that year, given the lag in time from submission to publication of any papers that could cite a November publication. The immediacy index is sometimes considered an indicator for the following year's impact factor. This also highlights the fact that the 2-year window for impact factors is less (1.5 years?) for papers published in the November and December issues due to this time lag between submission and publication of citing papers. This is another reason why the 2-year impact factor is flawed. The 2016 5-year impact factor was determined by the number of citations in 2016 from papers published in 2011, 2012, 2013, 2014 and 2015 (for OPO, 777/323 = 2.41). It is similar to the 2-year impact factor in that includes citations to editorials and letters in the numerator, but does not count them in the denominator. Web of Science also includes an indicator of the longevity of citations from journals, the journal half-life. This indicates the median age of papers cited in the given year, so that OPO's 2016 cited half-life of 9.9 years, indicates that half of the citations to OPO in 2016 were from papers 9.9 years old or younger. This citation metric is rarely discussed or used, but is a useful indicator of the longevity of papers in a journal. For example, our website includes a list of OPO's classic papers (http://onlinelibrary.wiley.com/journal/10.1111/(ISSN)1475-1313/homepage/classic_papers.htm) that remain relevant and still cited despite being up to 25 years old. The Eigenfactor is produced for Web of Science by Jevin West and Carl Bergstrom at the University of Washington, www.eigenfactor.com).3 It provides 5-year citation data and weights the prestige of citations using an algorithm similar to Google's PageRank, so that citations from high ranking journals such as Nature and Science count more than lower ranking journals. It does not include journal self-citations (e.g. Journal of Cataract & Refractive Surgery citations in Journal of Cataract & Refractive Surgery papers; author self-citations in different journals are counted), which works against highly specialised journals. However, its algorithm attempts to adjust for citation differences between different research fields.3 Unlike impact factors, eigenfactors are based on a total score of citations and are not divided by the number of papers published. This means that if two journals have the same impact factor, but one publishes 1000 papers per year and the other 500 papers, the former will have twice the eigenfactor. This may be partly because the eigenfactor is provided as a reference for libraries, in that journals can be ranked using a ‘Cost-Effectiveness Search’ based on the citation value per dollar that they provide. From the perspective of libraries, the 1000 paper volume journal provides twice as many articles for their readers compared to a 500 paper journal of equal impact factor. SCImago and CiteScore are the principal citation metrics developed within Elsevier's online subscription-based service Scopus to rival impact factors provided by Clarivate Analytics’ Web of Science.6 SCImago is the most well-established of the Scopus metrics. It is based on 3-year citation data, so that the 2016 SCImago Journal Rank Indicator (JRI) is based on citations in 2016 from papers published in 2013, 2014 and 2015.11 Like Eigenfactors, it weights the prestige of citations using an algorithm similar to Google's PageRank. It also differs from impact factors in that editorials and letters are included in the denominator. This means that journals which include a large number of editorials and letters, such as the Nature and Lancet series of journals, tend to have significantly lower SCImago JRIs (and CiteScores) than would be expected from their impact factors.5 CiteScore is a more recently introduced metric (December 2016) and is similar to SCImago, but differs in that it doesn't weight citations (Table 1). Both SCImago and CiteScore consider ‘documents’ beyond just journal articles, such as books and conference proceedings. Indeed, the Scopus database includes far more (they claim ‘double’) journals than Web of Science. For example, Web of Science includes four optometry journals: Ophthalmic & Physiological Optics, Optometry & Vision Science, Contract Lens & Anterior Eye and Clinical & Experimental Optometry. Scopus includes these four journals, plus the Journal of Optometry and Clinical & Refractive Optometry and Scopus includes a subcategory of optometry journals11 unlike Web of Science where Optometry journals are considered within the Ophthalmology subcategory or Google Scholar which has a joint Ophthalmology and Optometry subcategory.12 Google Scholar provides two citation metrics, the 5-year h-index and 5-year median h-index.12 These h-indices are Journal indices (h-indices can also be calculated for individual researchers and will be the topic for a future editorial). A h-index of N is calculated if N papers published in a journal have been cited N or more times. The median h-index is the median number of times that those N articles have been cited. In these ways the h-index avoids being distorted by one or a small number of very highly cited articles, which is a problem for the other metrics. Another strength is that it provides citations from all sources available on the web and can include citations in textbooks and important websites for example. This is also a potential downfall in that some of those sources are non-scholarly and some citations can be double (or more) counted if they appear in different places on the web. In addition, the h-index is typically driven by the papers in the two earliest years as these have the largest citations, so it provides potentially out-of-date information. For example, Optometry & Vision Science's 2017 h-index of 36 is based on 34 papers from 2012 and 2013, two from 2014 and none from 2015 or 2016. The 2017 h-index is therefore indicating the citation levels of the 2012 and 2013 volumes. It also provides a total score of citations and does not divide by the number of papers a journal publishes in the 5-year period, so that it is highly dependent on the number of papers. For example, although Ophthalmic & Physiological Optics (OPO) has much higher average citation metrics (Impact factor, SCImago and CiteScore) than Optometry & Vision Science, OPO's Google Scholar h-index (25) is lower than Optometry & Vision Science (36) as it published about three times fewer papers in 2012–2016 (391 vs 1207). Of the six main citation metrics, three are 5-year, two 3-year and one 2-year. Eigenfactors chose 5-years as ‘in many research areas, articles are not frequently cited until several years after publication. Therefore, measures that only look at citations in the first 2 years after publication can be misleading’.3 Scopus’ metrics use 3-years as ‘The time window that is chosen for a metric calculated for all serial titles in a multi-disciplinary database like Scopus should be a compromise that is the best fit across all subject areas……3 years is long enough to capture the citation peak of the majority of disciplines. A shorter time window, such as 2 years, is unfair to slower moving fields such as Physics and Astronomy. A longer time window, such as 5 years, is unfair to more rapidly moving fields such as Biochemistry and Molecular Biology.’6 The champions for a 2-year citation metric are, not surprisingly, those that this metric favours such as researchers in the biochemistry and molecular biology fields. Indeed, it has been suggested that the importance of the 2-year impact factor in rating research quality has led to the loss of clinical staff in UK academic departments of medicine in favour of basic scientists who are likely to provide higher impact factor papers (and thereby perhaps gain greater funding for their University) for the UK Research Assessment Exercise.10 Basic research is cited by both basic research and clinical research papers, while clinical research is cited by other clinical research papers, textbooks, professional guidelines etc. and much of the latter does not get included in impact factor calculations. Certainly, researchers in the biochemistry and molecular biology fields can readily publish in high impact factor journals and this can affect inter-discipline comparisons within Universities and with granting agencies as well as Government research assessments. Journals receiving numerous citations soon after publication might suggest that they have immediate and high impact and therefore must be high quality. However, papers in fields that cite a high volume of recent work tend to do so because older work rapidly becomes obsolete because of rapid advances in that field.13 Taking the example in Table 2, a hypothetical journal in which papers were cited highly in the first few years but then tails off, could have an impact factor of 10 but minimal impact after about 5 years. A journal in a discipline that is more of a slow burner would have citations in the other direction. Its impact factor could be 2.5, but its greatest impact is after 5 years and remains highly cited in subsequent years given the broader and sustained relevance of the research. The former journal does not appear to be providing papers that are four-times more impactful or higher quality and it could be argued that its overall impact is considerably less in the longer term. The latter example is not fanciful. For example, OPO's top cited paper, with 324 citations at an average of 25 citations per year, received 2 and 6 citations in its first 2 years after publication! The top cited papers in OPO tend to peak in terms of citations in years 3 and 4. For example, the top 10 cited OPO papers from the last 8 years that have a full 5-year history had mean citations rates of 12 and 13 in years 1 and 2, yet 15 and 17 in years 3 and 4, then drop to 13 in year 5. The lower cited papers follow this pattern even more strongly. Given that our impact factor is 2.3 currently, this highlights that impact factors tend to be greatly influenced by a relatively small number of highly cited papers rather than the breadth of papers published.2, 9 A 5-year window appears to be better, but has other problems. As discussed previously the Google Scholar h-index tends to be out-of-date and often reflects the two oldest volumes included, so that a 2017 h-index is essentially based on citations from 2012 and 2013. The issue is also present for other 5-year metrics such as the Eigenfactor and 5-year impact factor, although to a lesser extent. The 5-year impact factor score also takes many years to change in that a new editorial team would have to wait about 7 years before they could determine whether changes to the journal had influenced the 5-year citation metrics. Three or 4 year metrics may therefore be a good compromise. The impact factor's inclusion of citations from editorial and letters without including them in the denominator during its calculation seems inappropriate. I am not aware of any rationale for doing this and no other citation metric does so. The variable inclusion of case reports and other items means that there is a lack of transparency in the calculation of the impact factor and it is difficult to predict despite access to Web of Science. All journal articles other than news items should be included in both the numerator and denominator of citation metric calculations. Although the inclusion of journal self-citation is open to citation ‘game playing’ by journals, their removal as in Eigenfactors, seems harsh on specialised journals (such as Cataract & Refractive Surgery and the Journal of Refractive Surgery) which are bound to have a high level of journal self-citations. The weighting of citations in Eigenfactors and SCImago seems reasonable, but again leads to questions about transparency: what is the algorithm? Who decides what it is? I assume this is why Scopus have recently introduced the simpler CiteScore. Finally, should the main citation metric be an average or total score of citations? Is a journal of higher quality because it produces a good number of highly cited papers (but many more poorly cited ones) because it publishes a large number of papers? Or is it a journal whose average quality is high? I clearly prefer the latter, but this is likely due to the fact that OPO is a relatively small journal. The use of the total number of citations without division by the number of papers published means that Google Scholar and Eigenfactor citation metrics are heavily influenced by how many papers the journal publishes: they do not measure quantity or quality but a variable mixture of both. An earlier editorial argued that several citation metrics should be used, the standard 2-year impact factor plus another.1 That argument was predicated on the perception that the impact factor is so well established it is irreplaceable. It is not. Not too many years ago, the suggestion of Snellen visual acuity being replaced would have been laughed out of clinics. However, the Snellen chart is flawed and visual acuity using logMAR-based charts is now the standard in research studies, clinical trials and in computer-based systems and is quickly becoming the standard in everyday clinics.14 Similarly, the impact factor (i.e. the standard 2-year impact factor) is flawed, mainly because the 2-year window is inappropriate for so many journals and disciplines,2, 3, 5, 9 but also because of its illogical calculation, restriction to the relatively small range of journals included in Web of Science and lack of transparency. A range of metrics may well be ideal,2, 4, 6 but the 2-year impact factor being used as the standard surely needs to be challenged. To answer the question in the title, the best citation metric for a clinical journal (that provides an average citation score) currently available appears to be Scopus’ SCImago Journal Ranking Indicator or CIteScore. This may be best accompanied by a range of other metrics2, 4 that could include the Web of Science's 5-year impact factor and cited half-life, download metrics and acceptance rate. Thank you to Prof. Joanne Wood for helpful comments on an earlier version of this editorial. As the Editor-in-Chief of OPO, I am likely to favour the citation metrics that best portrays OPO; similarly as a researcher I am likely to favour citation metrics that favour my own clinical vision science research and highlight inadequacies in metrics that do not (such as the 2-year impact factor). OPO is listed in Web of Science, Scopus and Google Scholar.

The role of impact factor in evaluating the value of scientific publications is critically reviewed.
\nOther possible criteria for evaluation are assessed and proposed.

We investigated academic developmental trends in emergency medicine (EM) by analyzing the performance of EM journals. Data from the Journal Citation Reports (JCR) database for EM category journals, including journal titles, language, numbers, and impact factors (IFs) from 2000 to 2019 were collected. The aggregate IFs of EM and 11 other categories (cardiac and cardiovascular systems, clinical neurology, critical care medicine, gastroenterology and hepatology, infectious diseases, general and internal medicine, pediatrics, respiratory system, surgery, toxicology, and urology and nephrology) were collected from 2003 to 2019. The slope of the linear regression was used to evaluate the trend in EM journal IFs and the aggregate IFs of all categories. Pearson’s correlation coefficient was used to evaluate the correlation between EM journals’ IF in 2000 and their IF trend from 2000 to 2019. The EM journal number increased from 8 (all in English) in 2000 to 31 (26 in English) in 2019. In total, 28 EM journals had a positive IF trend since their initial enrollment into the JCR database, and the trend was significant for 18 journals. The correlation of the EM journals’ IF in 2000 and IF trends from 2000 to 2019 was 0.75. The increasing trend of aggregate IF for the EM category was significant. In conclusion, the increased number, language diversity, and IF trend for EM journals indicates that the development of academic EM is a continuous international trend. In the past 20 years, the IF trend increased faster for EM journals with a higher initial IF. The overall performance of EM journals was non-inferior to other medical specialties.

A deeper understanding of supplementary bibliometrics beyond the impact factor might provide researchers with a better understanding of the citation process. This study presents a multivariate analysis of gastroenterology and hepatology journals to evaluate the predictive ability of seven bibliometrics in the Web of Science to calculate total cites over a 2-year period. Coincidentally, bibliometrics appearing during 2008, 2009, and 2010, with their corresponding cites in 2010, 2011, and 2012, were recorded from the Journal Citation Reports Science Edition. A linear mixed-effects design using random slopes and intercepts was performed on 51 out of 74 journals in the Gastroenterology and Hepatology category. There was a significant global effect size (R(2) = 0.992; P < 0.001), which yielded a total variance of 99.2%. The strongest predictors in the model were the Eigenfactor Score and Cited Half-life (P < 0.001), followed by the Number of Articles (P = 0.011) and the Immediacy Index (P = 0.021). The impact factor was not a significant predictor. The Eigenfactor Score and Cited Half-life predictors might be the new standards to assess the influence and importance of scientific journals; this approach may help researchers select journals in which to publish their work.

Inappropriate use of standard error of the mean when reporting variability of study samples: A critical evaluation of four selected journals of obstetrics and gynecology

Here we show a novel technique for comparing subject categories, where the prestige of academic journals in each category is represented statistically by an impact-factor histogram. For each subject category we compute the probability of occurrence of scholarly journals with impact factor in different intervals. Here impact factor is measured with Thomson Reuters Impact Factor, Eigenfactor Score, and Immediacy Index. Assuming the probabilities associated with a pair of subject categories our objective is to measure the degree of dissimilarity between them. To do so, we use an axiomatic characterization for predicting dissimilarity between subject categories. The scientific subject categories of Web of Science in 2010 were used to test the proposed approach for benchmarking Cell Biology and Computer Science Information Systems with the rest as two case studies. The former is best-in-class benchmarking that involves studying the leading competitor category; the latter is strategic benchmarking that involves observing how other scientific subject categories compete.

BackgroundThe Anais Brasileiros de Dermatologia (Anais Brasileiros de Dermatologia) is the official journal of the Brazilian Society of Dermatology, which has been published since 1925, with free, bilingual access. There are several bibliometric indicators that estimate aspects of a journal’s scientific influence. Its analysis, based especially on the evolutionary trend, allows identifying the journal’s strengths and weaknesses, in addition to guiding editorial policies. ObjectivesTo evaluate the trends of the main bibliometric indicators of Anais Brasileiros de Dermatologia, in the decade of 2010–2019. MethodsMethodological study that proposed the analysis of bibliometric indicators published by Journal Citation Reports, SCImago, and Scopus, for the journal Anais Brasileiros de Dermatologia, from 2010 to 2019. The following aspects were evaluated: Impact factor, Immediacy index, SJR (SCImago), in addition to the number of citations, citable items, Cite Score, percentage of external citations, percentage of cited articles, percentage of citable articles, and the article influence score. ResultsThere was an increase in the main bibliometric indicators in the last decade: impact factor (0.337 to 1.121), immediacy index (0.054 to 0.204),Eigenfactor (0.00025 to 0.00394), SJR (0.176 to 0.453). The percentage of external citations (48.4% to 94.1%) and cited articles (24.7% to 51.4%) also increased. The increase in the impact factors of Anais Brasileiros de Dermatologia in the period was more significant than that of the dermatology journals (1,667 to 2,118) and the Brazilian journals (1,247 to 1,408), indexed in the Journal Citation Reports. Study limitationsFailure in the classification and registration of the databases. ConclusionsThere has been a consistent increase in the scientific influence of Anais Brasileiros de Dermatologia in the last decade.

Impact Factors and Other Measures of a Journal's Influence

A comparison among citation-based journal indicators and their relative changes with time

Impact factors and relevance of research outputs: one step forward, two back?

Background:Impact Factor (IF) as a major journal quality indicator has a series of shortcomings including effect of self-citation, review articles, total number of articles, etc. In this study, we compared 4 journals quality indices ((IF), Eigenfactor Score (ES), Article Influence Score (AIS) and SCImago Journal Rank indicator (SJR)) in the specific Pediatric Neurology journals.Methods:All ISI and Scopus indexed specific Pediatric Neurology journals were compared regarding their 2011 IF, ES, AIS and SJR.Results:Fourteen pediatric Neurology journals were identified, 3 of which were only Scopus indexed and the others were both ISI and Scopus indexed. High correlation was found between IF and AIS (0.850). Correlations between IF and other indices were not that high. Self-citation, total article number and review articles were related to the IF and other indices as well as their ranks. English language and citation to non citable item didn’t have any effect on pediatric neurology journals ranks.Conclusion:Although all the above mentioned indicators can be used interchangeably, using all considered indices is a more appropriate way than using only IF for quality assessment of pediatric neurology journals.

The Journal has experienced substantial growth in the last few years which resulted in an additional growth of 300 printed pages, a change in page budget in 2004. During this time, we have also experienced a high impact factor of 0.77 in 2004 which is among the highest for comparable journals in the field. The Immediacy Index for JSE exceeds all other structural engineering and related journals by a wide margin. The Impact Factor and the Immediacy Index are two standard metrics used to measure citations to journal articles over time. The build-up of citations typically follows a curve referred to as the “citation curve.” The impact factor is a measure of the relative size of the citation curve in the second and third years. It is calculated by dividing the number of citations to papers published in the two previous years by the number of papers published in those same years. The immediacy index, on the other hand, provides a measure of the skewness of the citation curve and is calculated by dividing the citations in the current year by the number of papers it publishes in that year. This is a measure of how quickly items in that journal get cited upon publication. While these figures are encouraging, there are some recent trends that are also a cause for concern. The most recent statistics for the Journal indicates that the median turnaround time from submission to publication has increased from 19 to 21 months, despite the fact that there has been an increased in printed pages. The primary cause for the increased turnaround time is mostly delays in the review process. While the blame can sometimes be attributed to Associate Editors handling the review of papers, some of the blame has also to point to those of us, who are called upon to conduct the peer review of submitted manuscripts. Since self-citations meaning citations in a paper to articles previously published in the same journal are included in the impact factor calculation, it is obvious given the 20+ month turnaround that the current impact factor of the Journal is generated almost exclusively by citations to JSE by papers published in other journals. A reduction in the mean turnaround time can contribute significantly to both the appeal of the journal and to further increasing the impact factor. I recognize that focusing on such metrics should not be the sole objective of a premier journal.

Longitudinal analysis of citation data provides useful historical information on research and publishing trends in our field. A citation-based analysis was performed on 5 core journals published between 1985 and 2006. Detailed historical information on citation rates, cited half-lives, immediacy indices, and impact-factor scores were extracted from the Thomson Scientific Journal Citation Reports via a custom order, and systematically analyzed. Occupational medicine journals have attracted an increasing number of citations in recent years, with this figure rising from just over 5,000 per year in 1985 to almost 17,000 per year by 2006. Impact factors have also increased between 108% and 186% per journal, with an average rise of approximately 158% during the time period examined. Overall, this study suggests that occupational medicine is now attracting increasing attention in the scientific literature, as evidenced by rising citation counts and impact factors.

Various kinds of metrics used for the quantitative evaluation of scholarly journals are reviewed. The impact factor and related metrics including the immediacy index and the aggregate impact factor, which are provided by the Journal Citation Reports, are explained in detail. The Eigenfactor score and the article influence score are also reviewed. In addition, journal metrics such as CiteScore, Source Normalized Impact per Paper, SCImago Journal Rank, h-index, and g-index are discussed. Limitations and problems that these metrics have are pointed out. We should be cautious to rely on those quantitative measures too much when we evaluate journals or researchers.

Purpose: This study explored changes in the journal publishing market by publisher and access type using the major journals that publish about 95% of Journal Citation Reports (JCR) articles.Methods: From JCR 2016, 2018, and 2020, a unique journal list by publisher was created in Excel and used to analyze the compound annual growth rate by pivot tables. In total, 10,953 major JCR journals were analyzed, focusing on publisher type, open access (OA) status, and mega journals (publishing over 1,000 articles per year).Results: Among the 19 publishers that published over 10,000 articles per year, in JCR 2020, six large publishers published 59.6% of the articles and 13 publishers 22.5%. The other publishers published 17.9%. Large and OA publishers increased their article share through leading mega journals, but the remaining publishers showed the opposite tendency. In JCR 2020, mega journals had a 26.5% article share and an excellent distribution in terms of the Journal Impact Factor quartile. Despite the high growth (22.6%) and share (26.0%) of OA articles, the natural growth of non-OA articles (7.3%) and total articles (10.7%) caused a rise in journal subscription fees. Articles, citations, the impact factor, and the immediacy index all increased gradually, and the compound annual growth rate of the average immediacy index was almost double than that of the average impact factor in JCR 2020.Conclusion: The influence of OA publishers has grown under the dominance of large publishers, and mega journals may substantially change the journal market. Journal stakeholders should pay attention to these changes.