No open-access location is known for this record yet.
No open-access location is known for this record yet.
Good measurement is essential to good science. Indeed, developing techniques that standardize assessments of experimental results is at the heart of the scientific method and essential for scientists to perform and report their empirical work objectively. Because it forms the basis for our understanding of the world and complex phenomena, advancements and innovations in measurement tools have always been a significant marker of scientific progress. The naked eye gave way to optical and electron microscopy, which is now leading to newer techniques of dynamic molecular imaging.1 Accordingly, there are few places where such advances in measurement have been more celebrated than in scientific journals, where editors continually spur investigators toward clearer descriptions of their methodological techniques and work. Yet while accurate and reliable measurement is a core principle of good science and the foundation for evaluating research, editors and scientific journals have often been less successful at applying similar standards to their own work—particularly in medicine. It is estimated that between 15 000 and 20 000 scientific journals dedicated to biomedical research are regularly published today.2 Remarkably, the purpose and influence of most of these journals is largely unknown, and it is challenging to gauge their quality—how many are good, average, bad, worthless, or, in the words of one editorialist from 1879 in the New England Journal of Medicine, ‘worse than worthless’.3 Not surprisingly, a lack of available measures is not the problem. We have several that are currently on hand for our own journal, the European Heart Journal, and it is not uncommon for us to review them periodically.4 Some are focused on the journal's ‘business’ side: ‘What is our circulation and how many new readers are we attracting?’; and ‘What is our margin after we consider revenue and costs of production?’ Naturally, these measures are important, since it is essential for the survival of journals to be financially sound and self-sufficient. Other measures are focused on our attractiveness to investigators and potential authors: ‘How many articles are being submitted to us and what is our rejection rate?’; ‘What is the time to decision and publication?’; ‘Do we have a process in place for expedited review?’; ‘Is there a standardized and fair appeal process?’ These measures are critical given that original research articles remain the ‘life-blood’ of medical journals, and authors are a key customer base for us. Yet if the ultimate goal of medical journals is to advance knowledge and affect patient care, there are far fewer measures available. Furthermore, those that are available have their limitations. As an example, consider the ‘impact factor’—a widely applied metric for journals and one which continues to be poorly understood.5 Technically, the impact factor is the measure of the average number of citations received by papers published in a journal in recent years. It is calculated by dividing the number of citations to the journal's articles in the year of interest (the numerator) by the number of citable items published by the journal in the two previous years (the denominator). The former is composed of citations of the journal as a whole regardless of the type of article (research paper, review, etc.), while the latter is composed of scholarly items, primarily research articles. As the impact factor is assessed at the level of each individual journal, it is meant to serve as a proxy for the journal's relative importance by describing how often its papers have influenced other scientists compared with other journals. The concept of the impact factor was first devised by Eugene Garfield >50 years ago and the idea has taken many forms.6 Yet its most common usage refers to the specific application by the Thomson Reuters' Web of Knowledge, which is released annually.7 Of course, use of the impact factor is not without controversy, and there have been several attempts to improve or modify it over the years, such as the Eigenfactor and SCImago Journal Rank.8,9 Indeed, well-recognized limitations of the impact factor include (i) its topic dependency (e.g. a less ‘important’ article on a common disease such as coronary artery disease is more likely to be cited than a more ‘important’ article on a rare disease such as inherited cardiomyopathy); and (ii) misapplication to an individual article or investigator within a journal (i.e. good journals can publish average articles and vice versa). A well-cited example of how this metric can be distorted was also highlighted by the dramatic rise of the journal Acta Crystallographica A in 2009 from its usual impact factor of ∼2 to 49.93 (placing it as the second highest journal in the ‘all journals’ category for that year). This rise was due almost entirely to a single paper by Sheldrick published in 2008 regarding an open-source program on crystal structure determination (SHELX programs). Since citations for this article were no longer counted in later impact factors, the impact factor for this journal subsequently fell and remained at 2.08 in 2011, consistent with its historic rates. A more recent concern has been the growing practice of ‘gaming’ the impact factor through controversial editorial policies. The role of self-citation within journals varies considerably, and may contribute substantially to their impact factor.10 Anecdotally, some journals have attempted to increase their impact factor by asking authors to cite specific articles previously published in their journal.11,12 While it may be appropriate under certain circumstances to ask submitting authors to cite guidelines or specific articles—if they are crucial to the work considered—there are certainly limits to such policies. Despite legitimate concerns that have been raised, the impact factor clearly remains an important yardstick for medical journals. As much as we at the European Heart Journal may not like to admit it, we have also kept a close eye on the impact factor and have employed it as a means for measuring our own progress over the years relative to that of our peers.13 We confess to be particularly proud that the European Heart Journal is among the minority of journals that have improved their impact factors (up to 10.5 in 2011), while also publishing more citable items (e.g. original research articles) and avoiding systematic self-citation policies.14 Another measure of journal quality that has received an enormous amount of attention recently from the lay media is the rate of retraction for a journal, and its relationship to the journal's quality and overall peer-review process.15 The rising concern is largely due to growing empirical evidence that rates of retractions are increasing. A recent study by Steen, for example, reported that rates of retractions increased sharply between 2000 and 2010.16 A particularly disturbing part of that study was that more than a quarter of retractions were for scientific fraud and that many involved high profile journals. In a subsequent analysis by Fang and Casadevall that was also published last year,17 standardized rates of retractions by 17 high profile journals over the last decade appeared to correlate directly with their impact factors (Figure 1). That is, higher impact journals had higher likelihoods of retraction. Correlation between impact factor and retraction index. The 2010 journal impact factor is plotted against the retraction index as a measure of the frequency of retracted articles from 2001 to 2010 (see text for details). Journals analysed were Cell, EMBO Journal, FEMS Microbiology Letters, Infection and Immunity, Journal of Bacteriology, Journal of Biological Chemistry, Journal of Experimental Medicine, Journal of Immunology, Journal of Infectious Diseases, Journal of Virology, Lancet, Microbial Pathogenesis, Molecular Microbiology, Nature, New England Journal of Medicine, Proceedings of the National Academy of Sciences (USA), and Science. Reprinted with permissions from: Infect Immun 2011;79:3855. doi: 10.1128/IAI.05661-11. Ferric C Fang and Arturo Casadevall.17 Two explanations that were proposed were: (i) the greater prestige associated with publishing in higher impact journals promotes risk-taking behaviour by investigators; and (ii) the broader visibility of articles in higher impact journals leads to greater scrutiny and detection. Either explanation points toward the fact that use of retraction rates in isolation is unlikely to be a good measure for determining a journal's quality. It also paradoxically suggests that journals aiming to improve their impact factor become increasingly vulnerable to the worst lapses in quality (scientific fraud and misconduct) by the very nature of their growing importance. In any case, these numbers show that the ethics of publishing as well as the honesty and precision of scientific activity are under increasing scrutiny. However, detecting such behaviour is a difficult, if not impossible, task for editors, and hence should rather be approached by the broader scientific community and in institutions in which research is performed. Most medical journals serve several key functions for a diverse readership. These functions include (i) reporting original scientific research; (ii) clinical education and reporting of best practices; (iii) serving as a forum for controversial debates; and (iv) to inform readers about changes within the cardiovascular community. Notwithstanding the well documented limitations of using the impact factor and rates of retractions that have been described above, both measures also do nothing to assess any of the functions which extend beyond the reporting of original scientific research. Of greatest concern is that neither assesses the performance of a journal in the eyes of the clinical reader, i.e. those individuals who make up the bulk of our audience, but who will never author a paper or cite one of our articles. To date, it is unknown whether we are successfully reaching this group and to what extent our articles are generating added value for this group by informing them of controversies and new ideas that can be applied to their practice and patients. As an example, consider the recent guidelines published in the European Heart Journal on ‘Prevention, diagnosis, and treatment of infective endocarditis’.18 This paper has received only 168 citations to date, possibly because this is a research area with currently little innovation. However, it has received an impressive 62 300 full-text downloads which reflects how important endocarditis is for clinicians in daily practice (Figure 2). Similar examples exist for guidelines on the ‘Diagnosis and treatment of pulmonary hypertension’ (39 100 downloads),19 the ‘Diagnosis and management of syncope’ (25 800 downloads),20 and ‘Pre-operative cardiac risk assessment and perioperative cardiac management in non-cardiac surgery’ (25 000 downloads).21 It is less likely that niche topics such as these will ever reach the number of citations that a document focused on highly prevalent conditions such as coronary artery disease or congestive heart failure would; however, these articles answer clinical questions that are of direct relevance to cardiologists in the ‘real world’. The number of downloads (since 1 January 2008) and citations for select Guidelines and Expert Consensus Documents published in the European Heart Journal extending back to 2004. Hence, if it is clear that we need to publish articles in these fields, how do we know we are succeeding in doing this in the best possible manner given that existing measures (e.g. the impact factor) are unlikely to be applicable? In the past, we largely relied on word of mouth and on occasional reader surveys which we conducted. Yet, the limitations of that approach are obvious. The European Heart Journal is the flagship journal of the European Society of Cardiology (ESC), an organization that represents ∼70 000 professionals and 54 national societies of cardiology. Even the numerous Emails and responses we receive from engaged readers are unlikely to represent a completely clear picture of how we can improve going forward. One possible solution might be regular queries or polls of a representative sample of the readers of a given journal such as the European Heart Journal, carried out with the help of Oxford University Press. Indeed, the results of such polls could help the editors to improve their editorial policy in response to their readers' needs. Until recently, our understanding of how we reach clinical readers and objective measurements of our success in doing so was limited. However, the well documented changes now occurring in the publishing industry due to technological advancements are creating new opportunities. We have already taken advantage of some of these and also aim to expand into new areas. For example, we believe that improved integration of the European Heart Journal in the annual ESC Congress makes a great deal of sense. This is especially critical given that the considerable size and extent of the ESC Congress and its close alignment with the goals of the European Heart Journal of clinical education and reporting original scientific research. One simple and ‘anecdotally’ successful endeavour the European Heart Journal launched was to have sessions at the ESC Congress dedicated to what our editors believe have been particularly important and influential articles published in the European Heart Journal during the last year. In selecting these articles, we did not focus on criteria to determine those that were the most scientifically impactful but rather those with the greatest relevance for clinicians. Early feedback from attendees has been generally enthusiastic. Another area where we have made great inroads is in the new world of multimedia for journals. For example, as a direct ‘tie-in’ to the ESC Congress, we now produce online video interviews—My Cardio Interviews—with our editors and high profile investigators as well as key opinion leaders which roughly coincide with the reporting of late-breaking clinical trials.22 These interviews are succinct and user-friendly summaries, providing viewers from all over the world with information on how new studies are likely to affect their everyday clinical practice. These new online video interviews have proven to be enormously popular, with >56 300 internet hits on our site at the time of writing. The opportunity to expand our ability to integrate traditional scientific reports with more user-friendly interfaces is only going to grow, particularly with the recent launching of the iPad and iPhone applications for the European Heart Journal (Figure 3). Already the applications from the Apple store have been downloaded 2850 times, and this figure will increase considerably at this year's ESC Congress. Our aim is to make these applications more than just an electronic version of the European Heart Journal. Instead, we want to leverage the ability to integrate online videos and to provide additional supplementary material (such as the European Heart Journal Image Bank). We are convinced that these applications will substantially promote an educational experience far beyond the conventional bounds of the traditional journal. The new iPad and iPhone Apps for the European Heart Journal (photo courtesy of S. Rogers). Finally, these technological advances are also generating broad insights into how we measure our influence as a journal by providing early insights and feedback into where we should be going in the future. An area that we have great interest in developing is article-level metrics.23 As opposed to the impact factor (and its variants), article-level metrics provide a major step forward in evaluating the performance of individual articles published in the European Heart Journal. The Public Library of Science (PLoS) has been instrumental in developing several article-level metrics that integrate traditional approaches such as the impact factor that measure scientific impact with overall interest and readership. That is, how often are others commenting on articles and how are peers ‘ranking’ them? How many of the articles are being viewed or downloaded, and for how long? How often are text, tables, and figures from the article being ‘cut-and-pasted’? Are articles being selected by blogs, Twitter, and lay media outlets?24,25 While article-level metrics are also admittedly prone to potential gaming and manipulation (as in the case of the impact factor), advances in the Web advertising industry have created methods for improving the comparability of these statistics.23 Finally, these approaches may enable us to expand immediate user engagement and feedback between readers and the editorial board. After reading an article in the European Heart Journal, it could be possible for readers to share how well that article accomplished the task of focusing on an important area or provided them with educational information that directly changed their clinical practices. These types of direct feedback could be better tied to continuing medical education requirements to give us a more real-time assessment of how we are doing. To that end, these approaches will move us from the concept of ‘impact’ to ‘influence’— a key goal for us as editors. The role of measurement is clear in biomedical research but less so for the journals which report the results of such work. While many measures related to the business and scientific impact of medical journals will continue to be required, there are increasingly new ways to measure how published articles directly affect clinical practice. By developing a better understanding of what works, we believe that editors (including us) can respond more thoughtfully and systematically to a diverse group of readers and ensure that their publications optimize their ability to educate clinicians, discuss controversy, and effectively inform the cardiovascular community. We are grateful to Sam Rogers and Susanne Dedecke for their careful review and editorial comments related to a previous version of this paper. We are grateful to Sacha Wason for compiling data for Figure 2 and to Sam Rogers for creating Figure 3. Conflict of interest: none declared.
Loading…
Stored only when both papers are in the corpus — 41 citations exist upstream.
Papers that cite the same references — the relational fallback while embeddings are still backfilling.
Imported from OpenAlex (CC0) — nothing here is generated.