Health research in the European Union: over-controlled but under-measured?

Abstract

Research will only increase the sum of human knowledge if it is accessible to those who can make use of it. In recent years, the funders, providers and users of research have been placing increasing emphasis on communicating results1 by ensuring that the findings are identifiable, freely accessible and understandable.2 Yet there is still a great deal to be done.3 For example, one study found that, nine years later, only 53% of research reported in conference abstracts had ever been published as full reports.4 In another recent Australian study, policy makers reported difficulty in accessing relevant research and only one-third of researchers had implemented strategies to inform policy makers of their findings.5 In this commentary we describe our experiences in conducting a European Union (EU)-funded project, Health Research for Europe (HR4E) that sought to assemble and disseminate findings from health-relevant research within the EU’s Fifth and Sixth Framework Programmes (FP5 and FP6). We offer a number of key lessons from the project that are relevant to future EU-funded research programmes. We examined health–related projects funded within the FP5 and FP6 programmes (∼5000) to assess the impact of EU research on policy and practice (K. Ernst et al., submitted for publication). These projects can be found in Community Research and Development Information Service (CORDIS)—the European Commission’s Publication Office ‘information space’ that purports to contain details of all EU research and development activities and projects (Box 1). It is available in searchable form at http://cordis.europa.eu/home_en.html. CORDIS, the Community Research and Development Information Service for Science, Research and Development, is the official source of information on the Seventh framework programme (FP7) calls for proposals; it offers interactive web facilities that links together researchers, policymakers, managers and key players in the field of research. CORDIS has three main missions: To facilitate participation in European Research activities; To enhance exploitation of research results with an emphasis on sectors crucial to Europe’s competitiveness; To promote the dissemination of knowledge fostering the innovation performance of enterprise and the societal acceptance of new technology. Unfortunately, we found the CORDIS database frequently inaccurate or obsolete. For example, the entry for ‘Europe for Patients’, a project on patient mobility within the European Union,6 contains only one ‘exploitable result’, a report on ‘Transgenic tomato plants altered in the expression of biosynthetic genes involved in flavonoid biosynthesis’, which was actually undertaken within a project entitled ‘Improved antioxidant content for food applications’. Notwithstanding such errors, CORDIS does provide a useful tool for locating individual projects. However, it is not possible to download outputs, such as lists of projects (with relevant variables) for subsequent analysis so it was necessary for us to obtain the underlying data from the EU Data Warehouse. We had expected to be able to find and use the often voluminous final project reports on CORDIS or in the Data Warehouse. Worryingly, these do not seem to be retrievable in any form; the very helpful staff in the Directorate General (DG) Research were equally unable to retrieve project outputs. Instead, the only information available was an ‘abstract’ or excerpt from the initial project proposal. Invariably, this had been written before the project started and provided no results or conclusions. In many cases, the ‘abstract’ was very technical or spiced with grammatical oddities that made it near-incomprehensible. Phrases such as ‘results will be classified policy objectives’ were not uncommon. An abstract on auditory and visual processes concluded with the bold promise of a ‘revision of unmoral perceptual models’. One abstract simply and beautifully declared: ‘ici l’abstract’. Given the difficulties in obtaining project results from CORDIS, we explored other methods. One possible approach was to locate publications arising from EU-funded research by querying their grant codes. Entries in the US National Library of Medicine’s PubMed database contain a field denoting the funding source. However, the list of codes is at present limited to certain American and British research funders. Another time-consuming possibility was to contact the named project coordinators. However, contact information on CORDIS was often absent or obsolete. CORDIS does have an encrypted e-mail facility but does not permit the actual e-mail addresses to be downloaded for verification; nor does it provide delivery confirmation. At a later stage we attempted to contact coordinators of a subset of 182 health-related projects. In this sample, accurate e-mail addresses were available for only 25% of the projects and where the e-mail address was incorrect or absent, the correct telephone number was available for 35%. Perhaps unsurprisingly, our attempts to contact each of these project leaders yielded replies from only 20%; and only 15% provided useful information. A third possibility was the use of Google Scholar, which reads the entire paper, thus making it possible to search, manually or programmatically, by the EU grant codes. This appears to work well with the official codes, providing papers that reflect the projects’ initial abstracts. However, not all projects in the EU database had official grant codes. Our own project’s entry, for example, is missing correct grant code and contact information. Finally, we noticed the new ‘dissemination’ tool on CORDIS (http://cordis.europa.eu/pids/). We found this extremely complicated and confusing. The tool did not return any results for its own grant codes (although it recognizes the acronyms). Furthermore, although it may be suitable for finding highly specific terms associated with incomplete research, it is unsuitable for meta-analysis on large topic areas. The process of obtaining EU funding is extremely complex, requiring very detailed submissions and with complex reporting requirements. However, this is almost entirely related to the need to account for what money is spent on rather than what is achieved with it. We believe that it is time to redress this balance, by placing at least as great an emphasis on the outputs of the research as is presently placed on the inputs. On the basis of our experience, we make a series of recommendations both about the existing reporting system and the wider issue of disseminating EU-funded research (Box 2). The CORDIS database is in need of considerable investment and additional resources to allow it to be verified and updated regularly. The CORDIS database should contain more information on actual project achievements, rather than on what projects planned to do. Fields containing implications for policy-makers, the public, and other researchers should be introduced. Researchers should be required to post their final reports on websites linked to CORDIS, as well as links to abstracts of published papers arising from their work. Future calls within the Framework Program should require that research outputs be published in open access journals (although this will not apply to other research funded by European taxpayers, such as that within the Public Health Program). The data generated should, after suitable safeguards have been put in place, also be publicly available.8 Authors of studies should be contactable by including their email, ideally as an image file instead of a hyperlink to minimise risk of spam. Alternatively, the EU could provide all researchers with a personal identification code, like the PIN used by Canadian government funding agencies. There should be a requirement to cite a project’s code number to facilitate location of papers. (The Commission has been liaising with the National Library of Medicine and it now seems likely that PubMed will include a code for EU funded research). We are conscious that these recommendations could be perceived as placing an additional load upon researchers. However, we believe that there is also considerable scope to reduce their workload. We have long suspected that the often lengthy reports generated from EU research projects are judged more by their weight than the quality of their content. We can now confirm that, once the reports enter into the system, they are essentially lost. Instead, we propose that deliverables be reconceptualized as a smaller number of draft papers that, hopefully, will in due course be published and thus enter the public domain. This is already possible, as was the case in our project, but will require a shift in the mindset of those drafting grant applications. Finally, our collective experience of many EU-funded research projects is that the most important results often emerge some time after the formal end of the project. In part, this is because the priority given to writing lengthy final reports (which often focus on processes rather than outcomes) displaces the task of writing scientific papers. We propose that project co-ordinators be required to submit a final and very brief report, a year after each project is finished. We recognize that it may be difficult to enforce this, unlike the current situation where the final report is required to release the final 20% of funding. However, failure to do so could be taken into account in future applications. We also propose that the European Commission creates a small budget that might receive bids from research teams approaching the end of their projects to support additional dissemination (such as conferences or books) that might not have been anticipated earlier. This is a model that has been adopted successfully by other funders, such as the UK’s Economic and Social Research Council. The European Union’s research programme is an evolving process and the current seventh Framework Programme has addressed many of the criticisms of its predecessor.7 Future Framework programmes might usefully concentrate on how better to use the research it generates. However, this will also require a fundamental reassessment of how the DG Research operates. Contrary to some popular views, the resources devoted to administering many parts of the European Union are extremely sparse and it is apparent that the workload of DG Research staff is unsustainably high. Our analysis suggests that some of the money currently spent on research would more usefully be retained by the Commission to ensure that it can effectively disseminate its funded research. This paper was written as part of the Health Research for Europe (HR4E) project, funded by the European Commission's Directorate General for Research, within the sixth Framework Programme. Grant number: LSSM-CT-2007-037397. Conflicts of interest: None declared. We are especially grateful to Kevin McCarthy and Jan Paehler in DG Research for their support throughout this project and to Sir Iain Chalmers for sharing his ideas on dissemination of research with us.

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 9 citations exist upstream.

Papers that cite the same references — the relational fallback while embeddings are still backfilling.