Funders have an essential role to play in shaping research culture. cOAlition S is committed to fostering responsible and fair approaches to research assessment.  This intention is set out in the Plan S principles: when assessing research outputs during funding decisions, cOAlition S funders will make decisions based on the intrinsic merit of the work and not the publication channel, its impact factor (or other journal-level metrics), or the publisher”.  As part of this commitment, the funders support the core principle of the San Francisco Declaration on Research Assessment (DORA) that research should be assessed on its own merits and not on the venue in which it is published.

Alongside DORA, several other key statements set out considerations that should be taken into account, such as the Leiden Manifesto and the Hong Kong Principles. There are also national level statements and frameworks. For example, the Academy of Finland is aligning its assessment practices with the national recommendation for the responsible evaluation of researchers and recommendation for the responsible application of publication metrics.

cOAlition S funders are addressing this commitment in a variety of ways. Several funders have updated funding guidance and processes to ensure that panels do not rely on the use of journal-based metrics in funding decisions, and that reviewers and panels consider all research outputs. In addition to publications, preprints, datasets, software, inventions, educational products, and (editorial and reviewing) service to the field are now recognized. A range of qualitative impact measures has been adopted, including influence via policy and practice and contributions to society at large.  The funders are working to eliminate the inappropriate use of journal-based metrics. They are committed to ensuring that any use of bibliometrics or indicators is done in a responsible way, taking into account the key dimensions summarised in The Metric Tide as:

  • Robustness: basing metrics on the best possible data in terms of accuracy and scope
  • Humility: recognising that quantitative evaluation should support – but not supplant – qualitative, expert assessment
  • Transparency: keeping data collection and analytical processes open and transparent, so that those being evaluated can test and verify the results
  • Diversity: accounting for variation by field and using a range of indicators to reflect and support a plurality of research and researcher career paths across the system
  • Reflexivity: recognising and anticipating the systemic and potential effects of indicators and updating them in response.

cOAlition S has established a Research Assessment Task Group to help to identify, exchange, and align good research assessment practices. A “basket” of best practices will aid and improve the funders’ practical implementation of responsible research assessment. cOAlition S is committed to monitoring the effects of the Plan S policies that will seek to assess impacts on career progression, publishing practices, and changes in research evaluation. This complements the aims of responsible research assessment.

In November 2020, UK Research and Innovation (UKRI), in collaboration with the UK Forum for Responsible Research Metrics and the National Research Foundation (NRF) in South Africa, delivered a Global Research Council (GRC) Virtual Conference on Responsible Research Assessment. Ahead of the conference, the Research of Research Institute (RoRI) launched a working paper on “The changing role of funders in responsible research assessment: progress, obstacles and the way ahead”. This made clear that the focus now should be on implementing solutions for responsible research assessment.

The GRC conference, which had over 1,000 registered participants from across the globe, explored the role that funders play in supporting responsible research assessment and their impact on the wider ecosystem through their research assessment processes and criteria. Johan Rooryck, Executive Director of cOAlition S, presented the Plan S principles and explained how cOAlition S supports a positive research culture through research assessment criteria and processes. Further analysis of the conference’s core themes and discussions will be explored in a conference report to be published in 2021. Work is also underway to explore how the GRC’s Annual Meeting in May 2021 can build on these discussions and take the topic forward.

cOAlition S is focused on aiding the development of practical solutions and will seek to work in partnership with other stakeholders and initiatives.

A number of case studies from universities and national initiatives are available on the DORA website, showing how responsible research assessment and evaluation are being addressed in practice.

 

Examples of Principle #10 in practice