Peer Review Survey Insights 2020

Foreword

We are constantly looking at how we can overcome these challenges, improve the peer review process and offer more support to reviewers. This led us to carry out this survey, with support from Publons, to truly understand the reviewer experience and what motivates researchers to review papers. The survey captures reviewers’ attitudes depending on their career stage and geographical location, providing rich and practical insights that will help to improve the efficiency and quality of the peer review process.

Methodology

We surveyed over 1,200 researchers who had either performed a review or had been invited to review for an IOP Publishing journal between January 2018 and March 2020. Targeted campaigns were sent to reviewers from China in order to build a representative sample.

The responses were heavily gendered, with 86 per cent of respondents identifying as male. Whilst this is reflective of the author and reviewer demographics in physics, we know that we can play a part in addressing this gender imbalance and are consciously trying to encourage more female reviewers to work with us.

Responses by geographical location

Fewer than 10 per cent of respondents identified themselves as an early career researcher, with over half identifying as either an associate or full professor.

To streamline analysis throughout the report, we have combined respondents that identified themselves as “early career researcher” or
“postdoctoral fellow” into the category “early career researchers”, whilst “faculty members” and “associate professor or higher” are collectively referred to as “experienced researchers”.

What motivates reviewers?

The biggest motivator for reviewers is an interest in the paper

Reviewers were asked to score a variety of factors that motivate them to review, where a score of one was very low and five was very high.

Weighted average of responses (1-5)

It is clear from the responses that it is important for journal editorial teams to focus on matching reviewers to the right paper using their knowledge of the community as well as research-interest matching technologies. This will ensure reviewers are receiving interesting and relevant content and will reduce reviewer fatigue.

Besides an interest in the paper, the reputation of the journal is also a strong motivator for a reviewer. This may however be a correlation effect, as higher-reputation journals may send papers of higher quality which could be perceived as more interesting to review. Seeing work ahead of time is also a motivating factor with the review process being a good way stay up to speed with developments in a field.

Reviewing is a way to “…keep up with the literature, an opportunity to read
articles regularly.”

Reviewer interview, 7 April 2020

Peer review is a reciprocal process and reviewers wish to “…support the scientific process to keep it credible”. There is also a strong desire to maintain the process so that they can benefit from it when they have an article of their own to publish. “It’s how the whole system is set up”. (Reviewer interview, 3 April 2020)

Finally, recognition for peer review activities is a key motivating factor. For many early career reviewers, simply being invited to review on a particular topic may be recognition of the individual’s expertise in the field. Formally recognising peer review efforts through Publons and ORCID is discussed further on in this report.

Cash is not king

It is notable that 59 per cent of respondents said that in-kind benefits or cash payments provide little motivation to review. This is perhaps because, within physics anyway, there is no culture of payment for peer review. Most publishers offer discounts on open access article publication charges (APCs) as a reward for reviewing. However, at IOP Publishing we have seen very little take up of these discounts.

Regional variations in peer review motivations

Reviewers in China and India are generally more motivated by the key factors, particularly the prospect of building a reputation with the editor or journal and seeing work ahead of time. In China the reciprocation of peer review appears to be less of a motivational factor.

Motivations based on experience levels

Motivations remain broadly the same across experience levels, which is encouraging to see, with an interest in the paper remaining the number one motivating factor, regardless of the stage of the respondent’s career.

The only notable differences are that early career researchers are more motivated by recognition and building a relationship with the editor or journal, possibly ahead of potential authorship. Early career researchers also show more interest in in-kind benefits, although this remains a poor motivator.

Motivations based on gender differences

Female reviewers are more motivated by an interest in the paper and recognition than their male counterparts. While the reasons for this are unclear, it is possible that this could be driven by existing gender imbalances in academia.

Recognising and rewarding reviewers

Providing reviewers with feedback makes the biggest difference to the peer review experience

Reviewers value feedback, whether that is being notified about the final decision on the paper or commentary on the quality of the review.

Feedback most valued

Most peer review systems enable reviewers to be notified of the final decision on a paper. There is a feature in Publons that enables reviewers to track if and where a paper they have worked on has been published, even if the paper is transferred to a different publisher.

Being notified about how important a paper becomes in the field was moderately popular as a recognition mechanism. This may be because receiving notifications is not yet common with limited article metric sharing capabilities within peer review systems. However, responses show that there is at least interest in such a service.

Reviewers also want to know if their review was useful, whether they could have done better and whether the publication decision varied from their recommendation. Historically, publishers may have been hesitant to provide such feedback due to time pressures, but reviewers appreciate an understanding of whether they went right or wrong with their own assessment. Many peer review systems enable this functionality so now may be the time to address this. The responses show that reviewers overwhelmingly value feedback on the work they have provided. IOP Publishing will be exploring ways to improve the feedback loop with reviewers and will work with suppliers to develop better mechanisms for this.

Regional differences

Reviewers in India rate feedback on the final decision and feedback on the review quality higher than the general cohort. Anecdotal feedback from recipients of the IOP Publishing Reviewer Awards in India shows that researchers see benefit in reviewing for an established publisher and that they value feedback on their work to give them the opportunity to improve.

Reviewers in China and India rate notifications of publication metrics for the articles they reviewed higher than the general cohort.

Experience

Early career researchers value feedback on their work more than experienced reviewers. The difference is less than that of other categories though, so it seems that feedback is important and valued for all reviewers, no matter the stage of their careers.

Feedback on the final decision of the paper was valued equally across experience levels, while feedback on the quality of the review was significantly more important for early career reviewers. For an early career researcher, having access to other reviewer comments, and receiving commentary on their own reviews can prove to be invaluable for their research skills.

Rewards and recognition

Access to papers post publication and a personal subscription to the journal are popular motivation factors for reviewers. With the drive for open science, accessing previously reviewed papers may become automatic, but until then it is an opportunity for publishers to explore. A personal subscription to the journal may act as a reward and it may also help in the examination of other studies referred to by the author.

Just as recognition is an important motivating factor, credit through third parties such as Publons and ORCID is important for reviewers. With reviewing activity now being tracked through ORCID and Publons the number of ways to gain recognition for work has increased.

Many publishers offer discounts or waivers on APCs for reviewers, yet this is only a moderately popular motivation factor for reviewers. This may be related to motivation being largely driven by intrinsic factors such as interest in the paper, rather than monetary rewards.

Being named as a reviewer in the published article received a mixed response as a motivator. Respondents indicated concerns over conflicts of interest, as reviewers could recommend that the paper is accepted so that their name is featured in a prestigious journal. There are also common concerns over a lack of reviewer anonymity leading to the peer review process becoming compromised. Naming reviewers is not uniformly unpopular however, with early career researchers especially in favour of the concept. These results at least show that any developments in
this field may need to be carefully introduced with levels of optionality.

Reviewers in India and China showed significantly more support for being named as a reviewer in the published article, which suggests an openness towards other methods of peer review, such as transparent or fully open. This may become more apparent during the current IOP Publishing transparent peer review trial.

Regional differences

Reviewers in India rated credit through third parties as an important factor, which matches the need for recognition as a motivating factor in the region, relative to the general population. Reviewers in India and China report more impact from being able to access papers post publication. This
is a strong factor across all regions, which could indicate a lack of access to subscription journals.

Experience

Broadly, recognition in the form of formal rewards was less valued amongst early career researchers compared to feedback and awards and certificates. Early career researchers rate that third-party credit as more impactful in terms of recognition.

Early career researchers are also more in favour of being named as a reviewer on the published article, possibly reflecting a growing trend for transparency amongst later generations of reviewers. This may also reflect a greater desire for recognition in the early stages of a career.

Awards and certificates

Acknowledgement in journal end-of-year lists was the most popular form of recognition for all respondents. This potentially provides a relatively anonymised method of recognising a reviewer’s efforts amongst a cohort of other reviewers.

A certificate or badge recognising peer review activity was moderately popular across all respondents while reviewer awards and annual peer review award schemes received less interest. It is worth noting that there may be a sampling factor here. IOP Publishing runs an annual peer reviewer of the year awards scheme which receives positive feedback from nominees and award winners. Unfortunately, there will only ever be a small fraction of the reviewer population who experience receiving such awards.

Regional differences

Broadly speaking, reviewers in India valued all recognition in the form of awards and certificates compared to the general population. Journal or annual peer review award schemes were particularly important.

Reviewers in India and China rank certificates for passing training and recognition of peer review activities higher than the average. A peer review certificate is especially important for Indian reviewers.

Experience

Recognition in the form of awards and certificates is where we see the greatest divergence between early career and experienced reviewers.

Early career researchers generally rate awards and certificates higher than more experienced reviewers, especially certificates and badges for passing training. Publishers wishing to nurture and bring onboard the next generation of reviewers may wish to focus on this area.

Time available and reviewer workloads

Reviewers were asked “what best describes the time you have available for peer review?” While the majority of reviewers (55 per cent) stated that they receive the right level of review requests for their time available, a quarter stated that they receive too many requests.

Regional differences

Breaking down responses from specific countries shows that reviewers from these institutions are feeling the strain compared to the general population.

Just 12 per cent of Chinese reviewers state they receive too many review requests, with nearly 75 per cent stating that the balance is
right for them. However, Chinese reviewers are no more likely to state that they have more time available to review. Despite not being overloaded, publishers may wish to temper expectations of Chinese reviewers if review invitations to that region increase over time.

Indian reviewers are significantly more likely to state that they have more time to dedicate to reviewing and are less likely to state that they receive too many review requests.

Over 40 per cent of reviewers from Germany, USA and the UK stated that they received too many review requests in relation to the time available.

Variations based on experience

Experienced reviewers were also more likely to report receiving too many review requests in relation to time available. By contrast, 25 per cent of early career researchers reported having more time available to spend on peer review.

Volume of requests – what can be done?

The 2020 IOP Publishing development partnership with Publons aims to address issues around reviewer fatigue. The strategic aims of the partnership are based around addressing three core challenges of diversity, efficiency, and recognition and transparency.

Some other activities the partnership will work on over the year include:

• Initiatives to improve gender and geographical diversity in our reviewer pool
• Improving efficiency and consistency of the peer review process for authors and reviewers
• Introducing new innovations such as interacting with reviewers through WeChat

Time available to review – what can be done?

Publishers may wish to focus on removing points of friction for reviewers throughout the review process. This could include access to referenced papers, a smooth review submission experience and clear guidelines.

Bias in peer review

Perhaps the biggest demotivator in any field is a perception of bias in the peer review process. All researchers need to know they are competing on a level playing field. In recent years there have been high-profile studies published that indicate systematic bias against female authors and papers from emerging nations.

Despite what these studies show, most researchers said they have not experienced bias in the peer review process.

Instead, researchers were more likely to report bias related to poor practice:

• Conflict of interest was the most common problem identified by researchers (11 per cent of people reporting bias). Here, researchers believe that reviewers are working on a similar topic and therefore delaying or preventing publication in order to get there first.
• Issues related to reputation were listed by 12 per cent of researchers reporting bias. Researchers reported it being more difficult to correct errors in the work of senior authors and that it can be risky to their reputation to criticise certain topics.
• The experience of having a poor review was seen as bias by 10 per cent, with reviewers “not reading the paper” frequently mentioned.
• Editors accepting papers even when reviewers ask for revisions (politics), was raised by 10 percent of researchers.
• Reviewers asking for too many citations to their own work (reviewer citation manipulation) was raised by seven percent of reviewers.

Variations in reported bias

While academic literature on peer review supports the existence of bias against female reviewers and inexperienced reviewers from emerging nations, the survey data revealed an inverse trend. Male researchers and researchers from the UK and USA were more likely to report bias than female researchers and researchers from China and India. This may be related to inexperience or lack of awareness of potential bias, as logic dictates that the more papers a researcher publishes, the more likely they are to experience what they perceive as bias. This is supported by the breakdown of responses by experience levels, with more than 25 per cent of researchers at associate professor or a higher-level reporting bias, compared to just 17 per cent for early career researchers.

Concluding thoughts