Skip to main content

Your web browser is out of date. Please update it for greater security, speed and the best experience on this site.

Choose a different browser

Peer review survey insights



Estimated time to
read: 30 mins

Foreword

Antonia Seymour, Publishing Director

At IOP Publishing, we are dedicated to providing the best possible publishing experience and our
commitment to ensuring high-quality, timely, rigorous and ethical peer review is at the heart of that.

We have more than 60 staff worldwide managing the peer review process and we recently
created two new roles to enhance the support we provide. We now have a peer review engagement manager,
Laura Feetham,
who focuses on how we can train, mentor and support new and existing reviewers. Our research integrity
and
inclusion manager, Kim Eggleton, ensures that our approach to peer review meets the highest ethical
standards
and also works with our author and reviewer communities on matters relating diversity and
inclusion.

High-quality peer review has never been more important to validate the science we publish.
The rise of pre-prints has only heightened the expectations placed on peer review, both in
terms of speed of publication and capacity to weed out poor-quality science. The number of papers
needing review continues to increase, while the number of researchers willing to review has not kept
pace. This global trend has led to fatigue among experienced reviewers, who are being asked to review
more frequently.

We are constantly looking at how we can overcome these challenges, improve the peer review process and
offer more support to reviewers. This led us to carry out this survey, with support from Publons,
to truly understand the reviewer experience and what motivates researchers to review papers. The survey
captures reviewers’ attitudes depending on their career stage and geographical location, providing rich
and practical insights that will help to improve the efficiency and quality of
the peer review process.

Kim EggletonResearch Integrity and
Inclusion Manager

Laura FeethamPeer Review
Engagement Manager

Research methodology

We surveyed over 1,200 researchers who had either performed a review or had been invited to review for an
IOP Publishing journal between January 2018 and March 2020. Targeted campaigns were sent to reviewers
from China in order to build a representative sample.

The responses were heavily gendered, with 86 per cent of respondents identifying as male. Whilst this is
reflective of the author and reviewer demographics in physics, we know that we can play a part in
addressing this gender imbalance and are consciously trying to encourage more female reviewers to work
with us.

Responses by geographical location

 

Fewer than 10 per cent of respondents identified themselves as an early career researcher, with over half
identifying as either an associate or full professor.

To streamline analysis throughout the report, we have combined respondents that identified themselves as
“early career researcher” or
“postdoctoral fellow” into the category “early career researchers”, whilst “faculty members” and
“associate professor or
higher” are collectively referred to as “experienced researchers”.

 

Key Findings

Motivating factors

1st

interest in the paper

2nd

journal reputation

3rd

engagement with scholarly community

76%

of respondents have not experienced bias in the peer review process

59%

of respondents say that
in-kind or cash benefits provide very little motivation

40%

of German, US and UK reviewers receive too many review requests

55%

of reviewers receive the right level of reviewer requests

Feedback 

is the greatest recognition that reviewers wish to receive

12%  

of reviewers from China receive too many review requests

Awards and certificates

are more popular in India and China

25%

of early career researchers have more time available for peer review

What motivates reviewers?

The biggest motivator for reviewers is an interest in the paper

Reviewers were asked to score a variety of factors that motivate them to review, where a score of one was
very low and five was very high.

4.2

Interest in the paper

3.9

Reputation of the journal

3.8

Expectation of scholarly community

3.5

Reciprocation of peer reviews received

3.2

Seeing work ahead of time

3.2

Recognition

Weighted average of responses (1-5)

It is clear from the responses that it is important for journal editorial teams to focus on matching
reviewers to the right paper using their knowledge of the community as well as research-interest
matching technologies. This will ensure reviewers are receiving interesting and relevant content and
will reduce reviewer fatigue.

Besides an interest in the paper, the reputation of the journal is also a strong motivator for a
reviewer.
This may however be a correlation effect, as higher-reputation journals may send papers of
higher quality which could be perceived as more interesting to review. Seeing work ahead of time is also
a motivating factor with the review process being a good way stay up to speed with developments in a
field.

Reviewing is a way to “…keep up with the literature, an opportunity to read
articles regularly.
” (Reviewer interview 7 April 2020).

Peer review is a reciprocal process and reviewers wish to “…support the scientific process to keep
it
credible”.
There is also a strong desire to maintain the process so that they can benefit from
it when they have an
article of their own to publish. “It’s how the whole system is set up”. (Reviewer interview 3
April
2020)

Finally, recognition for peer review activities is a key motivating factor. For many early career
reviewers, simply being invited to review on a particular topic may be recognition of the individual’s
expertise in the field. Formally recognising peer review efforts through Publons and ORCID is discussed
further on in this report.

Cash is not king

It is notable that 59 per cent of respondents said that in-kind benefits or cash payments provide little
motivation to review. This is perhaps because, within
physics anyway, there is no culture of payment for peer review. Most
publishers offer discounts on open access article publication charges (APCs) as a reward for reviewing.
However, at IOP Publishing we have seen very little take up of these discounts.

Regional variations in peer review motivations

 

Reviewers in China and India are generally more motivated by the key factors, particularly the
prospect of building a reputation with the editor or journal and seeing work ahead of time. In
China the reciprocation of peer review appears to be less of a motivational factor.

Motivations based on experience levels

 

Motivations remain broadly the same across experience levels, which is encouraging to see, with an
interest in the paper remaining the number one motivating factor, regardless of the stage of the
respondent’s career.

The only notable differences are that early career researchers are more motivated by recognition and
building a
relationship with the editor or journal, possibly ahead of potential authorship. Early career
researchers also
show more interest in in-kind benefits, although this remains a poor motivator.

Motivations based on gender differences

 

Female reviewers are more motivated by an interest in the paper and recognition than their male
counterparts. While the reasons for this are unclear, it is possible that this could be driven by
existing gender imbalances in academia.

Recognising and rewarding reviewers

Providing reviewers with feedback makes the biggest difference to the peer review experience

Reviewers value feedback, whether that is being notified about the final decision on the paper or
commentary
on the quality of the review.

Feedback most valued

 

Most peer review systems enable reviewers to be notified of the final decision on a paper. There is a
feature in Publons that enables reviewers to track if and where a
paper they have worked on has been published, even if the paper is transferred to a different publisher.

Being notified about how important a paper becomes in the field was moderately popular as a recognition
mechanism. This may be because receiving notifications is not yet common with limited article metric
sharing capabilities within peer review systems. However, responses show that there is at least interest
in
such a service.

Reviewers also want to know if their review was useful, whether they could have done better and whether
the publication decision varied from their recommendation. Historically, publishers may have
been hesitant to provide such feedback due to time pressures, but reviewers appreciate an understanding
of whether they went right or wrong with their own assessment. Many peer review systems enable this
functionality so now may be the time to address this. The responses
show that reviewers overwhelmingly value feedback on the work they have provided.
IOP Publishing will be exploring ways to improve the feedback loop with reviewers and will work with
suppliers to
develop better mechanisms for this.

Regional differences

Reviewers in India rate feedback on the final decision and feedback on the review quality higher than the
general cohort. Anecdotal feedback from recipients of the IOP Publishing Reviewer Awards in India shows
that researchers see benefit in reviewing for an established publisher and that they value feedback on
their work to give them the opportunity to improve.

Reviewers in China and India rate notifications of publication metrics for the articles they reviewed
higher than the general
cohort.

Experience

Early career researchers value feedback on their work more than
experienced reviewers. The difference is less than that of other categories though, so it seems that
feedback is important and valued for all
reviewers, no matter the stage of their careers.

 

Feedback on the final decision of the paper was valued equally across experience levels, while feedback
on the
quality of the review was significantly more important for early career reviewers. For an early career
researcher, having access to other reviewer comments, and receiving commentary on their own reviews can
prove to be invaluable for their research skills.

Rewards and recognition

Access to papers post publication and a personal subscription to the journal are popular motivation
factors for reviewers. With the drive for open science, accessing previously reviewed papers may
become automatic, but until then it is an opportunity for publishers to explore. A personal subscription
to the
journal may act as a reward and it may also help in the examination of other studies referred to by the
author.

 

Just as recognition is an important motivating factor, credit through third parties such as Publons and
ORCID is
important for reviewers. With reviewing activity now being tracked through ORCID and Publons the number
of ways to
gain recognition for work has increased.

Many publishers offer discounts or waivers on APCs for reviewers, yet this
is only a moderately popular motivation factor for reviewers. This may be related to motivation being
largely driven by
intrinsic factors such as interest in the paper, rather than monetary rewards.

Being named as a reviewer in the published article received a mixed response as a motivator. Respondents
indicated concerns over conflicts of interest, as reviewers could recommend that the paper is accepted
so that their name is featured in a prestigious journal. There are also common concerns over a lack of
reviewer anonymity leading to
the peer review process becoming compromised. Naming reviewers is not uniformly unpopular however, with
early career
researchers especially in favour of the concept. These results at least show that any developments in
this field may
need to be carefully introduced with levels of optionality.

Reviewers in India and China showed significantly more support for being named as a reviewer in the
published
article, which suggests an openness towards other methods of peer review, such as transparent or fully
open. This
may become more apparent during the current IOP Publishing transparent
peer review trial.

Regional differences

Reviewers in India rated credit through third parties as an important factor, which matches the need for
recognition as a motivating factor in the region, relative to the general population.

 

Reviewers in India and China report more impact from being able to access papers post publication. This
is a strong factor
across all regions, which could indicate a lack of access to subscription journals.

Experience

Broadly, recognition in the form of formal rewards was less valued amongst early career
researchers compared to feedback and awards and certificates. Early career researchers rate that
third-party credit as more impactful in terms of recognition.

 

Early career researchers are also more in favour of being named as a reviewer on the published article,
possibly
reflecting a growing trend for transparency amongst later generations of reviewers. This may also
reflect a greater
desire for recognition in the early stages of a career.

Awards and certificates

Acknowledgement in journal end-of-year lists was the most popular form of recognition for all
respondents. This
potentially provides a relatively anonymised method of recognising a reviewer’s efforts amongst a cohort
of other
reviewers.

 

A certificate or badge recognising peer review activity was moderately popular across all respondents
while
reviewer awards and annual peer review award schemes received less interest. It is worth noting that
there may be a
sampling factor here. IOP Publishing runs an annual peer reviewer of the year
awards

scheme which receives positive feedback from nominees and award winners. Unfortunately, there will only
ever be a
small fraction of the reviewer population who experience receiving such awards.

 

Regional differences

Broadly speaking, reviewers in India valued all recognition in the form of awards and certificates
compared to the general
population. Journal or annual peer review award schemes were particularly important.

 

Reviewers in India and China rank certificates for passing training and recognition of peer review
activities higher
than the average. A peer review certificate is especially important for Indian reviewers.

 

Experience

Recognition in the form of awards and certificates is where we see the greatest divergence between early
career and
experienced reviewers.

 

Early career researchers generally rate awards and certificates higher than more experienced reviewers,
especially certificates and badges for passing training. Publishers wishing to nurture and bring onboard
the next
generation of reviewers may wish to focus on this area.

 

Time available and reviewer workloads

Reviewers were asked “what best describes the time you have available for peer review?” While the
majority of
reviewers (55 per cent) stated that they receive the right level of review requests for their time
available, a
quarter stated that they receive too many requests.

 
 

Regional differences

Breaking down responses from specific countries shows that reviewers from these institutions are feeling the strain compared to the general population.

 

Just 12 per cent of Chinese reviewers state they receive too many review requests, with nearly 75 per
cent stating that the balance is
right for them. However, Chinese reviewers are no more likely to state that they have more time
available to
review. Despite not being overloaded, publishers may wish to temper expectations of Chinese reviewers if
review
invitations to that region increase over time.

Indian reviewers are significantly more likely to state that they have more time to dedicate to reviewing
and are less
likely to state that they receive too many review requests.

Over 40 per cent of reviewers from Germany, USA and the UK stated that they received too many review
requests in relation to the
time available.

Variations based on experience

Experienced reviewers were also more likely to report receiving too many review requests in relation to
time
available. By contrast, 25 per cent of early career researchers reported having more time available to
spend on peer review.

Volume of requests – what can be done?

The 2020 IOP Publishing development partnership with Publons aims to address issues around reviewer
fatigue. The
strategic aims of the partnership are based around addressing three core challenges of diversity,
efficiency, and
recognition and transparency.

Some other activities the partnership will work on over the year include:

  • Initiatives to improve gender and geographical diversity in our reviewer pool

  • Improving efficiency and consistency of the peer review process for authors and reviewers

  • Introducing new innovations such as interacting with reviewers through WeChat

Time available to review – what can be done?

Publishers may wish to focus on removing
points of friction for reviewers throughout the review process. This could include access to
referenced papers, a smooth review submission experience and clear guidelines.

 
75

Respondents said that getting access to related/cited papers is the primary challenge

101

Respondents said that having to judge novelty or new findings was the primary challenge

69

Respondents cited the language aspect of papers as the primary challenge

76

Respondents requested better or more specific guidance on what is required in a review

51

Respondents requested that editors be more “more courageous” in rejecting work pre-review

52

Respondents asked that they be sent papers more appropriate for their research expertise

Bias in peer review

Perhaps the biggest demotivator in any field is a perception of bias in the peer review process. All
researchers
need to know they are competing on a level playing field. In recent years there have been high-profile
studies
published that indicate systematic bias against female
authors
and
papers from emerging
nations.

Despite what these studies show, most researchers said they have not experienced bias in the peer review
process.

76%

Have not experienced bias in the peer review process

8%

Reported bias due to geographic location

2%

Reported bias due to gender

Instead, researchers were more likely to report bias related to poor practice:

  • Conflict of interest was the most common problem identified by researchers (11 per cent of people
    reporting bias).
    Here, researchers believe that reviewers are working on a similar topic and therefore delaying
    or preventing
    publication in order to get there first.

  • Issues related to reputation were listed by 12 per cent of researchers reporting bias.
    Researchers reported it being
    more difficult to correct errors in the work of senior authors and that it can be risky to their
    reputation
    to criticise certain topics.

  • The experience of having a poor review was seen as bias by 10 per cent, with reviewers “not
    reading the paper” frequently mentioned.

  • Editors accepting papers even when reviewers ask for revisions (politics), was raised by 10 per
    cent of researchers.

  • Reviewers asking for too many citations to their own work (reviewer citation manipulation) was
    raised by seven per cent
    of reviewers.

Variations in reported bias

While academic literature on peer review supports the existence of bias against female reviewers and
inexperienced
reviewers from emerging nations, the survey data revealed an inverse trend. Male researchers and
researchers from
the UK and USA were more likely to report bias than female researchers and researchers from China and
India. This
may be related to inexperience or lack of awareness of potential bias, as logic dictates that the more
papers a
researcher publishes, the more likely they are to experience what they perceive as bias. This is
supported by the
breakdown of responses by experience levels, with more than 25 per cent of researchers at associate
professor or a higher-level reporting bias, compared to just 17 per cent for early career researchers.

Concluding thoughts

David Evans, Peer Review Product Manager

We hope you have found the report an interesting read and want to conclude by thanking everyone who took
part in the
survey for their valuable thoughts and insights.

The rigour of published articles relies on the peer review process. Sitting at the heart of scholarly
research, the
demand for high-quality peer review continues to increase, yet the approaches and processes to
delivering it have
not evolved at the same rate. As publishers, we must improve our practices, address reviewer concerns
and evolve our
approach in line with their needs.

We are already making changes based on the insights we have gleaned from this report: changes to the way
we engage
reviewers, to who we approach to review, and to how we support and train early career researchers. We
look forward
to providing regular updates on the progress we are making.