State of peer review 2024
Discussion
3046 physical science researchers respondent to our survey which contained questions about their perceptions and experiences of peer review. Our sample had a wide geographical distribution and a male to female gender split of roughly 80% vs 20%. While this does not represent gender parity, it is broadly reflective of the gender split in the physical sciences.
We asked respondents how they felt the number of peer review invitations they receive has changed over the three years up to March 2024. Almost half of respondents reported an increase in the number of reviewer requests that they receive. There were no major differences by career stage or geography, with the exception of World Bank low-income country respondents, who were much more likely to report an increase in the number of requests they receive. In recent years, there has been a greater appreciation among publishers and editors for the need to include researchers based in low-and middle-income countries in the peer review process. This reported increase in invitations to low-income country respondents indicates that these inclusion efforts may be working. However, publishers should be wary of overburdening any group of researchers in particular.
When respondents were asked about the time they have available for peer review compared with the number of requests they receive, there was a noticeable difference between the responses in 2024 vs those in 2020. Respondents to the 2024 survey were much more likely to report that they either receive fewer requests than their time available to review or that they receive the right number of requests (84% compared with 74% in 2020). In recent years, IOP Publishing (along with many other publishers) has made a concerted effort to diversify and enrich the reviewer pool, to ensure that peer reviewers are representative of the physical sciences community as a whole. So while this result could represent an overall decrease in the burden of peer review, it could also be an indicator of a shift in the researchers who are invited to review for us and therefore a difference in the make-up of survey respondents between the two surveys.
There were significant geographical differences with regards to the time that respondents had available for peer review, with respondents in high-income countries much more likely to report that they receive more peer review requests than they had time for. Given that there were no major geographical differences in the overall volume of peer review requests, this is most likely to reflect different competing burdens and responsibilities for researchers in higher-income countries that preclude them from undertaking peer review.
In sub-analyses by career stage, respondents who work in industry consistently reported receiving fewer review requests and having more time available to review compared with respondents in academia. As publishers try to enrich their reviewer pools, they might consider inviting more industry experts to review.
The survey responses relating to bias in peer review highlighted the fact that many reviewers still feel that there are inherent biases in the peer review process, although the proportion of respondents reporting experiencing bias has decreased between 2020 and 2024. Geographical bias both for and against certain regions and countries was a common concern, as was bias for or against institutions, and bias against early-career researcher authors.
In 2021, IOP Publishing introduced the option of double-anonymous peer review on all of its owned journals. Authors submitting to these journals are now offered the option to submit their manuscript anonymously, with the intention that the reviewers will be unaware of the authors’ identity. The goal of double-anonymous peer review is to reduce bias in the peer review process and ensure that scientific manuscripts are judged solely on their own merit and not based on the identities of the authors. While full anonymisation is not always possible due to a variety of reasons (including pre-print servers, open data sharing, and authors clearly building on their own previously published work), studies have found that double-anonymous peer review at IOP Publishing has reduced bias in the peer review process.
While single-anonymous peer review is still the predominant model in the physical sciences, the results of this survey indicate that most respondents would be happy to review a double-anonymous paper, and would be happy to submit their own research under a double-anonymous model. It appears that double-anonymous is becoming more widely accepted in the physical sciences.
In terms of motivations to accept peer review invitations, our results show that not much has changed between 2020 and 2024. Interest in the paper and the reputation of the journal remain the most important factors for reviewers, indicating that publishers would do well to carefully match reviewers to papers according to their expertise and interests, and should aim to maintain high standards of peer review to maintain an engaged reviewer pool. As in 2020, the least motivating factor for respondents was cash or in-kind benefits, although this received a higher average score in 2024 than it did in 2020.
Similar to 2020, respondents were keen to receive feedback as part of the peer review process. Feedback on the final decision of the manuscript was the most popular feedback option, but respondents also cited feedback on the quality of their reviews as a potential motivating factor. In 2023, IOP Publishing became the first major publisher to offer reviewers feedback direct from the editorial team on the quality of their review. In this system, reviewers can opt in or out of receiving feedback, which is sent in the form of an evaluation on a scale of 1-5 alongside an evaluation matrix. A high proportion of reviewers (roughly 60%) have opted in to receive feedback on their reviews, and comments from the community have been overwhelmingly positive.
We also asked respondents to evaluate different initiatives and innovations in terms of how they might positively impact the peer review process. The highest-ranking response was “Improvements to online manuscript and review submission systems” followed by “More communication between authors, reviewers, and editors” and “Better and more accessible peer review training”. The latter response had increased drastically in the rankings since the survey was last conducted in 2020.
Following the 2020 survey, we sought to address the gap in peer review training by launching the Peer Review Excellence programme, which includes a fully accessible training element. Our training course can be accessed and completed by anyone globally here: www.iopexcellence.org, and since 2021 over 6000 people have signed up, indicating that there is strong demand for high-quality training.
A major development since our last survey in 2020 has been the release of open-source generative AI. We asked our respondents two questions about the impact that generative AI might have on peer review in future. The responses revealed polarisation in the research community’s views on generative AI, with 35% of respondents predicting a negative or very negative impact of generative AI, 29% predicting a positive or very positive impact of generative AI, and 36% predicting neutral or no impact. The free-text responses regarding the impact of generative AI on peer review reflected this polarisation in views.
Scholarly publishers have been working hard to create generative AI policies that protect the integrity of peer review, while also remaining pragmatic. Some researchers will use generative AI to some extent during the creation of their manuscripts, and we have also seen some reviewers using generative AI to write or augment their peer review reports. Currently, publishers have no way to accurately detect whether text has been generated by AI. At IOP Publishing, we have done extensive testing and research to find hallmarks of generative AI in manuscripts and peer review reports.
The use of generative AI to write or augment peer review reports raises a number of ethical issues, including data protection and breaches of confidentiality, and concerns about the veracity and accuracy of reviewer reports.
Given the rapidly changing landscape, publisher policies around generative AI need to be adaptable and fair. Currently, IOP Publishing does not allow the use of generative AI to write or augment peer review reports. We will also not accept generative AI tools to be named as authors of manuscripts, although we understand that some researchers will use these tools in the drafting of manuscripts, and we encourage authors to be open and transparent about their use of generative AI in the creation of their work.
Concluding remarks
Peer reviewers play a crucial role in advancing science and maintaining the integrity of the scholarly record. In the context of a changing technological and social landscape, it’s important that peer reviewers are they are being listened to, recognised, rewarded, and supported to undertake this important work.
The survey we conducted in 2020 helped us to design programmes and innovations that addressed the needs of the reviewer community. The results of this 2024 follow-up survey show that there are still ways that we can improve the experience of our reviewers.
Appendix
Please find here additional data supporting the State of Peer Review 2024 report.