PEER REVIEW
validity
originality
significance
Standing up for Science 3
The nuts and bolts
Peer review is the system used to assess the quality of scientific
research before it is published. Independent researchers in the
same field scrutinise research papers for validity, significance
and originality to help editors assess whether research papers
should be published in their journal.
A guide for early career researchers
Peer review now results in over 1.5 million scholarly
articles published each year and is fundamental to
the integration of new research findings in hundreds
of fields of inquiry. For scientific knowledge to
progress scientists need to share their research
findings with other scientists and this is done through
publishing in peer reviewed scientific journals. Peer
review is also the tool used for reviewing grant
proposals for research funding.
Peer review provides a system to select which
research should be brought to the attention of other
researchers. It also gives authors feedback to
improve the quality of their research papers before
publication. The peer review system judges the
validity, significance and originality of the work,
rather than who has done it. Because it indicates that
research has been scrutinised by independent experts
in the field, peer review is also an important
consideration for policy makers, reporters and the
public when weighing up research claims and
debates about science.
Peer reviewing is particularly important for early
career researchers because it allows them to gain
insights into other developments in their research
area and play a greater role in their research
community. Reviewers develop their own research,
writing and data presentation skills, and their ability
to look at their own work objectively.
However, there has been growing talk of “a crisis in
peer review” – with concerns raised about the global
expansion of scholarly research, and to particular
incidents of flawed papers making it into print,
leaked email exchanges showing researchers trying
to influence the process; as well as the mounting
pressures on researchers to get grants and publish
papers, leaving little time to review papers.
INTRODUCTION
JULIA WILSON
VoYS co-ordinator
1
In Voice of Young Science (VoYS)
workshops, early career researchers raised
questions about how to get involved in
reviewing, how to be sure of doing a good
job and what to expect as authors and
reviewers.
This is a nuts and bolts guide to peer review
for early career researchers written by
members of the VoYS network1. Using a
collection of concerns raised by their peers,
the VoYS writing team set off to interview
scientists, journal editors, grant bodies’
representatives, patient group workers and
journalists in the UK and around the world
to find out how peer review works, the
challenges for peer review and how to get
involved.
We have not avoided criticisms of the peer
review process in this guide but rather
entered into the debate, asking journal
editors and reviewers some challenging
questions about scientific fraud and
plagiarism going undetected; issues of trust
and bias; ground-breaking research taking
years to publish and the system benefiting a
closed group of scientists.
CONTENTS
THIS GUIDE WILL
HELP EARLY
CAREER
RESEARCHERS
UNDERSTAND:
1. How the peer
review process
works
2. Some of the
limitations of peer
review
3. The role of peer
review in society
BEHIND
THE SCENES1.
PEER REVIEW
WARTS AND ALL2.
PEER REVIEW
FOR THE PUBLIC3.
FURTHER
RESOURCES4.
2
P3
P17
P22
P251Voice of Young Science (VoYS) is a network of early career
researchers who stand up for science in public debates about
science. Further information at
www.senseaboutscience.org/pages/voys.html.
There are three key roles in peer review: the authors
who write the papers, the reviewers who provide expert
opinions and advice, and the editors who make the
decisions.
Figure 1: Diagram of a “typical” peer review process (there are many varieties)
BEHIND
THE
SCENES
Author submits
article to journal
Journal Editor
screens paper
Editor assessment
of reviews
Author makes
revisions
Accepted no
revisions required
Rejected after
screening
Reviewer
Reviewer
Rejected
3
To gain an insight into how peer review works, we asked editors
from a variety of peer reviewed journals, how they select reviewers,
reduce potential bias and make decisions about which manuscripts
to publish.
The
Editors
WHAT DO YOU DO WHEN A PAPER IS SUBMITTED?
HOW DO YOU THEN SELECT REVIEWERS?
“I have a whole load of manuscripts coming to me each day – far more
than I can publish. So I have to look at them and decide firstly, is this
paper relevant to the journal I’m editing? (Is it groundbreaking etc.) I’m
looking for the best papers, but I often know very little about the nitty
gritty of the research area. It is the experts that I send the paper out to
review to, who know the subject area well and can help me make a
judgement.”
CHRIS SURRIDGE
Chief Editor and Associate Publisher of Nature Protocols
4
“If I know the field intimately I will select people to review from my
knowledge base. If I don’t know the field, I select reviewers by searching
‘PubMed’ (a free online database of citations and abstracts) for authors
of similar research or pick suitable authors from the bibliography of the
paper. I don’t think it makes sense to carefully and precisely select and
invite only verifiable world leaders. Most luminaries are often too busy,
and the process of selection becomes far too slow.”
DR MICHAEL CURTIS
Editor-in-Chief of the Journal of Pharmacological and Toxicological Methods
“When your paper is submitted, we first of all look through it briefly to
check the format and length, the clarity of the discussion, research
methods and overall fit with the journal. This is a fairly quick process –
around two weeks or so. If it passes this 'desk review' procedure, we
then send it out for full review to subject experts.”
ROBERT BLACKBURN
Editor-in-Chief of the International Small Business Journal (ISBJ)
“We invite several reviewers in order to get a view which is independent
from the editorial team. If the reviewer and the assigned editor agree
that a paper should be rejected, we reject. But if there is reasonable
support, then we start a confidential online discussion with additional
editors. Usually it becomes clear very quickly whether a paper is going
to be accepted or rejected, but if there is no clear consensus, then as
Editor-in-Chief, I make my own assessment and provide a
recommendation to the handling editor.”
“To reach a decision on a paper, we take into consideration a
combination of the reviewers’ opinions and our editorial judgement. In
addition to looking at the broader recommendations made by the
reviewers, we think about the specific scientific points they raise, in light
of their areas of expertise, the feasibility of any requested revisions, and
the effects these revisions may ultimately have on the overall conceptual
interest and quality of the paper. All of these considerations factor into
our overall view of the appropriate next steps for the paper.”
Many journals have an editorial team with an editor-in-chief and a number of scientific
editors who are assigned responsibility for the peer review of individual papers. These
journals often hold discussions before accepting a paper.
Once the reviews have been submitted, it’s decision time.
Peer review is not a democratic voting system. It is the editor
who makes the final decision based on all the information
available to them.
HOW DO YOU REACH THE FINAL DECISION ON A PAPER?
HOW DO YOU OVERSEE THE PROCESS WITH YOUR EDITORIAL TEAM?
“Finding subject reviewers is a careful procedure because it is voluntary
and anonymous. We find these experts from our Editorial Board plus
others – you may have cited somebody extensively and we may ask
them, or we use our database of previously published authors and
reviewers. The ISBJ also provides you with the opportunity to suggest
possible reviewers – obviously not your friends or colleagues!”
ROBERT BLACKBURN
Editor-in-Chief of the International Small Business Journal (ISBJ)
DR MARIE BAO
Associate Editor, Developmental Cell, Cell Press
PROFESSOR PHILIP STEER
Editor-in-Chief, of BJOG: An International Journal of Obstetrics
and Gynaecology
5
The benefits of reviewing are diverse: from
improving your critical thinking, giving and
receiving feedback and gaining insights to
improve your future publications. Reviewing is
an essential skill to develop as a researcher.
2
Results from the 2009 Peer Review Survey: Sense About Science with support from Elsevier carried out
one of the largest ever peer review surveys of over 4000 authors and reviewers:
www.senseaboutscience.org/pages/peer-review-survey-2009.html
WHAT DO THEY HAVE TO SAY
ABOUT THE BENEFITS OF BEING
A REVIEWER?
WHY DO YOU REVIEW?
“Partly because it is an accepted part of membership in the academic
community. But also, it is always interesting to see the latest work in my
particular specialist areas and be able to comment on it and hopefully
sometimes improve it prior to publication; to act as a gatekeeper for
quality in an area of science that I know about and care about.”
DR STEPHEN KEEVIL
Medical Physicist, King’s College London
Enter the reviewers…
6
review because they
like playing their part
as a member
of the academic
community2
90%
just enjoy seeing
other papers
and being able to
improve them2
believe that their last
paper was improved
through the
peer review
process2
Almost
all researchers
85% 91%
When accepting the invitation to review you are agreeing to provide a fair, robust and
timely critique that is useful for the authors in improving their manuscript (whether or not
the journal accepts the manuscript).
Before you accept to review a paper, ensure you can submit within the time frame
because slow review times are a source of frustration for authors. Many journals record
how long a reviewer has taken to submit a review. If they are frequently very slow, editors
will take this into account and avoid inviting the reviewer again. Some journals also rank
your review once it is submitted, so if you do a good job; you are likely to be invited
again.
If, after agreeing to review, you find that you will not be able to complete the review in
the agreed time frame, contact the journal and let them know.
If you have any conflicts of interest – for example, you work closely with the author or are
in direct competition – you must declare these to the editor. If you are unable to accept
the invitation to review, suggestions of alternative reviewers are welcomed by editors.
“When reviewing, try to remember that you are an author too and be
professional and constructive in your approach. That can be hard but
don’t let your inner nitpicker get the upper hand. Leave 24 hours
between reading the manuscript and writing your review, to allow time
for your reasonable self to rise to the fore.”
SOME TIPS FOR NEW REVIEWERS PLEASE!
STEPHEN CURRY
Professor of Structural Biology, Imperial College London
WHAT CRITERIA DO YOU LOOK FOR?
“For me it is the originality of the work, the importance of the questions
addressed, the appropriateness of the techniques used, the quality of
the data and the reliability and significance of the conclusions that are
the most important criteria.”
PROFESSOR MIKE CLEMENS
Biochemistry & Molecular Biology, University of Sussex
7
If the science is sound but the language is poor, some reviewers may suggest edits,
whereas others might flag up to the editor that the paper needs an English language edit.
If the language is so poor it is difficult to assess the science you might recommend the
author improves the language and resubmit. There are English rewriting services available.
8
Does the paper fit the standards and scope of the journal it is being considered for?
Is the research question clear?
Was the approach appropriate?
Are the study design, methods and analysis appropriate to the question being studied?
Is the study innovative or original?
Does the study challenge existing paradigms or add to existing knowledge?
Does it develop novel concepts?
Does it matter?
Are the methods described clearly enough for other researchers to replicate?
Are the methods of statistical analysis and level of significance appropriate?
Could presentation of the results be improved and do they answer the question?
If humans, human tissues or animals are involved, was ethics approval gained and was the
study ethical?
Are the conclusions appropriate?
Aside from assessing the title, abstract, English language of the article and references,
reviewers assess the scientific quality of the work.
QUESTIONS REVIEWERS ASK
“When it comes to clinical trials and epidemiology papers, statistical
literacy is an important issue.”
DO I NEED TO GET UP TO SCRATCH WITH MY STATS?
DR STEPHEN KEEVIL
Medical Physicist, King’s College London
Most experienced peer reviewers have ‘learnt on the job’. If you are reviewing for the first
time, it is a good idea to ask an experienced reviewer with an analytical approach to be
your mentor.
Research groups and medical departments often hold their own 'journal club' where they
discuss a recent paper. This allows the group to keep up-to-date with scientific
developments and develop skills to critically appraise research papers that will be useful
when reviewing.
Some journals (eg. the EMBO Journal, BMJ Open) publish reviewers’ reports alongside
papers which can be useful for inexperienced reviewers to look at.
Once a decision has been made, journals often let reviewers know whether they
accepted or rejected the paper, and send them a copy of the other review(s). This allows
you to see the assessments and opinion of other experts and whether there is anything
you have missed in your own review. It can also help you judge whether you were too
stringent for the journal or too lenient. It can sometimes take a few attempts to gain a
sense of what the acceptance threshold is for a particular journal as each journal is
different.
Papers can go through several rounds of peer review, when a paper is rejected, the
author will in most cases submit it to another journal. The new journal editor will then
send the paper out to new reviewers. There is concern amongst the scientific community
that this leads to “wastage” of reviews as previous reviews are not always taken into
consideration.
9
“Most journals provide online guidelines for reviewers but in my
experience little other training is available. The skills are largely learned
from colleagues and mentors in the reviewer's own department.”
PROFESSOR MIKE CLEMENS
Biochemistry & Molecular Biology, University of Sussex
“When I started reviewing I had no formal training, but I did get
invaluable guidance from senior staff. Now there are also training days
and web courses which give advice on the structure and content of a
review, and, importantly, the expectations of the editor.”
IS THERE ANY TRAINING?
DR DEIRDRE HOLLINGSWORTH
Epidemiologist, Imperial College London
10
IS ANYTHING BEING DONE TO PREVENT “WASTAGE” OF REVIEWS?
“Cascading peer review (a.k.a. ‘waterfall peer review’) is when a paper that has been
rejected after peer review, is passed to another journal along with the reviewers’ reports.
The peer review process at the second journal can be kept relatively short because the
Editor considers the reports from an earlier round of peer review, along with any new
reviews. Variations on this process exist, according to the type of journal – but essentially
reviews can “cascade” down through various journals.”
DAN MORGAN
Executive Publisher of Psychology & Cognitive Science
Being a successful researcher involves
developing many skills including
reviewing the work of others as part of
the peer review process.
This skill will help
you in many
employment
destinations,
not just
research.
HOW THE VITAE RESEARCHER DEVELOPMENT
FRAMEWORK CAN HELP YOU WITH PEER REVIEW
These are further broken down into a number of characteristics, which you would be
developing through peer review including:
reputation and esteem
collegiality
publication
knowledge base
critical thinking and analysis
networking and responsiveness to opportunities
time management skills
continued professional development
The Vitae Researcher Development Framework (www.vitae.ac.uk/rdfresearcher) is a
guide to identify your strengths and priorities for professional and career
development. It sets out the knowledge, behaviours and attributes of successful
researchers and assists you in achieving higher levels of development.
The Framework is made up of four domains, which encompass
knowledge and intellectual abilities
personal effectiveness
research governance and organisation
engagement, influence and impact
11
“As a researcher it’s easy to get stuck into only thinking about the
knowledge and skills that are specific to your research field. The
Researcher Development Framework emphasises broadening your
horizons and identifying strengths and skills, such as those involved in
peer reviewing, that will help you become a better researcher.”
DR DANIEL WEEKES
Research Associate, Kings College, University of London
12
Peer review varies widely depending on the research field in terms of what reviewers are
looking out for and the time the process takes (in mathematics, peer review can take
years whereas in biomedical subjects it can take just weeks).
In some fields, like physics, it is more common to put research online in a subject
repository (such as ArXiv) before it is submitted to a journal. This allows the research to
be circulated and commented on before it is subject to peer review – whereas within
medicine there are ethical concerns about research being accessed before it has been
peer reviewed.
We asked Tommaso Dorigo, experimental particle physicist at CERN, to describe the
process in his field:
“In my opinion, in experimental High-Energy Physics (HEP), most
scientific papers could well do without external review. HEP collaborations
count dozens, and in a few cases thousands, of collaborators.
Each of them is responsible for what gets published and is entitled to
take part in the review process before a paper is sent to a peer
reviewed scientific journal. So a powerful internal screening blocks
anything that is even remotely questionable before it reaches a
journal.”
DR TOMMASO DORIGO
Experimental particle physicist at CERN
Some peer reviewed journals are tracked by Thomson Reuters and awarded
an impact factor, calculated annually. Impact factor is a measure of the
number of times the "average article" in a journal has been cited in a
particular year.
Peer review is not a one-size fits all system; there are variations across journals and
research fields.
Acceptance rates at journals vary widely with some only accepting a small percentage of
papers submitted. These journals tend to have extremely stringent requirements for
publication. For example, very general and high impact factor journals such as Science
or Nature will reject many good quality research papers if the editor feels the research is
not ground-breaking enough. Others, such as PLOS ONE (published by the Public
Library Of Science), use a peer review process that does not judge significance or
originality, but will publish all papers that meet the necessary standards of scientific
rigour. There are also smaller, more specialist journals which do not receive many
submissions and so the competition to publish is not as high. The average acceptance
rate for journals is 50%.
WHAT ARE THE DIFFERENT TYPES OF PEER REVIEW?
SINGLE-BLIND REVIEW
The reviewers know who the authors are, but the authors do not know who the reviewers
are. The most common system in science disciplines.
This allows reviewers to provide honest, critical reviews and opinions without fear of
reprisal from the authors.
Lack of accountability, allows unscrupulous reviewers to submit unwarranted
negative reviews, delay the review process and steal ideas.
DOUBLE-BLIND REVIEW
The reviewers do not know who the authors are, and the authors do not know who the
reviewers are. Main form of peer review used in the humanities and social sciences.
Reduces possible bias resulting from knowing who the authors are or where they
come from, work assessed on its own merits.
Involves some effort to make sure manuscripts are anonymised, reviewers can often
guess who the authors are (particularly if the authors have cited many of their own
papers), information important for a complete critical appraisal is missing.
DR IRENE HAMES
Editorial Consultant and author of Peer Review and Manuscript
Management in Scientific Journals
WHAT ARE THE DIFFERENT TYPES OF PEER REVIEW?
13
OPEN REVIEW
At its most basic, reviewers know who the authors are and the authors know who the
reviewers are. It can also mean inclusion of the reviewers’ names and/or reports
alongside the published paper, comments from others [subject community or wider
public] at pre-publication stage, or various combinations of these.
Greater accountability and reduced opportunity for bias or inappropriate actions.
Reviewers can be given public credit for their work.
Potential reviewers may be more likely to decline to review. Revealing reviewer
identity may lead to animosity from authors, damaged relationships and
repercussions for job prospects, promotion and grant funding.
DO YOU THINK KNOWING THE NAME OF THE AUTHOR AFFECTS THE
REVIEWER’S DECISION?
“It is probably impossible to ignore the effect of the author’s name,
whether they be an unknown or a big-shot scientist. By acknowledging that
potential impact, you can mitigate the most disturbing effects. Remember
that your job as a reviewer is to judge the work, not the scientist.”
STEPHEN CURRY
Professor of Structural Biology, Imperial College London
“Double-blind peer review can work effectively for some editors and
journals. For others, however, it doesn’t. It’s been shown that reviewers
can often – in around half of cases – identify who the authors are, and
the internet and online searching have increased the chances of this
happening. This is causing some journals in disciplines where
double-blind review has been the norm to move to single-blind review.
There are also concerns that some potential competing interests of
authors and other factors that might be important in assessing work are
not available in double-blind review.”
DR IRENE HAMES
Editorial Consultant and author of Peer Review and Manuscript
Management in Scientific Journals
IS THE DOUBLE-BLIND SYSTEM EFFECTIVE?
14
“At Diabetes UK we use peer review to ensure that the research we fund
will help to improve the lives of people living with diabetes. It helps us
know why an area of research is important and needs further
investigation and it also helps identify reasons why a research proposal,
that at first seems a good idea, might not be suitable for funding.”
DR IAIN FRAME
Director of Research, Diabetes UK
PEER REVIEW FOR FUNDING APPLICATIONS
Peer review is also used to assess scientists´applications for research funds. Funding
bodies seek expert advice on a scientist’s proposal to select which projects to fund.
15
Dr Liz Philpots thinks early career researchers should get involved in
peer reviewing grant applications as well as journal papers:
“If it’s your area, put yourself forward for peer reviewing grant
applications – and say [to your supervisor] I’d like to do this one. That’s
the only way to get experience.”
DR LIZ PHILPOTS
Head of Research at the Association of Medical Research Charities
“Based on the 2009 peer review survey results it is clear that reviewers
would like to be rewarded. The question is how should they be
rewarded? In the survey most reviewers indicated that they would like to
receive payment in kind for their reviews. Publishers are keen to do this
in a sustainable way and there are currently a variety of initiatives in
place on journals, including giving certificates to reviewers or providing
accreditation (CME/CPD points). Elsevier provides reviewers free access
to its Abstracting and Indexing service Scopus. Also popular among
reviewers is receiving an ‘Acknowledgement in the journal’, something
more and more journals are now doing.”
ADRIAN MULLIGAN
Deputy Director, Research & Academic Relations, Elsevier
SHOULD REVIEWERS BE REWARDED?
Many journals provide recognition to reviewers by publishing their names in
the journal as part of an annual list. Some journals send a certificate to
congratulate and express their appreciation to their best reviewers, clinicians
can claim CPD points for reviewing. However, journal editors have made the
case that the many hours of important work peer reviewers contribute need
to be recognised more formally by interview panels and researchassessment
exercises.
16
COULD JOURNALS REWARD REVIEWERS FINANCIALLY?
“I don't think so. This may encourage some people to review papers for
which they are not really qualified. However some other form of
recognition of the work involved, such as free online access to papers
published in the journal for a year, might be appropriate.”
PROFESSOR MIKE CLEMENS
Biochemistry & Molecular Biology, University of Sussex
Just as a washing machine has a quality kite-mark, peer review is a kind of
quality mark for science. It tells you that the research has been conducted
and presented to a standard that other scientists accept. At the same time, it
is not saying that the research is perfect (nor that a washing machine will
never break down).
“Bad papers sometimes make it through peer review and the system is
not set up to catch outright fraud. However, it acts as a useful first
barrier to junk science and journalists should treat information from
non-peer reviewed sources accordingly.”
JAMES RANDERSON
Environment and Science News Editor at the Guardian
“It's a good thing scientists are mostly honest, because peer review
offers the greatest possible temptation to steal ideas, to show favour to
former students, to boost favoured theories, or to do down rivals.
Honest they may be but they aren't saints, so we must expect all of these
things to happen from time to time.”
NIGEL HAWKES
Straight Statistics
“Regardless of its weaknesses, peer review is something the scientific world
cannot do without.”
PROFESSOR MAMMO MUCHIE
Editor of the African Journal of Science, Technology,
Innovation and Development
SO IS PEER REVIEW EFFECTIVE?
Peer review is not a perfect system.
It relies heavily on trust, and as
scientists are human like the rest
of us, there will always be cases
of misconduct.
PEER REVIEW
WARTS AND ALL
17
Reviewers could potentially slow down the publication of a paper to enable
them to get their paper out first. However, reviewers are given a deadline to
submit their review. If they are very late then journals will invite an expedited
review from a backup reviewer or consider the reviews they already have
in-hand at an editors’ meeting to minimize the delay for the authors.
One criticism of peer review is that it “shuts down new ideas” as research
that goes against the status quo may be rejected by reviewers. We put this
issue to the experts:
“Perhaps we do. It is easy to find plausible reasons to reject a paper,
especially at the highly competitive end of the market. If a reviewer has
a vested interest or a conflict of interest this is rarely disclosed. Indeed,
any 'expert' in the field must be a rival by definition, and conflicted by
definition. Yet we trust their judgements.”
DR MICHAEL CURTIS
Editor-in-Chief of the Journal of Pharmacological and Toxicological Methods
BUT WHAT DO EDITORS THINK? DO WE TRUST REVIEWS TOO MUCH?
“Reviewers are trusted to deliver an opinion but the editor knows this to be
subjective and so will carefully consider this when making a final decision
on a paper. And journals rarely accept papers based on only one review.”
COLLETTE TEASDALE
Development Editor – Economics Journals, Routledge Journals,
Taylor & Francis Group
“Rather than shutting down new ideas, the process of peer review
should mean that they are carefully considered and subject to close
scrutiny before being released to a wider audience. Often the processes
of peer review itself can specifically enhance a paper and the ideas it
seeks to communicate.”
COLLETTE TEASDALE
Development Editor – Economics Journals, Routledge Journals,
Taylor & Francis Group
CAN WE PREVENT REVIEWER BIAS?
18
New research that goes against current thinking might take longer to pass
peer review, but if it is scientifically sound, it will eventually be published.
We often hear about cases of fraud going undetected. But can peer review
ever really detect fraud?
“Fundamental physics sometimes advances with the presentation of ideas
which may sound crazy at first. This exposes the field to being hijacked by
deranged minds with their own “theory of everything” in their pocket. It
can be difficult for a reviewer to know whether a study is worthy of
publication and so there is a risk that reviewers decide on the basis of their
personal biases and turn down good work, or let crazy papers pass.”
DR TOMMASO DORIGO
Experimental particle physicist at CERN
“There have been numerous cases where highly original and
controversial ideas have been blocked for years before they have been
accepted, published and become popular.”
PROFESSOR MAMMO MUCHIE
Editor of the African Journal of Science, Technology,
Innovation and Development
“If a fraudster makes up data carefully, detection is very difficult.
However, made up data often include impossible enumeration. It is
astonishing how stupid fraudsters can be. I have seen published
photographs recoloured and relabelled as new data; blots that have
been touched up; numerical data that defy the laws of mathematics;
non-use of randomization; an absence of blinding; and wildly unequal
group size. Underpowered studies with meaningless statistical analysis
are also all too common. Mostly this is fraud by ignorance, but to
present such works as meaningful experimental data is fraud
nevertheless; it should be detected by peer review but it clearly escapes
detection in many cases.”
DR MICHAEL CURTIS
Editor-in-Chief of the Journal of Pharmacological and Toxicological Methods
19
The Committee on Publications Ethics (COPE) is an international forum for
editors of peer reviewed journals who discuss all aspects of publication
ethics. They have developed best practice flowcharts for editors on how to
handle cases of research and publication misconduct including plagiarism
and research fraud as well providing guidance on how editors can
responsibly carry out peer review.
“Unfortunately, the peer review process often doesn’t pick up plagiarism
as this would require the reviewer to know about every research paper
published on the subject area (and remember them!). However, journals
use a plagiarism checker that produces a report highlighting the
similarities with published papers. Reviewers can carry out their own
similar check using etBlast, a free database where they can paste the
abstract and see which papers are similar. This process is also useful to
help reviewers see where the paper fits within published literature, as
well as how novel a paper is.”
ELIZABETH HAY
Managing Editor, RCOG Journals
If someone sets out to falsify data, there is sometimes no way of knowing this until the
paper is published and others in the scientific community scrutinise and try and repeat the
work. Publication of a peer reviewed paper is just the first step: once a paper is
published, findings and theories must go on to be re-tested and judged against other
work in the same area. Some papers’ conclusions will be disputed or superseded after
further research is published. In a sense, long-term, peer review is a self-correcting
process.
If a researcher discovers there is a mistake in their published paper, the online version of
record cannot be altered in any way, but a correction (corrigendum) is published to
appear alongside the paper online.
If other researchers disagree with aspects of a published paper, or have identified flaws,
they can write a letter to the journal editor. Some journals ask the authors to
respond to the letter and publish the correspondences, which is a way of continuing the
scientific debate. Some journals also have rapid response comments attached to papers
online.
CAN PEER REVIEW DETECT PLAGIARISM?
A SELF-CORRECTING PROCESS
20
After publication, if a paper is found to be fraudulent or plagiarised, or researchers
realise they made a mistake in their calculations that invalidates the paper, the journal
publishes a retraction which appears alongside the paper online. These can be tracked
on Retraction Watch. If editors are concerned about the validity of a paper and there is
an investigation underway, they will publish an expression of concern.
The internet has created novel ways of reviewing research both pre and post peer review.
Some researchers have started to use blogs, wikis and other Web 2.0 technologies to
communicate their own research to other scientists in the field as well as share their
thoughts on the quality and conclusions of other research papers.
Pre-publication peer review is the conventional process of papers being
sent out for peer review before they are published in a journal.
Post-publication peer review is when a paper is scrutinised, replicated and
commented on by experts after it is published. New web technologies allow
readers to rate papers, and add comments and notes to online articles for
readers to see.
21
“I think it is important for science journalists to be as open as possible
about the sources for their stories. I don't think it is necessary to state as
a matter of course that a journal is peer reviewed (that is normally
implicit), but I think it is often useful to say if a story is based on work
from a non-peer-reviewed journal or work that has not been subjected
to peer review.”
JAMES RANDERSON
Environment and Science News Editor at the Guardian
When writing about research claims, should journalists report the status and quality of
research? For example, has the research been presented at a conference or is it
published in a peer reviewed journal?
Most people hear about scientific research through announcements in the media, so it is
the journalists who weigh up the status of research and decide what’s worth reporting.
ENTER THE JOURNALISTS…
Peer review is not only relevant to scientists.
Sense About Science has worked with the
public to promote an understanding of peer
review, encouraging the question ‘has it been
peer reviewed?’ to help the process of
weighing up conflicting scientific claims.
Understanding that published research has
been scrutinised by other experts in the field
can help people understand why a claim that
is backed up by a peer reviewed paper is likely
to be more reliable than someone’s opinion
which has undergone no such scrutiny.
DOES
PEER REVIEW
MATTER TO
THE PUBLIC?
“If patients have been diagnosed with a disorder and the medication
doesn’t seem to work immediately, they may search for an alternative on
the internet where there are a host of claims for miracle drugs. It can be
difficult to distinguish between claims that are backed by evidence and
have been tested by researchers, and those that are not. Understanding
peer review gives patients a tool to weigh up these claims.”
JANIS HICKEY, DIRECTOR
British Thyroid Foundation
22
“Peer review is not a guarantee that the science is right, just that it
seems to have been done properly. So whether I report the status of
research or not depends on the content. If some distinguished
cosmologist tells me – without benefit of peer review – that in his
opinion the universe went through a phase that resembled custard
before splashing into sticky globules that coalesced into galaxies, I
might very well make a story out of it. Right or wrong, such a conjecture
affects no one. On the other hand, if someone claimed a successful
treatment for multiple sclerosis without benefit of a peer reviewed
publication, I'd not touch it at all because it would be cruel to raise
unfounded hopes.”
TIM RADFORD
Freelance journalist
“Many of my editors – and many of the people that I write for – don't
understand the difference between research that has been peer
reviewed and research that hasn't, so I tend not to include those terms
in my writing. However I, personally, certainly do consider whether
research has been peer reviewed or not when considering how much
credibility to give to claims.”
CLAIRE COLEMAN
Freelance journalist who often writes about beauty treatments for the Daily Mail
Peer review may have its limitations, but it is also a remarkable process which relies on
the trust and co-operation of the scientific community and acts as a quality control
ensuring that published research is valid, significant and original. The process is essential
for the dissemination and advancement of scientific knowledge. Without peer review,
how would we weigh up claims and know what to believe?
PEER REVIEW MATTERS
In a survey3 of over 4000 researchers, most (84%) believed
that without peer review there would be no control in
scientific communication.
3
Results from the 2009 Peer Review Survey: Sense About Science, with support from Elsevier carried out
one of the largest ever peer review surveys of over 4000 authors and reviewers:
www.senseaboutscience.org/pages/peer-review-survey-2009.html
23
Reviewing is a role that is integral to the scientific community and so it is important that
early career researchers get involved in the process early on.
“One of the reasons I like to review papers is that it makes me feel like
an important part of the academic community, and that my opinion
about what is (or isn’t) good science actually matters.”
JAMIE MCCLELLAND
VoYS
“Reviewing for journals is my chance to stop bad science being
published and improve the quality of good science papers which
deserve to get published!”
MARGARET HESLIN
VoYS
“If the results in a paper have important consequences for the public, it
is essential that the work is reviewed by peers to check that the
conclusions are reliable.”
DR DEIRDRE HOLLINGSWORTH
Epidemiologist, Imperial College London
“Peer review is important because it helps people make decisions about
what to believe, what to treat with scepticism and what to trust. When
research work has been scrutinised and critically assessed by experts
before publication it helps prevent the release of work that is unsound,
inadequate or has been wrongly interpreted. Its role is to ensure the
scholarly record is as sound as possible. It isn’t, however, a guarantor of
absolute truth – it does sometimes go wrong and there are
shortcomings – but it is considered by many to be crucial to the
reputation and reliability of scientific research.”
DR IRENE HAMES
Editorial Consultant and author of Peer Review and Manuscript Management
in Scientific Journals
24
All are available as free downloads from www.senseaboutscience.org
I Don’t Know What To Believe
Peer review Survey 2009 Final Results
Peer review and the Acceptance of New Scientific Ideas
Peer review Education Resource
www.senseaboutscience.org/pages/peerrevieweducation.html
OTHER GUIDES TO PEER REVIEW
Peer review: a guide for researchers Research Information Network
www.rin.ac.uk/our-work/communicating-and-disseminating-research/peer-review-guide-
researchers
Anthony M. Vintzileos, MD, Cande V. Ananth, PhD, MPH 2010 The Art of Peer-Reviewing
an Original Research Paper; Important Tips and Guidelines J Ultrasound Med 2010;
29:513–518
BMJ training materials for reviewers:
www.bmj.com/about-bmj/resources-reviewers/training-materials
USEFUL RESOURCES FOR REVIEWING
PRE (Peer Review Evaluation) was formed to assist members of the scholarly publishing
community who are committed to preserving an ethical, rigorous peer review process:
pre-val.org
To find published papers with similar abstracts: etBlast: etest.vbi.vt.edu/etblast3
Clinical Trials registration information (all clinical trials should be registered before the
first patient is enrolled): www.icmje.org/faq_clinical.html
The Declaration of Helsinki; international ethical principles for medical research
www.wma.net/en/30publications/10policies/b3
Committee on Publication Ethics: publicationethics.org
Guidelines for research to be published in a biomedical journal, flowcharts and checklists
for e.g. systematic reviews, meta-analyses observational studies, and randomised
controlled trials: www.equator-network.org
International prospective register of systematic reviews: www.crd.york.ac.uk/prospero
SENSE ABOUT SCIENCE PUBLICATIONS
4. FURTHER INFORMATION
25
REFERENCES
Irene Hames (2007) Peer Review and Manuscript Management in Scientific Journals,
Blackwell Publishing in Association with ALPS.
A statistical review has been shown to improve the quality of manuscripts: Cobo E,
Selva-O’Callagham A, Ribera JM, Cardellach F, Dominguez R, Vilardell M. Statistical
reviewers improve reporting in biomedical articles: a randomized trial. PLOS ONE
2007;2(3):e332.
Mark Ware (2008) Peer review: benefits, perceptions and alternatives, Publishing
Researching Consortium,
www.publishingresearch.net/documents/PRCsummary4Warefinal.pdf
Effect of using reporting guidelines during peer review on quality of final manuscripts
submitted to a biomedical journal: masked randomised trial BMJ 2011; 343 doi:
10.1136/bmj.d6783.
26
THANK YOU
Julia would like to thank all the contributors and the following VoYS members: Mark
Ainslie, Marianne Baker, Emma Bell, Dave Bosworth, Mark Brook, Blanka Collis, Iain
Darby, Lewis Dean, Jaime Earnest, David Howey, Margaret Heslin, Jamie Horder, Jenny
Kent, Andrew Melbourne, Jamie McClelland, Diana Bowler, Philippa Shelton, and
Katherine Stapleton.
Join VoYS!
The Voice of Young Science network consists of hundreds of early career researchers who
want to stand up for science in public discussions. VoYS members tackle misconceptions,
challenge pseudoscience and respond to misinformation in all kinds of media.
VoYS members also encourage other early career researchers to get involved, sending
the message that it is important for scientists to stand up for science in public discussion
and that you don't need to wait until the end of your career to do so.
This guide is the third in the ‘Standing up for Science’ series of VoYS publications:
Standing up for Science 1: A guide to the media for early career researchers
Standing up for Science 2: The nuts and bolts
Further information about VoYS and their publications can be found here:
www.senseaboutscience.org/voys.html
Acknowledgements
This publication is part of the VoYS programme and is supported by:
This document is licensed under Creative Commons Attribution-Noncommercial-No derivative Works 2.0 UK:
England & Wales License.
Published in 2012 by
Sense About Science
14A Clerkenwell Green
London
EC1R 0DP
www.senseaboutscience.org
Registered Charity No.1146170
Company No. 6771027
Reprinted in 2014 with support from BioMed Central, Elsevier, PLOS, Taylor and Francis, Wiley and PRE
(Peer Review Evaluation).