SPECIAL TOPICS
Advancing Kinesiology Through Improved Peer Review
Duane V. Knudson
Texas State University
James R. Morrow Jr. and Jerry R. Thomas
University of North Texas
Peer review of scholarship is essential to journal quality, evidence, knowledge advancement,
and application of that knowledge in any field. This commentary summarizes recent literature
on issues related to peer-review quality and current review practice in kinesiology and
provides recommendations to improve peer review in kinesiology journals. We reviewed the
literature on the characteristics of peer review in scientific journals and describe the status of
peer review in kinesiology journals. Although the majority of scholars and editors strongly
support the peer-review process, systematic research in several disciplines has shown
somewhat positive but mixed results for the efficacy of peer review in evaluating the quality of
and improving research reports. Past recommendations for improvement have focused on
agreement between reviewers, standards for evaluating quality, and clarification of the
editorial team roles. Research on interventions, however, indicates that improving reviewer
performance is difficult. The specific research on peer review in kinesiology is limited. Six
recommendations to improve peer review are proposed: publishing clear evaluation standards,
establishing collaborative evaluation procedures and editorial team roles, utilizing online
submission data to help improve reviewer comments, creating author appeals procedures,
protecting reviewer time commitments, and improving reviewer recognition. There is
considerable variation in peer-review criteria and procedures in kinesiology, and
implementing several reasonable improvements may advance knowledge development and
the field of kinesiology.
Keywords: publication, referee, research, scholarship
Advancing the body of knowledge and application of
knowledge in science depends on high-quality research.
Peer review is traditionally employed for the initial
judgment of the scientific integrity and quality of research
reports. Recent efforts to systematize the integration and
interpretation of the evidence from research through
evidence-based practice (Amonette, English, & Ottenbacher,
2010; Faulkner, Taylor, Ferrence, Munro, & Selby, 2006) and
reporting standards for research (Hirst & Altman, 2012) have
likely increased the burden on editors and reviewers
conducting the peer review of research reports (Knudson,
Elliott, & Ackland, 2012).
Manuscript reviewers provide essential services to any
disciplines and professions related to the journal. Traditionally,
reviewers are tasked with evaluating two main criteria
on articles: (a) quality and (b) the relevance or contribution
that advances the field. Reviews are also expected to
contribute by improving the report through constructive
advice and feedback. Although differences exist in what is
considered quality or scientific standards, as well as in what
constitutes a contribution advancing the field, peer review is
a vitally important part of quality control of the evidence
and the development of theory/knowledge in a discipline
Correspondence should be addressed to Duane V. Knudson,
Department of Health and Human Performance, Texas State University, 601
University Drive, San Marcos, TX 78666. E-mail: dknudson@txstate.edu
Research Quarterly for Exercise and Sport, 85, 127–135, 2014
Copyright q SHAPE America
ISSN 0270-1367 print/ISSN 2168-3824 online
DOI: 10.1080/02701367.2014.898117
(Baker, 2002; Hojat, Gonnella, & Caelleigh, 2003).
Reviewer expertise/quality is also considered important to
journal quality (Callaham, Baxt, Waeckerle, & Wears,
1998). Although quality reviews cannot ensure that highquality
papers are submitted to a journal, they can decrease
the probability that erroneous designs, analyses, or
interpretations are published.
This commentary aims to summarize recent literature on
issues related to peer-review quality and current review
practices in kinesiology, and makes recommendations to
improve peer review in kinesiology journals. Although there
has been considerable research on peer review in many
disciplines, little research on peer-review practices exists in
kinesiology, with only one paper reviewing practices in
exercise and sport psychology journals (Holt & Spence,
2012) and another examining reviewer reliability (Morrow,
Bray, Fulton, & Thomas, 1992). It is possible that the
diverse, subdisciplinary structure of kinesiology has not
been conducive to unified interest in the nature and quality
of peer review across journals. We hope this commentary
stimulates discussion and implementation of some of these
recommendations to support reviewers and the peer-review
process in kinesiology. We believe improvements in peer
review will result in subsequent improvement in the
research and theory developed in kinesiology.
RECENT CRITIQUES OF PEER REVIEW
New forms of academic publishing such as open-access
journals have reinvigorated the longstanding debate about the
system of peer review in scholarly journals. Various forms of
more open reviewing have been proposed from not blinding
reviewers and authors to publication of reviewer identity and
posteriori reviewer and reader commentary. Several reviews
have documented the long history of controversy surrounding
peer review of scholarly articles in many disciplines
(Armstrong, 1997; Benos et al., 2007; Burham, 1990;
Campanario, 1998; Hardie, 2010; Hojat et al., 2003; Holt &
Spence, 2012; Lee, Sugimoto, Zhang, & Cronin, 2013).
Concerns About Peer Review
Critics of peer review question the logic of the binary accept/
unacceptable peer-review decision, when the quality or
contribution of research is clearly more subtle, and the
imprecision of its definition (Jefferson, Wager, & Davidoff,
2002; Lamont, 2009; Smith, 2006; Souder, 2011; Wheeler,
2011). High-influence journals see numerous submissions,
so it is likely that some papers have “fatal” errors or flaws. A
fatal flaw is a major design error (e.g., lack of control of
confounding variables, uncalibrated or incorrectly used
equipment) that compromises the internal validity of the
study. Critics also identify the rare cases of editorial decision
errors, where weak papers with clear problems are published
and papers with no fatal flaws or major weaknesses have
been rejected. For example, Smith (2006) has questioned the
high cost of time and money of peer review for the purported
benefit of eliminating papers with errors, sloppy design, or
inaccurate interpretation.
Perhaps the strongest criticisms are based on systematic
research on peer review, which has documented problems
including several kinds of bias and poor reliability.
Although most of the studies of intrareviewer and
interreviewer reliability report generally low-to-moderate
reliability (see reviews by Hardie, 2010; Hojat et al., 2003;
Horrobin, 2001; Lee et al., 2013; Miller, 2006; Smith,
2006), several studies report embarrassingly poor consistency
between reviewers. For example, Rothwell and Martyn
(2000) reported overall reviewer agreement not much better
than chance in two neuroscience journals. Morrow et al.
(1992) reported that interrater reliability of reviews in the
Research Quarterly for Exercise and Sport were somewhat
lower than in other disciplines.
Should we always expect strong agreement among
reviewers? Editors often send papers to reviewers for very
differentreasons.Somepapersarequitecomplexandareviewer
may know more about one theory or methodology than another.
For example, we sometimes see papers using mixed
methodology—both quantitative and qualitative methods.
Reviewers may be much more familiar with one methodology
than the other, which could contribute to inconsistent
evaluations. A reviewer may be selected for statistical expertise
but have limited knowledge about the discipline and hypotheses
being tested. In a recent review, Lee et al. (2013) contended that
some reviewer disagreement should be expected due to the
social nature of the process. When reviewers have directly
opposing evaluations of a manuscript, the role of the editor or
associate editor in summarizing comments, seeking additional
review, or resolving conflict is essential.
Some consider peer review so inconsistent and costly
that they have proposed extensive reform efforts to open up
the review and scholarly publication process (e.g., Hartley,
2012; Kaplan, 2005). Open peer-review systems typically
feature disclosure of both authors and reviewers, and they
publish both and sometimes collect discussion about both
the paper and the reviews. These open procedures have
several problems. First, these procedures run higher risks for
theft of intellectual property. Second, there is a higher risk
for publication and application of erroneous results. The
negative effects of this are magnified as most scientific
fields are notoriously slow in their role of collective selfcorrection
of erroneous reports and theories. We have found
scholars rarely challenge weak or erroneous studies through
letters to editors or replication in journals. The occasional
letter to the editor is quite rare given the thousands of papers
published with flaws in reviews of literature, design,
statistics, citation, and data interpretation. For example,
despite decades of research documenting the inflation of
Type I statistical errors testing numerous dependent
128 D. V. KNUDSON ET AL.
variables (Knudson, 2007, 2009; Lundbrook, 1998; Morrow
& Frankiewicz, 1979; Zhu, 2012), a large percentage of
papers continue to be published in kinesiology journals with
these errors (James & Bates, 1997; Knudson, 2005, 2009;
Mullineaux, Bartlett, & Bennett, 2001). Third, there is no
research on the hypothesized positive effects of these open
publication proposals (Lee et al., 2013).
Supporters of Peer Review
Proponents of the importance of peer review point to
anecdotal evidence of improved manuscripts and the
rejection of manuscripts with serious errors submitted to
journals they serve. Journal article rejection rates are quite
consistent over time within particular disciplines (Hargens,
1988) even though there are changes in editors and editorial
board members. Some research on peer review has shown
little evidence of bias and acceptable agreement between
reviewers (Morrow et al., 1992; Vecchio, 2006). A
Cochrane report (Jefferson, Rudin, Brodney Fose, &
Davidoff, 2008) on peer review in biomedical journals
concluded that there were a few studies reporting peer
reviews that were generally valid and that improved the
readability and reporting of evidence in the papers, but the
limited number of these studies meant they had little
generalizability. Valid reviews are unbiased judgments
using the evaluation criteria of the journal.
Although there is tentative supporting evidence of the
effectiveness of traditional peer review, there is generally
widespread approval of the system. A large-scale international
survey of scholars reported 85% approval of the
peer-review process for improving research (Ware &
Monkman, 2008). Most medical editors consider traditional
peer review to be the best method available (Alpert, 2007;
Hardie, 2010; Smith, 2006). In addition, most cases of
retention, tenure, and promotion reviews in universities
would be complicated without relying on peer review for an
initial level of evaluation of faculty scholarship.
Based on evidence and opinion, double-blind (masking
author and reviewer identities) peer review is likely the best
system for scholarly journals. Bornmann and Mungra
(2011) evaluated the research on improving peer review and
concluded that blinding, especially double blinding, reduces
the chances of review bias and improves objectivity. It is
likely that double-blind reviews provide protection from
bias, even though it cannot ensure reviewers will not know
the likely authors. In medicine, for example, there is
evidence of potential editorial favoritism, so double-blind
review has been recommended (Luty, Arokiadass, Easow,
& Anapreddy, 2009). Holt and Spence (2012) recommended
that, although research on the effectiveness of blinding in
reviews was inconclusive, more research be conducted
before editors of sport and exercise psychology journals
abandon double-blind peer review. Indirect support for
traditional double-blind peer review also comes from the
lack of research on the effectiveness of any of the open peerreview
alternatives (Lee et al., 2013).
Despite inconsistent support for the high ideals of
accuracy and reliability of peer review in journals, most
scholars and editors support the peer-review process and
some have published recommendations to improve this
important process. The following sections summarize recent
suggestions to improve the peer-review process.
RECENT SUGGESTIONS FOR IMPROVEMENT
Several recent narrative reviews have proposed recommendations
to improve peer review (e.g., Hojat et al., 2003; Holt
& Spence, 2012; Kaplan, 2005; Schwartz & Zamboanga,
2009; Souder, 2011; Wheeler, 2011). The primary issues
these recommendations address are improving agreement
among reviewers, standards for evaluating quality, and
clarifying the editorial team roles.
The field of medicine, perhaps, has seen the most
research on peer review and journal/editor efforts addressing
these issues. Concerns about peer review by medical
editors resulted in the establishment of international
conferences on the topic in 1998, with the Seventh
International Congress on Peer Review and Biomedical
Publication being held in Chicago in 2013. Good practical
advice for editors and reviewers resulting from these
collaborations includes the Committee on Publication
Ethics (COPE), its code of conduct (COPE, 2011), and
Hames’s (2007) book on peer review.
Systematic evaluations of medical reviewer quality
generally show relatively uniform quality ratings, usually
with small but not meaningful declines in mean ratings of
reviewers from 4 to 14 years (Callaham & McCulloch,
2011; Houry, Green, & Callaham, 2012). Interestingly,
studies of systematic programs of short-term training
(Callaham, Knopp, & Gallagher, 2002; Callaham &
Schriger, 2002; Schroter et al., 2004) and longer-term
mentoring (Houry et al., 2012) report little effect on quality
of peer reviews. These results indicate that review quality is
heavily dependent on the reviewer and is difficult to
substantially improve. Strong reviews require that scholars
remain up-to-date, that they recognize and put aside
personal biases, and that they invest considerable time (2–6
hr) to evaluate and provide constructive feedback to authors
(Black, van Rooyen, Godless, Smith, & Evans, 1998; Holt
& Spence, 2012). Although more research is needed in this
area, journals should still make all reasonable efforts to
improve the quality of peer review.
PEER-REVIEW PRACTICE IN KINESIOLOGY
The founding of physical education/kinesiology journals in
the United States gradually adopted the peer-reviewed
IMPROVED PEER REVIEW 129
model of journals established in 17th-century Europe (Park,
1980). To document current peer-review practice in
kinesiology, the authors reviewed the peer-review procedures
of 18 kinesiology journals (Table 1). Several peerreview
trends were consistent with journals in other
scholarly disciplines.
The kinesiology journals examined used both authorblinded
(28%) and double-blind (72%) peer review.
Reviewers were queried on up to 11 specific evaluation
criteria. There is considerable variation in these categories
and how they are rated, from yes/no responses to 10-point
Likert-type rating scales. Similar variation across theory to
practice criteria was reported by Holt and Spence (2012) in
four sport and exercise psychology journals.
All journals requested blinded comments and editorial
recommendations to the editor, along with comments to the
author. Most journals use electronic submission systems and
encourage reviewers to provide specific and constructive
comments in open boxes or attached electronic files. Rarely
are manuscript-specific (line number-identified) details
explicitly required of reviewers in their specific comments
to the editors and authors. During many years of experience,
we have anecdotally observed many times when reviewers,
in correspondence with journals, have not taken time to
provide specific feedback and comments in reviews.
This small review of kinesiology journals indicated there
is wide variation in the criteria and guidance provided to
reviewers in their roles in evaluating the quality/
contribution of and in improving papers submitted to
kinesiology journals. There is also considerable variation in
the communication of submission expectations and
evaluative criteria. These data are consistent with a 5-year
study of the reliability of reviews in the Research Quarterly
for Exercise and Sport. Morrow et al. (1992) reported that
three measures of reviewer reliability were lower (generally
less than 50% agreement) than the moderate reliability
observed in other disciplines. Although variation in
reviewer expectations could be expected given the variation
in the subdisciplines in kinesiology, there is room for
improvement in communicating submission expectations
and the criteria used in peer review in many kinesiology
journals. The next section of this article outlines some
recommendations to improve the traditional peer-review
process that could be easily implemented in kinesiology
journals.
RECOMMENDATIONS FOR KINESIOLOGY
JOURNALS
Editors of the major multidisciplinary and single subdisciplinary
kinesiology journals wield considerable authority
to establish appropriate peer-review procedures and
standards. They are the “gatekeepers” for the journal
and field and should work with their editorial boards and
sponsoring organizations to monitor and improve the peerreview
process. Six recommendations are proposed for
editors to consider with their editorial teams for improving
peer reviews and subsequently the articles, evidence, and
knowledge generated in kinesiology (Table 2). We
acknowledge that these recommendations are based
primarily on logic and expert opinion rather than
experimental evidence. There is limited prospective
evidence comparing typical review procedures to systematic
improvement programs (Callaham et al., 2002; Callaham &
Schriger, 2002; Houry et al., 2012; Schroter et al., 2004).
TABLE 1
Peer-Review Procedures in Kinesiology Journals
Journal Blinding Criteriaa
Decision Categories
British Journal of Sports Medicine Author 3 (e.g., ethics in research and publication) 4
European Journal of Sport Science Double 6 (e.g., impact, originality, rank) 4
Journal of Applied Biomechanics Author 9 (e.g., clear question, new insights) 5
Journal of Athletic Training Double 4 (e.g., relevance, presentation) 8
Journal of Motor Behavior Double 6 (e.g., varying levels of analysis) 4
Journal of Orthopaedic and Sports Physical Therapy Double 5 (e.g., importance or interest) 6
Journal of Physical Activity and Health Double 4 (e.g., study design and interpretation) 3
Journal of Sport and Exercise Psychology Double 4 (e.g., methodological adequacy) 4
Journal of Sports Sciences Double 3 (e.g., quality of data and methods) 5
Journal of Strength and Conditioning Research Double 4 (e.g., practical applications impact) 6
Journal of Teaching in Physical Education Double 10 (e.g., sound theory, new insights) 5
Measurement in Physical Education and Exercise Science Double Not provided 4
Medicine and Science in Sports and Exercise Author 7 (e.g., overall percentile rank) 4
Perceptual and Motor Skills Author 4 (e.g., sound design, analysis) 4
Pediatric Exercise Science Double 11 (e.g., relevance, originality) 4
Quest Double 5 (e.g., theoretical and practice-based) 4
Research Quarterly for Exercise and Sport Double 4 (e.g., impact to whole field) 4
Sports Biomechanics Author 10 (e.g., writing and organization) 5
a
These data were gathered and verified by contacting journal editors. Due to space limitations, only example evaluation criteria for each journal are listed.
130 D. V. KNUDSON ET AL.
Similar recommendations based on a review of editorials
and “limited” studies have been proposed for sport and
exercise psychology journals (Holt & Spence, 2012).
Publish Clear Evaluation Standards
Peer review of research is a subjective task, but the validity
and reliability of these evaluations will be better if journals
are clear regarding the review standards or criteria for
articles submitted for publication (Holt & Spence, 2012).
Editors should publish clear statements identifying the
criteria used to evaluate the quality of manuscripts
submitted to the journal. This is more than the typical
sentence or two on the mission, kinds of articles considered,
and general comments on peer-review evaluation for the
journal. Although this seems logical, surprisingly few
journals report the detailed criteria used in their peer review
in the purpose of the journal or in author instructions.
The journal should regularly publish in the author
instructions what constitutes expected characteristics, and
consequently the review criteria, for evaluating the quality
of manuscripts submitted to the journal. Authors should
know specific criteria that are used to judge quality and
alignment with the journal’s overall mission. For example,
the Journal of Sports Sciences has reviewers rate the impact
of the article, originality of the article and the quality of the
data and methods for submitted manuscripts. Medicine and
Science in Sports & Exercise has these same criteria,
includes an overall ranking, and a specific question if
additional statistical review of the paper is necessary.
Research Quarterly for Exercise and Sport instructs
reviewers to consider six major issues but uses four other
criteria to be rated (quality, impact, writing, impact to the
whole field of kinesiology) as the top 10%, 25%, or 50%, or
the bottom 50%. Prior knowledge of specific evaluation
criteria will prevent misunderstandings and encourage
authors to prepare higher-quality reports aligned with the
mission and expectations of the journal.
Ultimately, the quality of individual papers published
will be evaluated by the field in awards, citations to the
work, replications, and extensions of that research. Peer
review should not be considered a scientific guarantee or
approval of the results and conclusions of the study, but it
should be clear what journal standards were used to evaluate
the relevance and quality of the research. Scholars should
continue to critically evaluate each submission based on the
merits of the research report and not the perceived quality of
peer review of the particular journal.
A prerequisite to or sometimes part of the evaluation
standards is a subjective judgment on the issue of paper
suitability, relevance, merit, or importance to the field. This
is an area that may be most subjective and resistant to clear
operational definitions in evaluation standards; however, the
description of the journal and author instructions should be
very clear on this matter. One example of where journal
missions should be clear is if there is an expectation of either
theoretical or applied research. While this is not a
dichotomy, some journals and fields favor one approach to
research reports over the other. For example, Research
Quarterly for Exercise and Sport focuses more on theory by
TABLE 2
Six Recommendations to Improve the Quality of Reviews in Kinesiology Journals
Publish Clear Evaluation Standards
† Author instructions and explicit review criteria used for submissions (e.g., authors should know if the manuscript will be evaluated on criteria such as
originality of the work, importance to theory or application, or scientific impact).
† Standards or descriptions about judgments regarding relevance, merit, or importance to the field in reviews.
† Submission guidelines/checklists for electronic submissions.
Establish Collaborative Evaluation Procedures and Roles
† Publish who on the editorial team evaluates manuscripts for suitability if the journal uses pre-review to reject papers (e.g., based on alignment with
journal mission, perceived impact of research, or if certain rankings [top 25] are expected for entry into full review).
† Clearly identify roles among editor, associate editors, and reviewers.
† Foster culture of expert content reviewers with a committed, constructive, respectful evaluation of manuscripts.
† Where relevant, provide potential reviewers with information on why they have been invited to review specific submissions.
Use Online Manuscript Data to Help Reviewers Improve
† Provide reviewers with feedback on editorial evaluations of their reviews and other blinded reviews of manuscripts they evaluate.
† Provide reviewers with opportunities for training, including face-to-face training at professional conferences (e.g., SHAPE America reviewer
workshop).
Establish a Procedure for Author Appeals
† Define criteria for unusual cases where reviews disagree or a poor-quality review may be appealed to the editor by an author.
† Possibly establish ombudsperson to handle author appeals.
Protect Time Commitment of Editors and Reviewers
† Track frequency of requests for reviews and limit rounds of reviews for specific submissions.
† Establish recommended timelines for reviews.
Improve Reviewer Recognition
† Use recognition programs for reviewers beyond traditional letters, journal access, and listing annually in the journal (e.g., conference events with
financial or other awards for quality reviews).
IMPROVED PEER REVIEW 131
publishing research that contributes significantly to the
knowledge base of the field. Another example from many
new journals specializing in the scholarship of teaching and
learning (SoTL) is whether the journal favors quantitative or
qualitative methods. SoTL is specific research on learning
the knowledge and methods of an academic discipline.
Some disciplines have SoTL journals with a tradition of
qualitative learning research, so authors using mixed
methods or quantitative designs should know that their
methodology may be fairly unique to journal reviewers.
Journals that publish research on specific sports or
subdisciplines have fairly clear missions, but establishing
what is a relevant and meaningful contribution to the field in
a multidisciplinary journal is often open to interpretation.
Editors of multidisciplinary journals should publish general
descriptions of how relevance and significance to the journal
are defined so that there are fewer misunderstandings
between authors and the editorial team. Whatever the type
of journal, the author instructions should be clear regarding
what a journal considers a relevant or meritorious
submission and whether this involves a judgment about
scientific validity and/or is a prerequisite judgment of merit
prior to sending the report for peer review.
One recent issue we have noticed that may negatively
influence the evaluation of a manuscript is unclear author
instructions regarding electronic submissions. Often
electronic submission systems require that manuscript
and support materials be submitted in separate files that are
automatically assembled into a manuscript for internal
review. When this is the case, the author instructions
should be very detailed regarding how the elements of a
submission will be uploaded to the system. A checklist
with instructions for submission can help authors navigate
online systems that may be confusing and different from
journal to journal. This is important because some
reviewers are quite strict on format/structure issues and
use them as surrogate indicators of attention to detail and
quality.
Establish Collaborative Evaluation Procedures
and Roles
Clear evaluation standards will help the editorial team work
with the same vision of article quality, but a journal editor
should also establish and publish collaborative policies and
procedures for evaluating manuscripts. Author instructions
should state the journal’s philosophy of peer review as well
as the procedures and policies the editor employs to ensure
the journal mission is fulfilled. For example, authors and
reviewers should know who is responsible for initial
screening of papers for possible external review, synthesis
of reviews, and final editorial decisions. Collaborative roles
distribute the work tasks and reduce misunderstandings and
duplicated effort within the editorial team.
The editorial teams of journals change over time, so new
generations of the editorial team may propose changes in
policies and procedures. Normally when the editorial team
is led by an editor in chief, this editor has previously served
as a section or associate editor or has served overlapping
terms with a previous editor in chief. These apprenticeship
experiences ensure smooth transitions and that important
policy and procedural decisions are not lost over time. In
this way, proposed changes for continuous improvement are
more effective because any that were not successful in the
past will not likely be repeated.
Author instructions for a journal should state whether
the editor in chief or an associate editor makes an initial
evaluation of suitability before external reviews are
solicited and, if so, whether these judgments are limited to
a certain expectation of importance to the field (e.g., top
25%). Authors need to know how the important decision
on the suitability, merit, or contribution question will be
evaluated, as well as by whom and when this decision is
made. Considerable time for authors and the editorial
team will be wasted when articles outside the mission of
the journal are submitted. Journal acceptance rates will
also be more accurate when manuscripts that are
immediately rejected for lack of merit/suitability are
reduced.
Author instructions should specify the journal policy on
expected referee qualifications. Authors should know if
preferred or not-preferred referees can be nominated and
whether the journal requires specific reviews from
methodology or statistical experts. An expected timeframe
for initial and additional reviews should be published, along
with the expectation that reviews will be respectful and
constructive in the evaluation of strengths and weaknesses.
At least annually, editors should publish summary data
about the journal including: number of papers submitted,
rejection rate, average/median review time, average/median
time from acceptance to publication, and other relevant
information.
Some reviewers do not see authors as peers and equals
and provide openly antagonistic and disrespectful critiques
(Raff & Brown, 2013). Reviewers should not see themselves
as gatekeepers, judges of ultimate truth, or defenders of
specific theories or experimental methodologies. Overtly
disrespectful or hostile reviews have negative consequences
for the journal. Authors will interpret this disrespect as
evidence of bias in the reviewer, editorial team, and the
journal. Editors do not want hostile authors who may
communicate with other scholars about a battle with poor
peer reviews from their journal. The author(s) poisoned by a
disrespectful review might not submit research or review
research for the journal in the future.
Editorial policy should state that reviewers consider
authors as peers, so reviewers have an ethical responsibility to
evaluate submissions with openness and fairness and to
provide constructive critique. Editors should not return
132 D. V. KNUDSON ET AL.
biased, inaccurate, or antagonistic comments in reviews to
authors.
Use Online Manuscript Data to Help
Reviewers Improve
Many journals currently use online manuscript review and
management systems that facilitate the reviewers’ work in
searching databases for related literature and for submitting
reviews. The journal management system should provide
guidelines and training materials for reviews. Many of these
manuscript or publisher systems also allow for efficient data
collection about reviews, timing, quality, and subsequent
communication that should be used by editors to assess the
peer-review process and to help reviewers improve. Holt
and Spence (2012) also recommended that journals provide
training and feedback to reviewers.
Reviewers should be given copies of other reviewers’
reports and editorial feedback/decisions, as well as periodic
feedback or evaluations of their reviews (Hames, 2007;
Hartley, 2012; Holt & Spence, 2012). Internal reviewer
evaluation information (timeliness, tone, quality, and
specificity of feedback to authors) should form the basis
for feedback to individual reviewers on the quality of their
service. If the editorial team expects a certain level of
reviewer performance, feedback should be given so that
reviewers can determine if they need additional training or
mentoring. A simple 5-point scale for evaluating reviewers
has been shown to be moderately reliable and to correlate
with the ability to identify flaws in studies (Callaham et al.,
1998). Periodically, the editor should also share with all
journal reviewers examples of blinded exemplar or
outstanding reviews. Reviewer evaluation information is
important in planning training to improve review quality
and consistency. This information may also be useful in
identifying future section editors and editors in chief.
The training of reviewers for kinesiology journals should
also be extended to graduate students. Graduate programs
should provide training on critical review of literature and
best practices in blinded peer review. Strong graduate
training in research methodology has been associated with
high-quality reviews in medicine (Black et al., 1998).
Confidentiality issues related to submissions, reviews, and
inviting graduate students to monitor or even perform
reviews should be addressed in the journal editorial policy.
Establish a Procedure for Author Appeals
Several authors have recommended that journals develop
specific appeals procedures for authors of rejected papers
(Albers, Floyd, Fuhrmann, & Martinez, 2011; Hojat et al.,
2003; Schwartz & Zamboanga, 2009), and some journals
have successfully implemented ombudspersons to handle
problems with the editorial process (Horton, 1998; Wager &
Kleinert, 2012). Given the importance of publishing
research in certain journals for tenure/promotion and grants
for many faculty, journals should establish an appeal
procedure that can address the rare cases where inconsistent
or poor reviews may have disadvantaged a strong paper.
These situations would normally be quite rare, because the
editor in chief can serve as a reviewer or solicit another
review to resolve most problems. Journals that establish
appeal procedures or ombudspersons should establish
specific minimal conditions (e.g., evaluation criteria that
can be appealed, certain minimum standards on a positive
review, and evidence of error, bias, or inconsistency in
reviews) for an appeal to be considered. This would reduce
the nuisance appeals from articles that clearly are not of
adequate quality to meet a journal’s mission or standards.
Journals should determine if the editor/associate editors,
original reviewers, or new reviewers are used in these
appeals. Improvements in peer-review standards and the
specific minimal standards for appeals should ensure that
the number of papers in an appeal process would be quite
low and would not burden or affect the trust built within the
editorial team.
Protect Time Commitment of Editors and Reviewers
Reviewers donate several hours of time to review each
research report. Every effort should be made to protect the
reviewer’s time to encourage repeated and quality reviews.
Editors can track the frequency of requests for reviews and
work to ensure the review procedures and roles are closely
followed. Editors should integrate their own reviews with
those from external reviewers to create clear editorial
decision letters. This integrated feedback to authors is
important so that only one additional round of reviews is
needed for a report with potential. For example, the editors
handling conflicting recommendations from two reviewers
should clarify which position they support so authors know
where they stand on controversial issues. Editors who lay
out exactly which comments are important to integrate into
a new revision will make revisions more focused and not
leave authors guessing how to handle conflicting reviews.
Previous reviewers or new reviewers brought in to resolve a
controversy should not be subjected to numerous resubmissions
to deal with partially revised papers or relieve editors
from making publication decisions. Editors should also
consider returning to authors or rejecting poor revisions that
do not clearly address the general and specific comments
provided by the editors and reviewers.
Improve Reviewer Recognition
Intrinsic rewards for serving as a reviewer include service to
the field/discipline, academic prestige, and access to recent
advances in the discipline. These rewards are often not
enough for many editors to have an adequate pool of skilled
reviewers to select for submissions. There may be declining
IMPROVED PEER REVIEW 133
interest in many forms of professional service (Cardinal,
2013). We believe that editors, journals, and organizations
should consider additional kinds of recognition for
reviewers, beyond the traditional listing of reviewers in
the journal and occasional letters of appreciation.
Inexpensive acknowledgement could be recognition
meals or receptions at the conferences of the sponsoring
society of the journal. These events can be used for training
or can even involve awards for higher-quality reviews or
reward such as Reviewer or Reviewers of the Year. The
internal evaluation of reviewers by the editors of a journal
could be easily expanded to provide single or multiple
awards for excellent reviewer service. Recognition as a
substantial contributor to the journal might begin to provide
accolades that could be considered as meaningful academic
service and could possibly contribute to tenure and
promotion evaluations. Kachewar and Sankaye (2013)
have proposed a framework for calculating contributions as
a reviewer with various levels of recognition. They propose
an evaluation system creating a reviewer index, directories
of these ratings, recognition of high-quality reviews, and
publication of the identity of reviewers who provide highquality
reviews.
Some journals provide financial compensation through
subscriptions and electronic access, and some even pay
for editorial and review services. Naturally, financial
compensation for fast or highly rated reviews could be
considered to increase the incentive for high-quality reviews.
CONCLUSIONS
Kinesiology journals use a variety of peer-review
procedures and standards, and there is limited research on
the effectiveness of peer review in the field. Research across
other disciplines indicates strong support for traditional
blinded peer review by scholars; however, a lack of clarity
and subjectivity of the process often makes for poor-tomoderate
interrater reliability. Article peer review is an
important and difficult task, so there has been little
prospective evidence documenting the effectiveness of
training to improve reviews. Despite this lack of progress,
several recommendations to improve peer review in
kinesiology are logical and should be implemented and
tested by future research.
WHAT DOES THIS ARTICLE ADD?
Continued controversy around the process of peer review of
scientific research has called into question the validity and
advancement of knowledge. There is limited research on
this topic in kinesiology, so this narrative review documents
peer-review practice in kinesiology and an extensive body
of research on peer review from other disciplines and efforts
to improve peer review. Research in a variety of disciplines
has shown somewhat positive but mixed results on the
efficacy of peer review in evaluating the quality of and
improving research reports submitted to journals. Because
editors and reviewers wield much power in the peer-review
system and provide such important services to the
advancement of knowledge and the field of kinesiology,
we propose six recommendations for peer-review improvement.
Improved peer review will reduce the publication of
erroneous and weak designs, statistical errors, and
inappropriate interpretation of results in kinesiology
research. Improved peer review will also improve the
quality and clarity of communication of kinesiology
research, as well as further the editorial mission of
kinesiology journals. There is a need for editors and editorial
teams of kinesiology journals to consider implementing
strategies to improve peer review and subsequently the
articles, evidence, and knowledge in kinesiology. Systematic
research documenting the effectiveness of proposed
improvements in peer review is also needed.
REFERENCES
Albers, C. A., Floyd, R. G., Fuhrmann, M. J., & Martinez, R. S. (2011).
Publication criteria and recommended areas of improvement within
school psychology journalsas reported by editors, journal boardmembers,
and manuscript authors. Journal of School Psychology, 49, 669–689.
Alpert, J. S. (2007). Peer review: The best of the blemished? American
Journal of Medicine, 120, 287–288. doi:10.1016/j.amjmed.2007.02.013
Amonette, W. E., English, K. L., & Ottenbacher, K. J. (2010). Nullius in
verba: A call for the incorporation of evidence-based practice into the
discipline of exercise science. Sports Medicine, 40, 449–457.
Armstrong, J. S. (1997). Peer review for journals: Evidence on quality
control, fairness,andinnovation.ScienceandEngineeringEthics, 3,63–84.
Baker, D. (2002). The peer review process in science education journals.
Research in Science Education, 32, 171–180.
Benos, D. J., Bashari, E., Caves, J. M., Gaggar, A., Kapoor, N., LaFrance,
M., . . . Zotov, A. (2007). The ups and downs of peer review. Advances
in Physiology Education, 31, 145–152. doi:10.1152/advan.00104.2006
Black, N., van Rooyen, S., Godless, F., Smith, R., & Evans, S. (1998). What
makes a good reviewer and a good review for a general medical journal?
Journal of the American Medical Association, 280, 231–233.
Bornmann, L., & Mungra, P. (2011). Improving peer review in scholarly
journals. European Science Editing, 37, 41–43.
Burham, J. C. (1990). The evolution of editorial peer review. Journal of the
American Medical Association, 263, 1323–1329.
Callaham, M. L., Baxt, W. G., Waeckerle, J. F., & Wears, R. L. (1998).
Reliability of editors’ subjective quality ratings of peer reviews of
manuscripts. Journal of the American Medical Association, 280,
229–231.
Callaham, M. L., Knopp, R. K., & Gallagher, E. J. (2002). Effect of written
feedback by editors on quality of reviews: Two randomized trials.
Journal of the American Medical Association, 287, 2781–2783.
Callaham, M. L., & McCulloch, C. (2011). Longitudinal trends in the
performance of scientific peer reviewers. Annals of Emergency Medicine,
57, 141–148.
Callaham, M. L., & Schriger, D. L. (2002). Effect of structured workshop
training on subsequent performance of journal reviewers. Annals of
Emergency Medicine, 40, 323–328.
134 D. V. KNUDSON ET AL.
Campanario, J. M. (1998). Peer review for journals as it stands today—Part
1. Science Communication, 19, 181–211.
Cardinal, B. J. (2013). Service vs. serve-us: What will your legacy be?
Journal of Physical Education, Recreation and Dance, 84(5), 4–6.
Committee on Publication Ethics. (2011). Code of conduct and best
practice guidelines for journal editors. Retrieved from http://
publicationethics.org/resources/code-conduct
Faulkner,G.,Taylor,A.,Ferrence, R., Munro,S.,& Selby, P.(2006).Exercise
science and the development of evidence-based practice: A ‘better
practices’ framework. European Journal of Sport Science, 6, 117–126.
Hames, I. (2007). Peer review and manuscript management in scientific
journals: Guidelines for good practice. Oxford, England: Blackwell.
Hardie, J. (2010). Is this article worth reading? A critical analysis of peer
review. Oral Health, 100(9), 70–72, 74, 76–77.
Hargens, L. L. (1988). Scholarly consensus and journal rejection rates.
American Sociological Review, 53, 139–151.
Hartley, J. (2012). Refereeing academic articles in the information age.
British Journal of Educational Technology, 43, 520–528.
Hirst, A., & Altman, D. G. (2012). Are peer reviews encouraged to use
reporting guidelines? A survey of 116 health research journals. PLoS
ONE, 7, e35621. doi:10.1371/journal.pone.0035621
Hojat, M., Gonnella, J. S., & Caelleigh, A. S. (2003). Impartial judgment by
the ‘gatekeepers’ of science: Fallibility and accountability of the peer
review process. Advances in Health Sciences Education, 8, 75–96.
Holt, N. L., & Spence, J. C. (2012). A review of the peer review process and
implications for sport and exercise psychology. Athletic Insight, 14(1).
Retrieved from http://www.athleticinsight.com/Vol14Iss1/Feature.htm
Horrobin, D. F. (2001). Something rotten at the core of science? Trends in
Pharmacological Sciences, 22, 51–52.
Horton, R. (1998). The journal ombudsperson: A step towards scientific press
oversight. Journal of the American Medical Association, 280, 289–299.
Houry, D., Green, S., & Callaham, M. (2012). Does mentoring new peer
reviewers improve review quality? A randomized trial. BMC Medical
Education, 12, 83.
James, C. R., & Bates, B. T. (1997). Experimental and statistical design
issues in human movement research. Measurement in Physical
Education and Exercise Science, 1, 55–69.
Jefferson, T., Rudin, M., Brodney Fose, S., & Davidoff, F. (2008). Editorial
peer review for improving the quality of reports of biomedical studies.
Cochrane Database of Systematic Reviews, 2, doi:10.1002/14651858.
MR000016.pub3
Jefferson, T., Wager, E., & Davidoff, F. (2002). Measuring the quality of
editorial peer review. Journal of the American Medical Association, 287,
2786–2790.
Kachewar, S. G., & Sankaye, S. B. (2013). Reviewer index: A new proposal
of rewarding the reviewer. Mens Sana Monographs, 11, 274–284.
Kaplan, D. (2005). How to fix peer review: Separating its two functions—
improving manuscripts and judging their scientific merit—would help.
Journal of Child and Family Studies, 14, 321–323.
Knudson, D. (2005). Statistical and reporting errors in applied
biomechanics research. In Q. Wang (Ed.), Proceedings of the XXIII
International Symposium on Biomechanics in Sports (Vol. 2,
pp. 811–814). Beijing, China: China Institute of Sport Science.
Knudson, D. (2007). Analysis of multiple comparison errors using ground
reaction force data. In H.-J. Menzel & M. H. Chagas (Eds.), Proceedings
of the XXVth International Symposium on Biomechanics in Sports
(pp. 176–179). Belo Horizontae, Brazil: Federal University of Mineas
Gerais.
Knudson, D. (2009). Significant and meaningful effects in sports
biomechanics research. Sports Biomechanics, 8, 96–104.
Knudson, D., Elliott, B., & Ackland, T. (2012). Citation of evidence for
research and application in kinesiology. Kinesiology Review, 1,
129–136.
Lamont, M. (2009). How professors think: Inside the curious world of
academic judgment. Cambridge, MA: Harvard University Press.
Lee, C. J., Sugimoto, C. R., Zhang, G., & Cronin, B. (2013). Bias in peer
review. Journal of the American Society for Information Science and
Technology, 64, 2–17.
Lundbrook, J. (1998). Multiple comparison procedures updated. Clinical
and Experimental Pharmacology and Physiology, 25, 1032–1037.
Luty, J., Arokiadass, S. M. R., Easow, J. M., & Anapreddy, J. R. (2009).
Preferential publication of editorial board members in medical specialty
journals. Journal of Medical Ethics, 35, 200–202.
Miller, C. C. (2006). Peer review in the organizational and management
sciences: Prevalence and effects of reviewer hostility, bias, and
dissensus. Academy of Management Journal, 49, 425–431.
Morrow, J. R., Jr, Bray, M. S., Fulton, J. E., & Thomas, J. R. (1992).
Interrater reliability of 1987–1991 Research Quarterly for Exercise and
Sport reviews. Research Quarterly for Exercise and Sport, 63, 200–204.
Morrow, J. R., Jr, & Frankiewicz, R. G. (1979). Strategies for the analysis
of repeated and multiple measures designs. Research Quarterly, 50,
297–305.
Mullineaux, D. R., Bartlett, R. M., & Bennett, S. (2001). Research design
and statistics in biomechanics and motor control. Journal of Sports
Sciences, 19, 739–760.
Park, R. J. (1980). The Research Quarterly and its antecedents. Research
Quarterly for Exercise and Sport, 51, 1–22.
Raff, H., & Brown, D. (2013). Civil, sensible, and constructive peer review
in APS journals. Journal of Applied Physiology, 115, 295–296.
Rothwell, P. M., & Martyn, C. N. (2000). Reproducibility of peer review in
clinical neuroscience. Brain, 123, 1964–1969.
Schroter, S., Black, N., Evans, S., Carpenter, J., Godlee, F., & Smith, R.
(2004). Effects of training on quality of peer review: Randomized
controlled trial. British Medical Journal, 328, 673–675.
Schwartz, S. J., & Zamboanga, B. L. (2009). The peer-review and editorial
system. Ways to fix something that might be broken. Perspective on
Psychological Science, 4, 54–61.
Smith, R. (2006). Peer review: A flawed process at the heart of science
journals. Journal of the Royal Society of Medicine, 99, 178–182.
Souder, L. (2011). The ethics of scholarly peer review: A review of the
literature. Learned Publishing, 24, 55–74.
Vecchio, R. P. (2006). Journal reviewer ratings: Issues of particularistic
bias, agreement, and predictive validity within the manuscript review
process. Bulletin of Science, Technology & Society, 26, 228–242.
doi:10.1177/0270467606288595
Wager, E., & Kleinert, S. (2012). Cooperation between research institutions
and journals on publication integrity cases: Guidance from the
Committee on Publication Ethics (COPE). Maturitas, 72, 165–169.
Ware, M., & Monkman, M. (2008). Peer review in scholarly journals:
Perspectives of the scholarly community—an international study.
Loodrecht, The Netherlands: Publishing Research Consortium.
Wheeler, B. (2011). The ontology of the scholarly journal and the place of
peer review. Journal of Scholarly Publishing, 42, 307–322. doi:10.1087/
20110109
Zhu, W. (2012). Sadly, the earth is still round ( p , 0.05). Journal of Sport
and Health Science, 1, 9–11.
IMPROVED PEER REVIEW 135
Copyright of Research Quarterly for Exercise & Sport is the property of Routledge and its
content may not be copied or emailed to multiple sites or posted to a listserv without the
copyright holder's express written permission. However, users may print, download, or email
articles for individual use.