1
Good Research Practices
by A.
2
Outline
̶ Before...
̶ Replication crisis
̶ Science and degrees of freedom
̶ ...After
oReplications
oTransparency
oOpen science
oEffect sizes
oPre-registration of studies
̶ Your decision!
̶ Building brands
3
Born: June 10, 1938, in
Denver, Colorado
PhD: in social psychology
from the University of
Michigan, 1964
Status: started his
academic career as a
respected mainstream social
psychologist
4
Before...
̶ Publication bias, "file-drawer effect"
̶ "Publish or perish"
̶ Novelty
̶ Impact
...and doubts
5
Born: June 10, 1938, in
Denver, Colorado
PhD: in social psychology
from the University of
Michigan, 1964
Status: started his
academic career as a
respected mainstream social
psychologist
6
D.Bem (2003) - Writing the Empirical Journal Article
7
Daryl Bem
̶ Self-perception theory
̶ Exotic becomes erotic theory
̶ Precognition: Feeling the future
8
Feeling the Future: Experimental Evidence for Anomalous
Retroactive Influences on Cognition and Affect (2011)
̶ Uni: Cornell University
̶ Journal: Journal of Personality and Social Psychology
̶ 8-year study
̶ 9 experiments
̶ Over 1,000 participants
̶ Finding: people can predict future
9
Born: October 19, 1966, in
Oegstgeest, Netherlands
PhD: in social psychology
from the University of
Amsterdam in 1997
Status: a prolific and talented
scientist but one of the most
significant cases of academic
fraud in modern psychology
10
Diederik Stapel
̶ fabricated data for at least
55 papers
̶ exposed by a group of
three junior researchers in
2011
11
Diederik Stapel
12
Replication crisis
13
Replication crisis
̶ The Marshmallow Test (Walter Mischel, 1972)
o much weaker relationship
̶ The Stanford Prison Experiment (Philip
Zimbardo, 1971)
o methodological flaws
̶ Power Posing (Amy Cuddy, 2010)
o non-replicable results regarding hormonal changes
̶ Ego Depletion (Roy Baumeister, 1998)
o overstated effect
̶ Social Priming (John Bargh, 1996)
o non-replicable
̶ The Facial Feedback Hypothesis (Strack, 1988)
o non-replicable
̶ Priming Intelligence with Stereotypes (Claude
Steele, 1995)
o mixed evidence
14
Replication crisis in science
15
Science and degrees of freedom
Statistics
̶ Data
o1, 2, 3, 4, 5
̶ Mean = parameter
o(1+2+3+4+5)/5 = 3
̶ Df = N –1
o5-1 = 4
̶ Why?
o4 numbers can vary independently, but...
othere is just one option for the 5th number
to get the same mean value
o2 + 2 + 3 + 3 + 5 ...mean = 3
o3 + 3 + 3 + 3 + 3 ...mean = 3
Research
̶ Research goal
̶ Multiple ways of reaching it
16
Science and degrees of freedom
Good research
practices (GRPs)
Questionable research
practices (QRPs)
Bad research
practices
17
Science and degrees of freedom
Good research
practices (GRPs)
Questionable research
practices (QRPs)
Bad research
practices
P-hacking
HARKing
Cherry-picking
Data-fishing
Salami publication
Fabrication of data
Falsification of data
Plagiarism
Ghostwriting
Conflict of interest
Preregistrations
Appropriate N
Open science
Replications
Transparency
The Nine Circles of Scientific Hell
18
After...
19
Replications
̶ Direct / exact replication
̶ Systematic replication / replication-plus-extension
̶ Conceptual replication
20
Transparency
̶ What was done?
̶ How was it done?
̶ Is that all that has been
done?
̶ Has anything been
modified? Preregistration?
̶ What are the limits?
̶ Were there any problems?
̶ Has anything been done
additionally?
21
Open science
̶ Open access (paid)
̶ Sharing data, code,
materials...
̶ Transparency
̶ Free software (R, JASP,
jamovi)
̶ Knowledge should be free
and available to everyone
oSci-Hub, LibGen
22
Effect sizes
̶ Sufficient sample size
̶ Statistical vs. practical significance
23
Effect sizes
̶ Sufficient sample size
̶ Statistical vs. practical significance
Does treatment reduce social anxiety?
Treatment 1
Treatment 2
24
Effect sizes
̶ Sufficient sample size
̶ Statistical vs. practical significance
Does treatment reduce social anxiety?
Treatment 1
Treatment 2
25
Effect sizes
̶ Sufficient sample size
̶ Statistical vs. practical significance
̶ Power analysis (G*power)
26
Pre-registration of studies
27
Registered ReportPreregistration
• practice of registering the hypotheses,
methods, and/or analyses of a scientific study
before it is conducted
• format of empirical article where a study proposal
is reviewed before the research is undertaken
Introduction Methods Results Discussion
Research paper (IMRAD)
28
Registered ReportPreregistration
• practice of registering the hypotheses,
methods, and/or analyses of a scientific study
before it is conducted
• format of empirical article where a study proposal
is reviewed before the research is undertaken
Introduction Methods Results Discussion
Research paper (IMRAD)
Collect dataBuild an idea State hypotheses
Specify how to
test your
hypotheses
Analyze data
& describe the
results
Discuss the
results
29
Introduction Methods Results Discussion
Research paper (IMRAD)
Collect data
Build an idea
State hypotheses
Specify how to
test your
hypotheses
Analyze data
& describe the
results
Discuss the results
30
Introduction Methods Results Discussion
Research paper (IMRAD)
Collect
data
Build an
idea
State
hypotheses
Specify how to
test your
hypotheses
Analyze data
& describe the
results
Discuss the
results
Change the
hypotheses
Specify how to
test your
hypotheses
31
Introduction Methods Results Discussion
Research paper (IMRAD)
Collect
data
Build an
idea
State
hypotheses
Specify how to
test your
hypotheses
Analyze data,
use all possible
tests
Discuss the
results
Choose the
best analysis
Describe the
results
32
Replication crisis
33
Registered ReportPreregistration
• practice of registering the hypotheses,
methods, and/or analyses of a scientific study
before it is conducted
• format of empirical article where a study proposal
is reviewed before the research is undertaken
Introduction Methods Results Discussion
Research paper (IMRAD)
Collect dataBuild an idea State hypotheses
Specify how to
test your
hypotheses
Analyze data
& describe the
results
Discuss the
results
34
Registered ReportPreregistration
• practice of registering the hypotheses,
methods, and/or analyses of a scientific study
before it is conducted
• format of empirical article where a study proposal
is reviewed before the research is undertaken
Introduction Methods Results Discussion
Research paper (IMRAD)
Collect dataBuild an idea State hypotheses
Specify how to
test your
hypotheses
Analyze data
& describe the
results
Discuss the
results
35
Registered ReportPreregistration
• practice of registering the hypotheses,
methods, and/or analyses of a scientific
study before it is conducted
• format of empirical article where a study proposal
is reviewed before the research is undertaken
Public web
platform
Locking the first half of
the journal article before
data collection.
Journal
36
Registered ReportPreregistration
• practice of registering the hypotheses,
methods, and/or analyses of a scientific
study before it is conducted
• format of empirical article where a study proposal
is reviewed before the research is undertaken
Registered Reports (cos.io)
Public web
platform
Journal
37
Preregistration
• practice of registering the hypotheses,
methods, and/or analyses of a scientific
study before it is conducted
OSF Registries | Searching for an affordable method of modern response
time analysis in measuring the analytic/holistic cognitive style
OSF Registries | Method of Loci: The capacity of individual
places of the memory palace
Public web
platform
38
Registered Report
• format of empirical article where a study proposal
is reviewed before the research is undertaken
Advertising cooperative phenotype through costly signals facilitates
collective action | Royal Society Open Science (royalsocietypublishing.org)
39
̶ PCI-RR community
̶ Popularizing GRPs
"Because the study is accepted in advance,
the incentives for authors change from
producing the most beautiful story to the
most accurate one."
--Chris Chambers
40
Your decision!
41
Building brands
42
Building brands
̶ Researcher's limits & biases
̶ Resources and facilities
̶ Unexpected problems
̶ Workplace requirements
43
Thank you
...and use magic responsibly!
alexandra.ruzickova@mail.muni.cz