Session 6
Maya Hadar
2
 Seriousness check
 Your presentations
 My presentation
On The Agenda For today
Seriousness Checks
in Internet- based
research
Why Seriousness Checks?
 Who might provide non-serious answers?
 Random people, browsing the internet out of curiosity
 Other researchers interested in methodology
 Subjects, participating in experiments for financial gains only
 Non-serious answers…
 Increase statistical ‘noise’
 Reduces experimental power
Different Screening Methods (Aust et al. 2013)
 Consistency Check
 Checking for consistency and plausibility of answers
 Problem: double questions, longer experiment, annoying?
 Unique IP Check
 To control for multiple submissions
 Problem: multiple users might have same IP address
 Completion Time Check
 Exclude participants with very short completion times
 Problem: difficult to determine threshold
Different Screening Methods (Aust et al. 2013)
 Instructional Manipulation Check / Attention Check Questions
 Questions which look similar to ‘real questions’ with specific instructions to
do/avoid doing something
Seriousness Check
 Just Ask! Ask participants about the seriousness of their participation
 Goal: Analyze how data from self-reported non-serious participants differs
from serious ones and whether exclusion can increase validity
 Method: Online survey concerning political attitudes and voting intentions (prior
to the German 2009 federal elections)
 Hypotheses =>
 Serious participants would provide more consistent answers
 Exclusion of data originating in non-serious participants will increase
validity (‘Power of the test’- a test is valid if it measures what it is supposed to
measure)
Main Measures =>
 Correlation between self-reported political attitudes and sympathies
towards the main political parties
 Correlation between voting intentions and past voting behavior
Seriousness Check
Seriousness Check (after survey) =>
 “It would be very helpful if you could tell us at this point whether you have taken
part seriously, so that we can use your answers for our scientific analysis, or whether
you were just clicking through to take a look at the survey?”
 “I have taken part seriously”
 “I have just clicked through, please throw my data away”
 3.2% self-reported non-serious participation
 Rate of failing seriousness check in Reips (2009): 30%-50%
Seriousness Check
Non-serious participants compared to serious participants showed:
 Lower correlation between self-reported political attitudes and
sympathies towards the main political parties
 Lower correlation between voting intentions and past voting behavior
 Higher cumulative deviations of self-reported voting intentions from
official final election result
 Non-serious participants answered questions in a less consistent and
predictively valid way
Seriousness Check
Validation of Seriousness Checks
Consistency Check =>
 Inconsistent answers (reported household income lower than individual income)
Completion Time Check =>
 No significant difference between nonserious and serious participants
 Excluding fastest 10% of participants has only marginal effect on data validity
Unique IP Address Check =>
 No significant effect of multiple submissions from same IP address on validity and
consistency
 Seriousness check is the most effective way to screen out invalid data
Seriousness Check, Reips (2009)
 Seriousness check before the study:
 Single best predictor for dropout (around 75% of self-reported non-serious
participants will drop out)
Seriousness Check, Reips (2009)
Advantage =>
- Might increase motivation and decrease dropout rate
Drawbacks (Aust et al., 2013) =>
- May signal that non-serious responses are expected
- Can not capture potential change of mind during participation
Instructional Manipulation Check
(Oppenheimer, 2009)
 Participants who failed IMC:
 Produced less reliable data;
 Took significantly less time;
 Use IMC early in study;
 Potential risk of backlash:
 Participants may feel insulted to find that they are not trusted by the
researchers;
 Embarrassment may lead to foiling of study;
 May signal that non-diligence is expected;
Instructional Manipulation Check
(Oppenheimer, 2009)
Only usefull if
careful reading
of instructions
important for
experiment
Seriousness Check
Conclusions =>
 The most effective ways to screen out invalid data are:
 In studies without financial compensations: seriousness checks
 On platforms such as Mturk: reputation of workers
 Additional useful methods:
 IMCs / ACQ
 Completion time checks
 Consistency checks (only if certain combination of answers impossible)
To be tested in a future study => is there a difference in data quality if seriousness
check is applied BEFORE or AFTER the completion of the study?
Together we stand?
War Outcomes, Patriotism and Social trust: The
effect of variations in violent conflict outcomes on social
trust & national pride
On The Agenda
 Introduction
 Theory
 Hypotheses
 Methodology
 Findings
 Conclusions
 Conflicts are intrinsic to the life and dynamics of groups
 The research of conflicts’ economic effects and propensity of conflict cycles is
prominent in the literature
 The multidimensional nature of conflicts => can enhance positive group
attributes (cohesiveness) BUT can also enhance friction & polarization, leading to
a divided society
 The effect of conflict outcomes’ on national identification is neglected in the
literature
 Need to research the link between experiencing political violence & Individual
identities (social and national)
Introduction
Analyzing conflict outcomes’ effect on social trust & national pride
• How would individuals react when confronted with a situation of
increasing/diminishing group value in the context of a real life violent conflict?
• Research question => What is the effect of different war outcomes on social &
national Identification of citizens?
• Different conflict outcomes: victory\defeat\stalemate\agreement
• Primary contribution: Interdisciplinary study of Social identification theories
applied in a political context
Together we stand?
Theory => Social Identity Theory
 BIRGing (Basking in Reflected Glory)- Strategic impression management
technique, which enables individuals to raise their self-esteem in the eyes of
others by publicizing their connection with a successful other/s
 The one who basks in the glory of another has done NOTHING tangible to bring
about the group's success
 BIRGing In the sports realm => wearing school-identifying apparel and an
increase usage of “we” following school team victories
Together we stand?
Theory => Social Identity Theory
 Individuals are motivated to proclaim their associations
with others who are successful, and dissociate themselves
from others who fail
 We can expect individuals to identify less/ disassociate
themselves from their in-group following defeat (e.g.,
reluctance to deliver bad news as to avoid association with
the news + negative evaluations that follows)
Together we stand?
 CORFing (Cutting Off reflected failure) => image protection tactic, allows the
individual to avoid negative evaluations, resulting from association with
unsuccessful others
 H1: an increase in social + national identity’s saliency among
group-members will be demonstrated only following positively
valuated group performance
 H2: a decrease in social + national identity’s saliency among
group-members will be demonstrated following negatively
valuated group performance
 H3: no significant differences in identification levels among
group-members will be demonstrated between positively &
negatively valuated group performances
Hypotheses
Internet based research (web experiment) using Wextor
 Recruiting- mainly via social networks + academic mailing lists
• Four groups of participants:
 University students, not Poli-Sci/Peace research
 Students of Political Science or Peace Research
 Conflict researches
 Others
Methodology and Data
Internet based research (web experiment) using Wextor
 Participants
 321 visits from unique IPs were reported on the introductory page of the
experiment
 238 participants completed the experiment
 120 participants provided sufficient data to be analyzed
Methodology and Data
Procedure =>
 “You are a citizen of the country of Narvia, a peaceful country situated in a galaxy far far away.
Narvia is a beautiful country with an average temperature of 24 degrees Celsius, low unemployment
rate, high life expectancy, successful economy and rainbows. Since Narvia is a small country, people
are usually kind and friendly and enjoy camping, long walks by the beach and drinking bio-Narvia,
Narvia’s most popular and beloved drink”
The Conflict:
 “Two years ago, a meteor made of pure gold fell from the sky, right on the border between Narvia
and its’ neighboring country Barekia. Since both countries wanted to keep the gold and could not
agree on which side of the border the meteor fall, a war erupted”
Methodology and Data
• Experimental conditions (randomly assigned)
 Participants were given additional information related to the termination of
the conflict:
 Participants were presented with five statements and were asked to grade
their agreement on 5 points scale, ranging from “strongly disagree” (0) to
“strongly agree” (5)
Methodology and Data
Victory
Stalemate
Negotiated Agreement
Defeat
• Experimental conditions (randomly assigned)
 The statements referred to:
• National identity
• Sense of community
• Sense of belonging
• Self & Group relationship
Independent variable => Type of conflict termination
Dependent variables => Agreement with the statements
Methodology and Data
Participants
 120 participants
 44.17% females, 53.33% males
 Aged 20 to 69 with an estimated mean: 28.9. 11.67% of the participant > 40
National Identification & the Sense of Belonging
 The statements (“Narvia is my home”, “staying in Narvia is important to me” & “I’m
proud to be a Narvian”) addressed participants’ sense of belonging, secure
relationship and affinity to their country
 In large, conflict outcome did not significantly affect the level of agreement with
the statements relating to group member’s sense of belonging to their country and
to their national pride
Results and Discussion
National Identification & the Sense of Belonging
 Significant effect was found between the “victory” & “agreement” groups
 Agreement scores with the relevant statements across conditions
• Dotted line marks the overall average scoring
• Bold line marks the different average grade per condition
Narvia is my home Staying in Narvia is important to me I’m proud to be a Narvian
Results and Discussion
The Sense of Community
 “I don’t consider other Narvians as my friends”
addressed participants’ attitude towards
other Narvians, members of their ingroup
 High levels of agreement with the statement
were found across experimental conditions
 The effect was NOT statistically significant
Results and Discussion
The levels of agreement with statement 2
across conditions
Self and Group relationship
• The statement “I would like to see Narvia prospering” addressed participants’
attitude towards their nation, country
• Expected effect => individuals will report higher levels of identification with a
winning group
• However, we may observe strong identification of individuals with their low-status
group if their group identity is embedded in their self-identity and if they
perceive their fate as “linked” to their group (“linked faith”, Dawson 1995)
Results and Discussion
Self & Group relationship
 NON statistically significant effect
 No participant in the “defeat” group
indicated that he/she “strongly disagree”
(0) or even “disagree” (1) with statement
4
 None of the participants indicated the
highest level of agreement with the
aforementioned statement
Results and Discussion
Levels of agreement with statement 4
across conditions
Confounding variables =>
 Witnessing a conflict
 Political Science, Peace research & study of conflicts;
 Non significant
Limitations =>
 Sample size
 No baseline measurement
Future research => aggregating the current results + test cases
Results and Discussion
1. No significant effect => conflict outcomes & group members’ sense of community with
the group, sense of community
2. Low significant effect => conflict outcomes & national identification, sense of
belonging; “Narvia is my home“,
• “Victory” and “agreement” groups, NOT between the “victory” & “defeat”
groups, as suggested by the literature (based on the effect of group performance
on group identification)
*** The Data supported the hypothesis (H3) according to which poor performance (in
comparison with superior performance) did not result in a lower social identification
levels among group-members
Conclusions
36
Thank you!