Session 4
Maya Hadar
2
1. Conducting behavioral research on
Amazon’s Mechanical Turk
2. Twitter
 Twitter in the Czech Republic
3. Mining Twitter, a source for ‘wisdom of
the crowds’
On the Agenda for Today
3
Conducting Behavioral Research on Amazon’s
Mechanical Turk
*** The Turk, an 18th-century fake
chess-playing machine
4
„How to“ Guide to MTurk
 What is Amazon’s Mechanical Turk?
 Why Mechanical Turk?
 Who are the Workers?
 How to Become a Requester?
 How to Create a Study?
 How to Ensure Quality?
 What about Ethics and Privacy?
 Turker Communities and Useful Websites
Amazon‘s Mechanical Turk
What is Amazon‘s Mechanical Turk?
 Amazon’s Mechanical Turk is a crowdsourcing Internet marketplace that
enables individuals and businesses (known as ‘Requesters’) to coordinate the
use of human intelligence to perform tasks that computers are currently
unable to do
 A labor market for microtasks (Huang, Zhang, Parkes, Gajos, & Chen, 2010)
 A job outsourced to an undefined group of people in the form of an open call
(Howe 2006)
Amazon‘s Mechanical Turk
What is Amazon‘s Mechanical Turk?
 Large set of people willing to do task for relatively low pay
 Initially invented for human computation tasks, microtasks that are very
difficult/impossible for computers to perform: extracting data from images,
labeling images, filtering adult content, etc.
Amazon‘s Mechanical Turk
Why Mechanical Turk?
 Stable subject availability
 Large set of people + low cost
 Subject pool diversity (age, ethnicity, socioeconomic status, language, country
of origin)
 Low cost and built-in payment mechanism
 Faster theory-experiment cycle
 Validity of worker behavior (e.g. Paolacci et al. 2010)
 Replications of standard judgment and decision making tasks (Paolacci et al.
2010)
Amazon‘s Mechanical Turk
How does it work?
Requester Worker
HITs
Human Intelligent tasks
Amazon‘s Mechanical Turk
How does it work?
Who are the Workers?
 Countries of origin:
 100K-200K unique workers on Amazon (Difallah et al. 2018: Jan 2018)
 500k workers (Amazon)
 50% USA, 40% India (Ipeirotis, 2010) => Most of the workers are from the US
and India because amazon allows cash payments only in USD and Rupees
 Country of origin tends to change the motivation of workers to participate in the
marketplace
 India => the online marketplace is a primary source of income
 US => most workers consider Mechanical Turk a secondary source of income
Who are the Workers?
 Patterns of activity:
 On average, 2K-5K workers are active on Amazon at any given time
(equivalent to 10K-25K full-time employees)
 On average, 50% of the worker population changes within 12-18 months
 Workers exhibit widely different patterns of activity, with most workers being
active only occasionally, and few workers being very active
 MTurk has a yearly transaction volume of a few hundreds of millions of dollars
Amazon‘s Mechanical Turk
Who are the Workers?
 Mean age: 32 years
 Gender: Slightly more females (55%)
 Income: income level of US workers on Mechanical Turk is shifted towards lower
income levels (Ipeirotis, 2010)
 Main Motivation to participate in the marketplace => Money
 Other reasons => entertainment and education
Amazon‘s Mechanical Turk
 http://www.newyorker.com/culture/culture-desk/video-turking-for-respect
Amazon‘s Mechanical Turk
Who are the Workers?
 Create Requester Account and Amazon Payments Account
What you need =>
 E-Mail adress (advise: use unique email adress for running studies)
 Credit Card
 U.S. Billing adress (create one with International Parcel Services)
How to become a requester?
HITs Internal HIT (using Amazon Templates)
External HIT (Link to Study)
18
Creating a Study
Creating a Study
20TT.MM.JJJJ
Creating a Study
21
Creating a Study
Creating a Study
Payment =>
 10% service fee to amazon
 Reservation wage: $1.38 per hour
 Average effective hourly wage of $4.80 (Ipeirotis, 2010)
 Bonus
 Little to no effect of wage on quality of work (Marge et al., 2010; Mason &
Watts, 2009)
 Reject or accept work
Creating a Study
 Problem => Spammers, Bots
 Solution =>
 Set qualification criteria (e.g. 95% approval rate)
 Verifiable questions (e.g. “What is 2 + 2 ?“ ), attention checks
 Seriousness checks
 Make clear that Turkers won‘t be paid if the questions are incorrectly
answered
How to Ensure Quality?
What about Ethics and Privacy?
 Informed consent (purpose of the study, risks and benefits of the research,
contact information of researcher)
 Debriefing (purpose of experiment, contact details of researcher)
 Compensation: hours and working conditions wholly determined by workers
 Confidentiality: with template HIT, Amazon has access to data
Amazon‘s Mechanical Turk
Off-site Turker Communities
Off-site reputation
systems
 Turkopticon =>
 Turker Nation
 Rate requesters
based on
communicativity,
generosity,
fairness and
promptness
Off-site Turker Communities
Off-site reputation systems
 Turkopticon
 Turker Nation
 Rate requesters based on
communicativity,
generosity, fairness and
promptness
 Requesters should introduce themselves before posting hits
 Workers reactions to study can provide useful insights into method
 Keep the relationships with workers professional as if they were employees
 Experimental Turk https://experimentalturk.wordpress.com/
 Blog reporting evidence concerning the reliability of Amazon Mechanical
Turk as an online subject pool for experiments
 CrowdFlower https://angel.co/crowdflower/jobs
 A platform that enables data scientists to enrich their data by distributing
work to on-demand human contributors located around the world
28
Useful Websites
29
Mining Twitter, a Source for
Psychological Wisdom of the Crowds
Introduction
 Twitter is a Microblogging service => personal broadcasting media,
information and opinion are mixed together, usually tightly linked with current
reality
 A web application (also on mobiles) in which users can post text-based messages of up
to 140 characters (plans to increase to 280)
 Twitter also works as a social network, allowing its users to follow other users, group
them in lists, forward other users’ messages or send private messages
 Using location awareness and promptness (via the Internet) researchers can
detect changes in voiced (twittered) emotions, cognitions, and behaviors
Introduction
 iScience Maps is a free Web service for researchers, technologically based on
Twitter’s streaming and search application programming interfaces (APIs)
 Allows researchers to assess (via Twitter) the effects of specific events in
different places as they are happening, to compare cities, regions, or
countries and test the evolution of responses in the course of an event
• In the article, results from a study on affective and personality
characteristics inferred from first names are replicated by mining Twitter data
with iScience Maps in two regions (western U.S. and the U.K./Ireland)
Introduction
 Crandall and colleagues created
maps of world regions from ca. 35
million geotagged photos uploaded
to flickr
 These maps show relative interest
in motifs & places and may lead
to applications in tourism, city
planning, ecology, and economics
Introduction
 Social media => an interesting resource for social-behavioral research (the
“wisdom of the crowds”)
 The “wisdom of the crowds of researchers” => identify “hot topics”
(interdisciplinary)
Background
• Twitter has more than 321 million monthly active users (2018)
• Number of monthly active
Twitter users worldwide
from 1st quarter 2010 to
4th quarter 2018 (in
millions) =>
Background
• Twitter accounts with
the most followers
worldwide as of
February 2019 (in
millions)
Background
• Distribution of Twitter
users worldwide
from 2012 to 2018,
by region
Background
• Leading countries based
on number of Twitter
users as of January
2019 (in millions)
Background
• Percentage of U.S.
adults who use Twitter
as of January 2018, by
age group
Background
• Less than 1% of tweets contain information about the sender’s location
• Rough location can often be inferred from a user’s profile
• Simple Twitter search is available in many browsers and online applications,
from Twitter directly or via Twitter application programming interfaces (APIs)
• Simple Twitter search in the form of monitoring certain terms has been used as
early as 2010 in research concerning elections
39
http://www.slideshare.net/andrej_probst/twitter-in-czech-republic
Twitter in the Czech Republic
Twitter in the Czech Republic
http://www.slideshare.net/andrej_probst/twitter-in-czech-republic
May 2012
http://www.slideshare.net/andrej_probst/twitter-in-czech-republic
Twitter in the Czech Republic
http://www.slideshare.net/andrej_probst/twitter-in-czech-republic
Twitter in the Czech Republic
 iScience Maps for Twitter => a set of Web applications:
• Assisting researchers interested in social media analysis— mining tweets
• Developed in order to implement comparative searches of Twitter
• Targeted at behavioral researchers (almost all other available tools are
designed for personal Twitter users, marketing purposes, or a simple search)
 In iScience Maps, it is possible to combine terms, using Boolean operators,
compare searches for different locations and to download results in several
formats
Features:
 Temporal and geospatial content analysis + rich set of features for
comparative search options
 Trends within a date interval can be detected (via the Global Search
panel) and visualized as an animated movie using the Scalable Vector
Graphics based worldwide map’s animations
 Local Search also enables comparative searches
 Can calculate relative proportions of search term combinations in Twitter
for a given geo location
Features:
 Can combine the location information with aggregated data available via
zip code = useful in determining the extent to which tweeting on a particular
topic is concentrated in (wealthy communities across the U.S.)
 The results can be easily exported to the clipboard or to CSV or Excel format
 The tweets can be accessed
 Researchers can combine raw results from official Twitter APIs with refined
results taken from the iScience Maps API and can cross-check trends or
proportional ratios
• Two query panels grouped side by side; easily to compare searches between
places, date ranges, and search terms.
• Where?
• Location can be defined using the map to set the area range or by typing in
the text field above the map
• When?
• A horizontal scale with two sliders allows users to define a date range. Twitter provides
data only for the last 3–7 days
• What?
• Pressing the “+” button initiates a Boolean search for two search terms. Three operators
(and, or, and not) can be defined
Using the global and local search features
2 APIs to be queried:
 Local Search’s Twitter API: provides more
results than its alternative but …
 1,500 results limit
 Slow if there are too many results
 Limited date range (2,500 km)
Using the global and local search features
2 APIs to be queried:
 Global Search’s Twitter API:
 No result limits (date range or distance),
 Shows geotagged tweets only (no profile-based
location inference).
 Draws on a random sample of 1%–10% of all
tweets.
Using the global and local search features
 http://maps.iscience.deusto.es/#local
Using the global and local search features
Step By Step Example of Local Search
The goal =>
• Replicate a study on personality characteristics inferred from first names
(1993) Using 6 male names
• For 3 of the names (Alexander, Charles, Kenneth), strong connotation of the
dimension “successful” (“ambitious,” “intelligent”, “creative”***) was found
• Weak connotation was found for the rest (Otis, Tyrone, Wilbur)
 Hypothesis: If these names’ having the specific connotation of a personality
characteristic really holds, this should be apparent when Twitter is mined
(***e. g., “Charles is an intelligent guy”)
Step By Step Example of Local Search
The Method =>
• Searching for one name at a time, 3 days period (to avoid reaching the
threshold for number of tweets per search imposed by Twitter- 1,500)
In “Local search” tab =>
1. Define locations in the 2 map areas: Western U.S. and U.K. & Ireland (also
works with geographical coordinates of a point and radius)
2. “When?”: 3 days
3. Base rate: simple search for each name (What? “Otis”=> “Get Results.”)
4. Search for each name in combination with an attribute (“Charles” and
“intelligent”)
5. Comparing the U.S. west coast with the U.K. and Ireland.
Step By Step Example of Local Search
The Results =>
 All the high-connotation names indeed appear in the same tweets with some of
the aforementioned terms
 Opposite connotation search:
• US => No combination of low-connotation names with any of the terms
“successful,” “ambitious,” “intelligent” or “creative” was found
• UK => some combinations for 2 of the 3 high connotation names were found
*** The base rate of high-connotation vs. low-connotation names is a confounding
factor and may also explain the findings in the original study (less frequent names
may cognitively be less associated with any personality characteristics)
 Web services that were developed around the Twitter APIs (not specifically for
scientific research) may be useful in research projects:
 Monitter- monitoring the Twitter world in real time for a set of keywords and
watch what Twitter users are writing
 Web service providers which don’t provide search options but do provide
content:
• Twitter Halfhose (~50% of all Twitter content, delivered in real time)
• Twitter Link Stream (all Twitter statuses containing URLs, delivered in real time)
• Twitter User Mention Stream
iScience maps vs. other Twitter search services
Main drawback => Price (unsuitable for low-budget research initiatives)
Alternatives:
 Semiocast- provides semantic analysis services through a public API. This API
can be used to analyze, filter, and prepare Twitter statuses in terms of their
language or location. Allows up to 1,024 API calls per day for free.
 140kit- free web service, enables complete data pulls for a set of
users/terms on Twitter. Only provides on-demand Twitter data pulls and
content based filtering, not location based
iScience maps vs. other Twitter search services
Next Session...
60
 Internet research in political science
 Experimental research on the web- practical guidelines
 Discussion of experimental designs
61
Thank You For Your Attention!
Questions???