INTERNET-BASED RESEARCH: THEORY AND APPLICATION SPRING 2019 Session 5 Maya Hadar MVZ507 § Web Experiments: Advantages & Disadvantages § Why conduct web-experiments? § Musch & Reips (2000), Reips (2000), Krantz & Dalal (2000) § Comparability to Traditional Research § Davis (1999) § Serioussness Check in Internet-based Research On The Agenda For Today Why Conduct Web Experiments? Musch & Reips (2000) Ø Sample Size Ø Statistical Power Ø High Speed Ø Ability to reach participants in other countries Ø High external or ecological validity Ø Low cost Ø Ability to replicate a lab experiment with more statistical power Ø Special populations However, web experiments are not without issues: Reips (2000) Ø Limited Sample Population Ø Limited External Validity Ø Less than Optimal Voluntariness Ø Motivational Confounding Ø Experimenter Bias Ø Non-transparency Ø Limitations of what is feasible to research Web Experiments: Deficiencies Is the Sample Representative? Krantz & Dalal (2000) Ø Important to remember the base of comparison (Ideal vs. Lab) Ø Much more diverse that most lab samples, especially in age and education range Web Experiments: Deficiencies Sample Characteristics: Gender § Is the sample gender biased? Krantz & Dalal (2000) % Females GVU 1st (1994) 5% Reips (1996): English German 43% 18% Krantz, et al. (1997) 44% Pasveer & Ellard (1998) 3rd Study 71% Caddell & Utt (2004) 77% § Is the sample age biased? § Are we still testing college sophomores? Papers Age groups Krantz, et al. (1997) 43% > 30 Smith & Leigh (1997) 35% > 30 Pasveer & Ellard (1998) 45% > 25 Caddell & Utt (2004) 60% > 30 Pattison & Rouse (2003) 16% > 30 Sample Characteristics: Age How diverse are the samples? § Limited diversity in the past § Depends on the requiting technique § Web experiments can make it possible to access special populations Papers % white respondents Krantz, et al. (1997) 89% white Smith & Leigh (1997) 86% white O’Neil, Penrod, & Bornstein (2003) 82% white Meyerson & Tryon (2003) 93% White Sample Characteristics: Race Where do the subjects come from? Largely North American (US), even in some European studies: § Krantz, et al. (1997) => 86% North America (study was conducted in the US) § Senior, et al (1999) => >80% North America (study was conducted in England) § Ease of reaching American subjects (e.g., Amazons’ mechanical Turk) § Big numbers § Depends on the requiting technique Sample Characteristics: Nationality Data Quality § Is the data, obtained from the web, any good? Ø Yes! (proven in replication papers) Ø Some studies still run both laboratory and web experiments => encouraging results § How to ensure the quality of online data? § Comparisons to off-line data Ø Studies comparing data obtained online to previously published data sets (obtained offline), E.g., Myerson & Tryon; 2003, Watt & Ewing; 1996 Ø Matched sample characteristics Ø Found same internal consistency Ø Form of administration was not a significant factor § Direct validity comparison Ø Should the same techniques (offline data) be used in order to validate results obtained online? Ø Yes Ø Not => Developed new scales (Pasveer & Ellard, 1998) Ø Check for Internal consistency Data Quality Recruiting Subjects General sample => Ø Social Psychology: http://www.socialpsychology.org/expts.htm Ø The Web Experiment List: http://genpsylab-wexlist.unizh.ch/ Ø Wextor: http://wextor.org/wextor/en/ Ø WEBEXP: Edinburgh webexp.info Special populations => Ø Advertising Ø Email lists Ø Discussion groups Ø Social networks Careful, get permission! Can be perceive as spam Careful, selection bias! Important Issues to Bare in Mind Ø No control over participant’s behavior Ø No control over motivation Ø Inability of participants to ask questions Ø Non-representative sample Ø Manipulation and fraud Ø Ethical problems Ø Eliminating multiple entries Ø Dropouts Ø Data integrity Musch & Reips (2000) Multiple entries Ø Double clicking, subjects click ‘submit’ multiple times while waiting for feedback Ø Subjects re-enter and re-submit, pose as other people (e.g., if rewards are given) Ø Same IP address Ø Smartphones/Mac vs. PC comparability issues Security & Data Integrity Ø Private server/Uni server => expensive, unavailable, more complicated to work with Ø Public services are often used for online storage => data is accessible to non authorized persons, can manipulate it/download it Ø Persons might fake pages to alter data (unlikely) Ø It is recommended to keep the data in non-public directories so only researchers have access Ø Check the data repeatedly + save an offline copy Dropouts Ø Subjects will start the experiment but won’t finish it => Incomplete data (unanswered questions) Ø Tracking (dropout rate) is important! Ø Will occur no matter what we do Ø We can try and motivate our subjects with/by: § Designing short experiments § Explaining why its important to stick around § Offer financial incentives (if possible), lottery? § Fast upload/warm up pages to speed up the upload Ethical Issues Ø Think what they may be (depends on the research topic and design) Ø Compare to ethics of an offline research (not only) Conclusions => Ø Many benefits to online experiments Ø Easy to design (technically), faster results, more date, cheap Ø Not a 100% alternative to traditional methods and research Ø Not without issues (relatively new tool), however, most issues can be solved or, at least, be accounted for Internet based Experimental Research The WWW holds great promise as a mechanism for experimental research: 1. Allows individuals to send data to the researcher at their convenience (time & location) => potentially more eligible research participants 2. Automatic transformation of raw data into an analyzable format (SPSS data file), using procedures such as CGI scripts 3. Efficient in terms of time and the resources it requires 4. Provides a degree of anonymity to research participants + eliminating observer bias § The study assessed the equivalence of the Ruminative Responses Scale (RRS) in a Web-based format and in a paper-and-pencil format *** Rumination a tendency to engage in passive thoughts and behaviors that focus one's attention on one's depressed mood and on the implications of these symptoms, rather than taking action Participants => ü Psychology collage students (2 different groups) in the US ü Non Psychology students ü A web-based student sample § Measurement technique => Assessing the extent to which people tend to respond to feelings of sadness or depression with self-focused rumination*** using a 10-items form. Findings => ü Students who completed a personality questionnaire on the Web reported higher levels of self focused rumination than did students who completed the same questionnaire in P&P format ü The internal Consistency of the web version was comparable with the P&P version ü Locations did not affect responses in any systematic manner ü Women in both P&P and Web samples reported higher levels of rumination than did men Robustness Checks => ü Was the web sample more motivated to participate in the study? Unlikely, since all groups were solicited at comparable, although not identical, moments of convenience ü Did the Web sample reported higher levels of self-focused rumination than did the other groups owing to computer anxiety? Unlikely, since: n Subjects in the Web sample volunteered n Previous research found greater levels of education to be associated with lower computer anxiety (participants were college students) Conclusions => ü By allowing people to have more control over their environment when they disclose information about sensitive issues, Web-based questionnaires may encourage increased frankness of response and self disclosure ü Some people may gain greater access to their feelings and personality when completing a Web-based questionnaire, compared with a questionnaire in a large lecture class/in a lab Conclusions => ü The internal consistency of the RRS was similar across the four groups. This finding indicates that results from the Web version are comparable with those obtained in the other samples ü Important to note => Although permitting access from multiple locations decreases experimental control, it does NOT appear to affect questionnaire results adversely Conclusions => ü The results of the present study increase confidence in the use of the Web as a method for collecting questionnaire data ü Restrictions: Participants were sampled from a college campus and were restricted in terms of age range Next Session... § In 2 weeks- paper’s presentations Thank You For Your Attention! Questions???