A Brief History of Web Experimenting
Jochen Musch Ulf-Dietrich Reips
Psychological Institute Experimental and Developmental
University of Bonn Psychology
D-53117 Bonn University of Zurich
Germany CH-8032 Zurich
Switzerland
A small, but growing number of researchers have begun to use the
World Wide Web as a medium for experimental research. To learn more
about the circumstances and results of the first Web experiments, wc conducted
a WWW-based online survey directed to researchers currently engaged
in Web experimenting. We hoped to get an impression of the experiences
of the pioneering generation of Web researchers. We summarize the
results of this survey, which showed that an increasing number of Web
experiments with promising results is now being conducted, and give a brief
overview on the short history of Web experiments.
THE HISTORY OF WEB EXPERIMENTS
The introduction of computerized experimenting in the 1970s (e.g.,
Connes, 1972; Hoggatt, 1977) revolutionized traditional laboratory research.
Among the attractive new features were a standardized and controlled presentation
of stimuli, item-branching capabilities, immediacy of data entry, elimination
of missing responses, elimination of transcription costs and errors, and
accurate measurements of response times. Adaptivity, interactivity, and ease of
data storage and analysis were additional advantages of the new technology. It
has also been argued that the use of computers reduces the tendency to respond
in a socially desirable way (Booth-Kewley, Edwards, & Rosenfeld, 1992;
61
Psychological Experitnents on the Internet
Copyright © 2000 by Academic Press. All rights of reproduction in any form reserved.
62 Jochen Musch and Ulf-Dietrich Reips
Martin & Nagao, 1989) and helps to avoid experimenter biases and demand
characteristics (Hewson, Laurent, & Vogel, 1996; Reips, chap. 4, this volume;
Smith & Leigh, 1997). Clearly, the computerized administration of experiments
and questionnaires offered possibilities unavailable in traditional paperand-pencil
research. With hindsight, it is hardly surprising therefore that the
computer revolution in experimental psychology in the 1970s was an overwhelming
success.
Twenty years later, most human experimental research in psychology is
aided by computer automation. Extending computerized experimenting beyond
single PCs, local computer networks have been used for the collection of
data (Hoffman & MacDonald, 1993). Programs are written in high-level
languages such as C + + , Pascal, or Delphi, or with program packages such as
SuperLab, PsyScope, MEL (Micro Experiment Laboratory), and ERTS (Experimental
Run Time System). Although computerized experiments have
become the method of choice in conducting psychological research, there are
many signs that another revolution is now beginning. It is associated with the
recent exponential growth of the Internet.
The Internet's early purpose in the 1960s was to link a U.S. Defense
Department network called the Advanced Research Projects Agency Network
(ARPAnet) with a variety of other radio and satellite networks (Abbate, 1994;
Hardy, 1995). In the 1980s, Ethernet local area networks were developed to
allow computers at a single site to connect to a time-sharing computer site
(Salus, 1995). These capabilities were extended to include access to the
ARPAnet. The latest and most influential part of the Internet, a global
hypertext system called the World Wide Web, was born in the early 1990s at
the European Laboratory for Particle Physics (CERN). Within a few years it
became clear that it might become the basis of the next generation of
computerized experiments in psychology.
Although the term **hypertext** was coined by Theodor Nelson in 1960,
the concept of hypertext dates back to the work of Vanevar Bush (1945).
Around 1980, Tim Berners-Lee wrote a notebook program based on the
hypertext concept and called it **Enquire-Within-Upon-Everything.** The
program created links between arbitrary nodes which were given a title and a
list of bidirectional typed links. While working at CERN, Berners-Lee proposed
a hypertext project in 1989 that extended his program to a more global
level. It was created under the pretense that many people would be able to
work collaboratively by putting information on a web of hypertext documents.
The documents would be put on servers and client software, called a
browser, would allow one to access the information stored on the server. The
software calls up the information by searching for a link's uniform resource
locator (URL). It then uses the hypertext transfer protocol (HTTP) to get the
document, which is coded in hypertext markup language (HTML). The
Chapter 3 Brief History of Web Experimenting 63
world's first Web browser ran on a NeXt computer at CERN in 1990. In 1991,
Marc Andreesen and a group of students at the National Center for Supercomputing
Applications (NCSA), located on the campus of the University of
Illinois at Urbana Champaign began work on what would be the first publicly
available browser, called Mosaic. Mosaic, released in 1993, was the first
point-and-click graphical user interface for the World Wide Web. The ability
to combine words, pictures, and sounds on Web pages excited many computer
programmers who saw the potential for publishing information on the Internet
in a way that can be as easy as using a word processor.
When the people that previously constructed Mosaic started developing
the Netscape Navigator in 1994, they decided to implement several new
HTML features, and the Navigator soon became the dominant tool for
browsing the Web. A little later, Microsoft came along and created another
browser also based on HTML, the Internet Explorer. HTML—the hypertext
markup language—became the lingua franca for publishing on the World
Wide Web.
The World Wide Web Consortium (W3C), the most prominent members
of which are Netscape and Microsoft, works on and publishes proposed
recommendations of new HTML specifications. To date, the W3C has released
three versions of HTML. The first one, in 1995, was HTML 2.0, which was a
simple language that attempted to gather the various previous implementations
and concatenate them into a concrete specification. (Whatever existed earlier
is collectively called HTML 1.0, although there never was such a specification).
Then came HTML 3.2, which aimed to gather all the most popular features
Netscape had introduced into a concrete specification. HTML 3.2 still is the
most reliable specification to refer to for simple applications and allows for the
creation of simple Web pages. W3C's latest HTML specification is version 4.0,
which was released in December 1997. It is supported by the latest version of
both Netscape (version 4.0) and Internet Explorer (version 4.0). Today,
starting a browser and going online gives the user access to millions of Web
servers and hundreds of millions of HTML pages. Having gone through several
stages of evolution, HTML 4.0 has a wide range of features reflecting the
needs of a very diverse and international community wishing to make information
available on the Web. Among the most important of these features are
forms.
Forms (or **fill-out forms") were first introduced in HTML 2.0. They are
interactive, unlike the typical static Web page, and their introduction changed
things in an important way. For the first time, the reader of a Web document
could communicate back to the server. Forms in HTML are the computer
equivalent of paper forms, such as an application form. There is a button or
link at the end of every WWW form, often labeled **Submit.'* When this
button is pushed, two things are sent to the server: the data that were typed
64 Jochen Musch and Ulf-Dietrich Reips
into the form, and an ACTION, which basically tells the server the name of
the program that knows how to process that formes data. The server simply
invokes that program and passes the form's data to it (using CGI, the common
gateway interface), and arranges for the output of that program to be sent
back to the browser (ordinarily another document that contains some form of
feedback).
Common uses of forms are surveys, online order forms, or really any
Web page in which input is required from the user to accomplish a given task
or provide a service to the user. Of course, for a psychologist, sending a
participant's experimental or questionnaire data back to the experimenter is
the most interesting application of forms. Drawing on forms, the W W W first
offered the possibility of conducting psychological surveys and experiments
independent of any geographical constraints.
HTML was soon supplemented by JavaScript, a compact, cross-platform,
object-based scripting language that was first supported in version 2.0 of the
Netscape Navigator and was also adopted by Microsoft in version 3.0 of its
Internet Explorer (Flanagan, 1998). JavaScript code is embedded directly into
the HTML page and can be used to create interactive Web pages. The browser
interprets the JavaScript statements embedded in an HTML page and executes
them. JavaScript statements can respond to user events such as mouse-clicks,
form input, and page navigation. For example, JavaScript functions can be
used to verify that users enter valid information into a form requesting a fixed
number format. Without any network transmission, the HTML page with
embedded JavaScript can check the entered data and alert the user with a
dialog box if the input is invalid.
Another important technology became available in 1995, when James
Gosling and a team of programmers at Sun Microsystems released an Internet
programming language called Java. It again radically altered the way applications
and information could be retrieved, displayed, and used over the Internet.
Client-side Java applets are small programs that are transmitted over the Web
and run on the user's machine, offering a large variety of possibilities for
sophisticated experiments. Java was first built into version 3.0 of the Navigator
and version 3.0 of the Explorer. Owing to these technological developments
and its exponential growth during the past few years, the World Wide Web
presents researchers with an unprecedented opportunity to conduct experiments
with participants from all over the world rather than with the usual
student sample from their local universities. It thus has the potential to serve as
an alternative or supplemental source of subjects and research environment for
traditional psychological investigations (Buchanan & Smith, 1999; Reips, 1996a,
1996b, 1997b; Smith & Leigh, 1997). Using the Internet to conduct research
offers several advantages over traditional research practices (see Hewson et
al., 1996, and Reips, 1995b, 1996a, 1996b, 1997b, this volume, for a summary).
Chapters Brief History of Web Experimenting 65
In a sense, using worldwide networks such as the Internet for experimental
research therefore is the logical next step in what began with the first
experiments on stand-alone computers (cf. Buchanan & Smith, 1999).
However, the use of the WWW as a medium for experimental research
also poses a unique set of challenges (Reips, 1999a). To learn more about the
circumstances and results of the recently growing number of Web experiments,
we conducted two online surveys directed to researchers currently
engaged in Web experimenting. We thus hoped to get an impression of the
experiences of the pioneering generation of Web researchers. In the present
chapter we summarize the results of this survey and make an attempt to write
the early history of Web experiments.
As in writing a general history of the Internet (e.g., Musch, 1997), the
frequency and ease with which WWW documents are changed, combined
with the lack of an effort to comprehensively collect those documents during
the first years of the WWW, make it a difficult task to determine what really
happened when. This difficulty is even reflected in recommendations for
references to online documents (e.g., Ott, Kriiger, & Funke, 1997), which
advise adding the lookup date to the reference. On the other hand, the WWW
is still very young, the number of Web experiments is rather small, and so
people*s memory (including our own) should be still fresh.
In the fall of 1994, when one of us began planning what later became the
Web*s Experimental Psychology Lab (Reips, 1995a), an Internet scan for Web
experiments produced no results. However, before the Web*s Experimental
Psychology Lab went online with its first two experiments in September 1995
there were already a few experiments online, as we later discovered. Obviously,
the time for Web experimenting had come.
As it seems, the very first Web experiments were Norma Welches (1995)
experiments on auditory perception, which were simultaneously run at McGill
University, Montreal, Canada, and Technical University, Darmstadt, Germany
(Welch & Krantz, 1996). However, as Krantz and Dalai (chap. 2, this volume)
put it, these experiments **could not be really called studies carried out over
the Web*' as they were attached to tutorials in auditory perception. In May
1995, Andreas Weigend and a class he taught at Colorado University put up
three Web experiments on music recognition (Weigend, 1995). Unfortunately,
the code of these Web experiments was lost when Weigend left Colorado
University (A. Weigend, personal communication, November 6, 1998), and we
were not able to determine whether these studies were really experiments in
the sense that some variable was manipulated (as contrasted with an online
questionnaire). A little later John Krantz and colleagues started their Web
experiment on the determinants of female attractiveness (Krantz, Ballard, &
Scher, 1997). This might well have been the first true Web experiment that
66 Jochen Musch and Ulf-Dietrich Reips
went online. It appears to be the first psychology Web experiment that was
published in a scientific journal.
Krantz et al. used a within-subjects design, the first Web experiment
with a between-subjects design appears to be the Web experiment on cognitive
consistency of causal mechanisms (Reips, 1996a, 1996b, 1997b), with which
the first virtual experimental psychology laboratory (Reips, 1995a) opened its
doors. Between-subject designs require random assignment of participants to
experimental conditions, which can be realized in Web experiments through
the use of CGIs (Kieley, 1996; Reips, 1996b, 1997b, 1999a), JavaScript, or Java.
Today, the Web*s Experimental Psychology Lab at Tubingen (Reips, 1995a),
which is now at Zurich, is still a place for methodological discussions on Web
experimenting and actively invites participation of experiments from other
researchers which can be hosted by the lab.
Since 1995, the following sites (with their opening dates) have gone
online:
• Interactive CyberLab for Decision-Making Research ( h t t p : //
www. e t 1. go. j p / -- e6930, April 1996)
• Laboratory of Social Psychology Jena ( h t t p : // www. uni j
ena. de / '^ ssw / l a b o r . htm, June 1996)
• Experimental Server Trier ( h t t p : // cogpsy. u n i t
r i e r . d e : 8000 / TEServ- e . html, June 1997)
• Max-Planck Institute for Biological Cybernetics Tiibingen
( h t t p : // exp. kyb. tuebingen.mpg. d e / web- experiment /
ndex.html, November 1997)
• Online Psychology Lab Padua
( h t t p : // w w w . p s y . u n i p d . i t / p e r s o n a l / l a b o r a t o r i o /
s u r p r i s e / h t m l t e s i / i n d e x . h t m l , May 1997)
• Decision Research Center ( h t t p : // psych, f u l l e r t o n . e d u /
mbirnbaum / d e c . html) (started online experiments in March 1998)
• Psycholinguist Laboratory Scotland ( h t t p : // surf . t o /
experiments, September 1998)
• PsychExps ( h t t p : // www. o l e m i s s . edu / PsychExps /, fall
1998; invites participation of Web experiments from other researchers)
• Systems Analysis Lab at Helsinki University ( h t t p : // www.hut. f i
/ U n i t s / SAL)
• Jonathan Baron*s questionnaires ( h t t p : // www. psych. upenn. edu
/ '^ baron / q s . html)
Some additional WWW laboratories engage in teaching or demonstration of
experiments, for example, the excellently designed Internet Psychology Lab
( h t t p : // kahuna.psych . u i u c . edu/ i p l / ), which went online in
April 1998. However, all of the laboratories in this list are mainly used for
chapter 3 Brief History of Web Experimenting 67
experimental data collection. In addition to the preceding list of Web laboratories
and sites where Web experiments are being conducted, a growing
number of other sites have gone online to offer participation in psychological
experiments.
The most comprehensive list of Web experiments on the W W W can be
found on the Psychological Research on the Net page (American Psychological
Society, 1995), which was created and is maintained by John Krantz. John
Krantz and many (probably more than half) of the other currently active Web
experimenters generously agreed to participate in our survey on the experiences
of the first generation of Web researchers.
METHOD
All respondents were recruited via the Internet. To promote its existence,
we announced the Web experimenter survey to the following mailing
lists:
• PSYCGRAD (Psychology Graduate Student Internet Project)
• RESEARCH (Psychology of the Internet: Research and Theory)
• GIR-L (German Internet Research List)
• SCiP (Society for Computers in Psychology)
Additional invitations to participate were posted to the following Usenet
newsgroups:
• sci.psychology.research
• sci.psychology.announce
• sci.psychology.misc
• alt.usenet.surveys
• bit.listserv.psycgrad
• de.alt.umfragen
• de.sci.psychologie
• de.sci.misc
• z-netz.wissenschaft.psychologie
Personal invitations were sent via e-mail to all researchers who announced a
Web experiment at one of the following places:
• The Psychological Research on the Net page of the American Psychological
Society, maintained by John Krantz
http: // psych. hanover. edu / APS / exponnet. html
68 Jochen Musch and Ulf-Dietrich Reips
• The list of Online Social Psychology Studies by Scott Pious
http: //www.wesleyan.edu / spn / expts .htm
• Ulf-Dietrich Reips* Web Experimental Psychology Lab in Zurich and
Tiibingen
ht tp: // www. psych. uni zh. ch / genpsy / Ul f / Lab /
WebExpPsyLab.html
• The Psychology/Tests-and-Experiments pages at Yahoo International
h t t p : // www.yahoo.com/ S o c i a l - S c i e n c e / Psychology/
D i s c i p l i n e s / P e r s o n a l i t y / Online-Tests /
• The Psychology/Tests-and-Experiments pages at Yahoo Germany
h t t p : / / w w w . y a h o o . d e / G e i s t e s w i s s e n s c h a f t e n /
P s y c h o l o g i e / O n l i n e - T e s t s - u n d - V e r s u c h e /
Finally, we sent a call for participation to a number of researchers that
we knew had conducted or planned to conduct a Web experiment.
In a first wave of the survey, we received 21 submissions from Web
experimenters between October 17 and October 30, 1998. Additional submissions
came from researchers who had conducted surveys. Conducting surveys
on the Web is a promising new way of online research (cf. Batinic, 1997;
Batinic, Graf, Werner, & Bandilla, 1999). However, because we were interested
in Web experiments rather than surveys, we did not include them in the
analysis. The criterion used for classifying submissions as describing experiments
was that at least one independent variable was manipulated.
In this first wave of the survey, we told the respondents that if they
conducted more than one Web experiment, they should answer all questions
with regard to their first Web experiment. A second wave was online from
April 16 to April 28, 1999. In this second wave, we asked participants who had
already participated in the first wave to answer some questions with respect to
the last Web experiment they had conducted. First-time participants were
asked to describe the first experiment they had conducted. The number of
questions was reduced for the second wave of the survey, which was announced
to the same mailing lists and to the same newsgroups to which the
first wave had been announced. In addition, it was announced to SJDM, the
mailing list of the Society for Judgment and Decision Making. There were 14
submissions from researchers who conducted a Web experiment in the second
wave of the survey. Additional submissions from researchers who conducted a
Chapter 3 Brief History of Web Experimenting 69
survey rather than an experiment were not included in the analysis. Thus, the
final sample consisted of 35 submissions from 29 different researchers currently
engaged in Web experimenting. The regional distribution of the 29 researchers
was as follows: Germany (8), United States (7), United Kingdom (6), Canada
(1), Austria (l), Switzerland (l), Australia (l), Soviet Union (l), Finland (l),
New Zealand (l), Unknown (l). After the data were collected and analyzed,
all respondents were provided with a summary of the results. An additional
WWW scan and notes from the Web Experimental Psychology Lab*s archives
(Reips, 1995a) showed that at least eight more people have conducted Web
experiments, but did not respond to either wave of our survey.
PROCEDURE
The survey consisted of three WWW pages and was written in HTML,
the computer programming language most often used to display pages on the
WWW. On the first page survey participants were greeted and informed
about the rationale for conducting the survey. Also, participants were told that
only online data collection was considered a Web experiment that met the
definition of **any undertaking in which some variable is manipulated; thus, in
contrast with a survey, at least two conditions must be involved in an
experiment.'* Then, participants were asked to indicate the number of Web
experiments they had conducted and to provide us with their e-mail address
for feedback and possibly additional questions. Most researchers had conducted
one ( N = 15) or two (N = 6) Web experiments at the time of the survey.
Eight experimenters already had conducted a higher number of studies (ranging
from 3 to 20 experiments).
Submission of the first page of the survey sent the form data (i.e., the
data that were filled in by the respondent) to a server-side plug-in (Mailagent
1.1, Netdrams Software, 1998), which wrote the data to a tab-delimited file.
Also, it triggered the display of the second survey page in the respondent's
Web browser window. This second page contained almost all of the questions
asked. A variety of WWW specific answering formats was used. For example,
to answer the question on announcement media respondents had to use their
mouse arrow to click on checkboxes. Checkboxes allow for multiple nonexclusive
selections. To limit responses to one out of several possible answers both
possibilities offered by HTML were used: a question about the area of
research, for example, was presented as a pop-up menu, while a question about
the use of counterbalancing had to be answered by clicking on radio buttons.
Such electronic answering formats, which allow for only one answer, have
been shown to reduce the frequency of response errors (Kiesler & Sproull,
1986). Certain answers, especially to open-ended questions such as the question
70 Jochen Musch and Ulf-Dietrich Rcips
on the experimental factors used, required text fields. Text fields can be limited
to a maximum number of characters. When a survey respondent sent off the
second page by clicking on its submit button, again the data were sent to
the Web server (WebSTAR by StarNine Technologies) and processed by the
Mailagent plug-in. In addition to writing the data to a file the plug-in sent an
e-mail to the respondent and the experimenters to report the processing of the
data.
RESULTS
When did you start the experiment? When did you end the experiment?
John Krantz and colleagues from Hanover College started their first
Web experiment in April 1995. We are not aware of any psychology Web
experiment with at least two conditions (i.e., two levels of an independent
variable) that appeared on the W W W before this date. The number of Web
experiments has been constantly rising since, with the majority of Web studies
starting during 1997 and 1998.
In the first wave of our survey, we told the respondents that if they
conducted more than one Web experiment, they should answer the following
questions for their first Web experiment for which they had reported the
starting date.
How important were the following factors for your decision to conduct
the Web experiment ( 1 , not important at all; 7, very important)?
Mean SD N
large number of participants
high statistical power
high speed
ability to reach participants from other countries
high external or ecological validity
low cost
ability to replicate a lab experiment with more power
chance to better reach a special subpopulation on the Web
(e.g., handicapped, rape victims, chess players)
The factor that experimenters rated as most important was reaching a large
number of participants. The high statistical power associated with a large
5.5
4.5
3.6
3.6
3.4
3.2
2.9
2.6
1.9
2.2
2.4
2.2
2.1
2.2
2.5
2.5
20
20
20
20
20
21
20
20
3.6
3.4
3.3
2.9
2.8
2.4
1.5
2.0
1.7
1.7
2.1
1.8
1.4
1.0
21
21
21
21
21
21
21
Chapter 3 Brief History of Web Experimenting 71
sample size, the high speed with which Web experimenting is possible, and the
chance to reach participants from other countries were also considered important
by most of the respondents.
How problematic do you think were the following potential problems in your
Web experiment (1, not problematic at all; 7, very problematic)?
Mean SD N
no control of a participant's behavior during participation
no control of participant's motivation
inability of participants to ask questions
no control of participant's hardware and equipment
nonrepresentative samples
manipulation and fraud
ethical problems
The biggest concern of the Web experimenters participating in the first
wave of our survey was the lack of control of a participant's behavior during
participation. However, a numeric value of 3.6 translates to not more than an
assessment of this lack of control as "somewhat problematic.** Ethical problems
were not considered a problem by most Web experimenters. Obviously,
it is important to note that all these ratings came from researchers who
themselves are conducting Web experiments, and the results may well have
been different if another sample of researchers had been asked.
In which media did you announce your experiment?
Naturally, the WWW was used most often to promote Web experiments
(22 out of 35 experiments were promoted on the Web). Many researchers
also relied on newsgroups (18), e-mails (15), and search engines (14)
to advertise their experiment. Few experiments were also announced in print
media (2) and radio programs (l).
How many design factors did you include in your experiment? For each of
these factors, please specify the number of levels on which it was varied.
The mean number of factors participants of the first wave of the survey
indicated. they had manipulated was 2.1 (with a median and a mode of 2.0).
There were six experiments in which as many as three or more factors were
72 Jochen Musch and Ulf-Dietrich Reips
varied. The following designs were reported in the first wave of the survey:
Levels per factor N
5 X 5 X 2 X H 1
5 X 5 X 4 1
5 X 3 X 2 1
3 X 3 X « 2
3 X 2 X 2 1
5 X 2 2
3 X 3 1
3 X 2 1
2 X 2 5
H X 2 1
3 1
2 4
not specified 1
60% of the designs involved between-subjects factor manipulations, another
20% of the designs involved within-subjects factor manipulations, and 20% of
the designs involved both kinds of experimental manipulation.
In what area of research did you conduct the experiment?
To aid the respondents* decision, we offered a list of 54 subject areas
used for classifying poster contributions to the 10th Annual Convention of the
American Psychological Society (1998). Just as one would expect from a
theoretical perspective (Reips, 1997b), the Web experimental method obviously
seems best suited for cognitively oriented areas of research (see, e.g.,
Klauer, Musch, & Naumer, 1999). The areas in which most experiments were
conducted are Cognition (4), Thinking and Reasoning (3), Psycholinguistics
(2), Sensation/Perception (l), Memory (l), Judgment/Decision Making (l),
Attention/Performance (l). Personality (l), Social/Groups (l), Social/Cognition
(l), Social/Attitudes (l), and Computers in Psychology (l).
What was the main hypothesis? What were other hypotheses?
Please name the factors you varied and their levels.
The following selective list gives an impression of the theories and
hypotheses that were tested in Web experiments and the independent variables
that were manipulated. Hypotheses that are difficult to understand without
sufficient knowledge of the subject domain or that were not explained by the
survey respondents are not included in the list.
chapter 3 Brief History of Web Experimenting 73
Judges violate stochastic dominance in coalesced gambles, but satisfy
stochastic dominance when gambles are presented in split form (high
versus low variance of gambles; .01, .50, .99 probability of winning
higher prize; value of prizes; combined versus split consequences of
gamble; stochastic dominance between gambles). There is a lower
degree of violation among people with more training and education in
judgment and decision-making.
Comparison of Bayes Theorem, Subjective Bayesian Model, and Averaging
Model of Inference—participants use the base rate in the
taxicab problem contrary to the idea of base rate fallacy (base rate,
witness credibility, witness report).
Persons relying on schema-based interpretation of incoming text information
are less likely to have accurate recall than aschematic readers
(schematic versus aschematic text, temporal delay between reading
and recall).
Pronominal case-ambiguous objects elicit a preference toward accusative
case assignment, whereas nonpronominal, case-ambiguous objects
elicit no preference (object noun phrase type is ambiguous
nonpronominal, ambiguous pronominal, or unambiguous pronominal;
object case is accusative or dative).
Answers to questions in online surveys are potentially subject to biases,
depending on the number of questions per page (one, two), scale type
(pop-up, radio buttons), reading directionality (from left-top, from
right-bottom), cursor entry position (top, bottom), question order
(donation question first, expense question first), and numerical labeling
( - 5 to + 5 , 0 to 10).
Experts use their specific and their general knowledge for data evaluation
(expertise, high versus low; data are in accordance with versus in
contradiction to expert knowledge).
Subjects perceive feminine male and female faces as more attractive
(masculinity of male face, more feminine to more masculine).
Background color influences response to emotionally laden statements
(different shades of background color, print color white versus red).
Web questionnaires produce response effects that are similar to paperand-pencil
questionnaire response effects (questionnaire length is short
or long).
People with low context constraint encode individual words deeper
(context constraint is low or high).
Syllogisms with believable conclusions are accepted more often than
unbelievable ones, irrespective of their validity (**belief bias**). The
suggested base rate of valid conclusions influences the willingness to
accept a given conclusion if the participant cannot determine its
74 Jochen Musch and Ulf-Dietrich Reips
validity (suggested base rate of valid conclusions is low, medium, or
high; validity of conclusion is valid or invalid; believability of conclusion
is believable or unbelievable).
• What kind of errors are being made when reconstructing 2D information
from a 3D model (object colors, object shape, map orientation)?
• Is syntactic priming (in German verb-final constructions) sensitive to
subcategorization information or linear precedence, or both? What is
the baseline proportion of ditransitive versus simple transitive responses,
given different target types (prime type is accusative < dative,
dative < accusative, accusative, dative, or control; target type is
accusative object or dative object)?
• Previously acquired knowledge about causal mechanisms influences
later acquisition and use of causal knowledge involving those mechanisms
(acquired type of causal mechanism is consistent, inconsistent, or
none).
• Syntactic primes affect completions of targets; there is a possible
interaction with case, especially dative (prime type is neutral, ditransitive-accusative~dative,
ditransitive-dative-accusative, transitive-accusative,
or transitive-dative; target type is accusative or dative).
• Participants recruited from newsgroups likely to be read by people
who are high self-monitors score higher on a WWW-mediated version
of the Self-Monitoring scale than participants recruited through
newsgroups likely to be read by people who are low self-monitors
(likely self-monitoring is low or high).
• Does bimodal presentation (hearing, seeing) of objects increase memory
performance? Does contradictory bimodal presentation decrease
memory performance (acoustic naming of pictures shown is no acoustic
naming, true naming, or wrong naming).
• Content of sites visited before answering a survey systematically
influences answer behavior; time of watching instructions influences
strength of context effects (content of simulated W W W site is pictured
or not; social desirability via instruction is high or low).
• Important computer-specific concepts are explained in more detail if
addressee is a beginner (concept importance is important or less
important; addressee's knowledge level is beginner or advanced).
Did you assign participants to the experimental conditions randomly?
If yes, what did you use to obtain random assignment?
Of the 34 experimenters who answered this question, 29 assigned participants
to the experimental conditions in a random order. (Four of the remaining
experiments varied conditions within subjects, and one study was a quasi
Chapters Brief History of Web Experimenting 75
experiment). The techniques that were used most often to obtain random
assignment were CGI, Java, and JavaScript. Few respondents indicated they
used yet another (unspecified) technique to obtain random assignment. (Birth
month, birthday, social security number, etc. are alternative criteria that can
be used to obtain random assignment, for example, by offering different
hyperlinks to participants born in odd vs. even months of the year; cf.
Birnbaum, 1999a, in press).
Did you apply counterbalancing of any sort?
Although the use of counterbalance is certainly an important aspect of
internal control in many experimental designs, only a little more than half of
the experiments (20 out of 35) applied counterbalancing of some kind. This
may in part be attributable to the technical and programming difficulties
associated with the use of counterbalancing. Five of the 14 experiments that
did not make use of counterbalancing measures were one-factor experiments,
however, which may have been simple enough not to require this technique.
What did you do to guard against multiple submissions?
Entry errors, the intent to foil the experiment, and curiosity are just
some of the possible reasons respondents may have for submitting their
answers more than once; Because all WWW browsers have a **back button**
built in, it is not easy to prevent users from reexamining their Web form page
and submitting it again, possibly after altering their data (Schmidt, 1997; cf.
Schmidt, chap. 12, this volume). However, several techniques exist that can be
used to deal with the problem (Buchanan & Smith, 1999; Reips, 1996b, 1997b,
1999a; Schmidt, 1997; Smith & Leigh, 1997). Most of them try to uniquely
identify each participant and to filter out all suspicious submissions. This can
be done by examining a variety of aspects of the data provided by the
participants. We wanted to know what data Web experimenters used to tackle
the problem of multiple submissions. In many of the 35 experiments, researchers
relied on checking the e-mail (24) and IP addresses (18). Both
variables help to uniquely identify the participants. However, e-mail addresses
can be faked, repeated participation may take place from a different computer
address, and proxy servers can make the submissions of different users falsely
appear to have been sent from one and the same person. Perhaps because of
these problems, passwords were used in addition to control of e-mail and IP
addresses in three experiments. In two experiments, no special precautionary
measures against multiple submissions were taken.
76 Jochen Musch and Ulf-Dietrich Reips
Several researchers also used the following procedures to secure data
integrity:
• Cookies (3)
• Date or time (2)
• Bank account (2)
• Asking not to participate more than once (2)
• Asking for seriousness of the submission (l)
• Checking for identical records arriving in close temporal interval (l)
One researcher told us that he felt any attempts to cheat by multiple
submissions would have been very unattractive regarding the low reward
participants received in his experiment.
On what technical HTML level was your Web experiment programmed?
Advanced HTML levels offer a number of sophisticated layout features
such as tables, frames, and style sheets. However, the lower the HTML level
used, the wider the population of potential respondents will be. This is because
not all users have installed the latest version of their Web browser, and older
versions do not support the latest HTML features.
Only three researchers made use of features of the latest HTML 4
version. The vast majority of researchers preferred a conservative approach
and restricted themselves to the more established HTML versions 2 and 3.
Which of the following techniques, programming languages,
and tools did you use?
A large variety of techniques and programming languages is being used
to implement Web experiments. CGI was used by 19 respondents; other
popular tools that were named are Perl (13), JavaScript (ll), Java client-side
(lO), Java server-side (7), and Cookies (4). A few experiments were also
conducted using VRML (2), C / C + + (2), DHTML (l), XML (l), and
ActiveX (1).
What operating system was your Web server running on?
To conduct a Web experiment, one requires access to a W W W server
(Kieley, 1996). This can be practically any machine connected to the Internet:
an account on a multiuser UNIX system or a networked PC or Macintosh is all
the hardware that is required (Reips, 1996b, 1997b; Schmidt, Hoffmann, &
MacDonald, 1997). Because the software available for implementing Web
experiments largely depends on the type of operating system that is installed
on the WWW server, we asked participants of the first wave of the survey
chapters Brief History of Web Experimenting 77
which operating systems they used. Answers indicated a clear preference for
Unix and Unix derivatives such as Solaris, Linux, and so on, followed in this
order by MacOS and Windows 95/98/NT.
In which language(s) did you offer your experiment?
There were 17 experiments that were offered in English and 10 experiments
that were offered in German. Offering experiments in parallel versions
for different languages reliably increases the number of participants and allows
for cross-cultural comparisons. This possibility was used by seven researchers.
What was the total number of participants included for final data analysis
(after deletion of multiple submissions etc.)?
Respondents indicated the final sample size for 26 of the 27 experiments
that were already finished at the time of the survey. The mean number of
participants in these experiments was 427, with a standard deviation of 650.
The median number of participants was 158. The smallest number of participants
was 13 and the largest was 2649.
Can you tell (in percent) how many of those who looked at the very first
access page to your experiment decided to start the experiment?
Several experiments recorded how many of the visitors of the first access
page proceeded to start the experiment. However, it is important to note that
the definitions of "the first access page*' to a web experiment may vary quite
extensively, as indicated by the responses. This first access page could be the
main page of an online laboratory, an informed consent page, the first
instructions page, or the last page of a "warm-up phase** (a technique used to
reduce dropout during Web experiments; see Reips, 1996b, 1997b, 1999a). On
average, 68% (median, 80%) of all visitors of the very first access page
proceeded to start the experiment (N = 10). The minimum click-through rate
was 7%, the maximum was 100%. The chance to obtain this information is a
unique advantage of Web experiments if one considers the number of visitors
to the first access page of a Web experiment as equivalent to the number of
people who consider or are asked to take part in a standard laboratory
experiment. In the case of a laboratory experiment, the researcher usually does
not know how many students decide not to sign up for an experiment (or
would have decided to do so if given the right to choose among studies).
Can you tell (in percent) how many of those who started the experiment
by looking at the first page after the instructions page did complete it?
78 Joclien Musch and Ulf-Dietrich Reips
On average, 66% (median, 65%) of the participants who started an
experiment also completed it ( N = 20). The respective figures ranged from 13
to 99%. In 15 experiments, the dropout rate was not (yet) determined; four of
these experiments were still running at the time of the survey.
Did your participants have to download a plug-in or an applet first
to be able to participate?
The only plug-ins used were VRML (virtual reality modeling language),
offering the opportunity to display three-dimensional objects, and Macromedia
Flash. With these two exceptions, all participating researchers indicated that
their experiments did not require the (somewhat time-consuming and troublesome)
download of a plug-in. (Note, however, that the PsychExps group
[McGraw, Tew, & Williams, this volume] at the University of Mississippi
uses the Shockwave plug-in to deliver its experiments; / h t t p : / / w w w .
o l e m i s s . e d u / P s y c h E x p s / ) . Eight experiments required the download
of an applet prior to participation.
Did you ever have a problem with a hacker trying to intrude your
experiment? If yes, please give a description of what happened.
On the W W W , nothing prevents anybody from downloading and examining
the HTML source code of Web experiments. Hackers can attempt to foil
the experimental results, and they may even try to gain control over the Web
server (Schmidt, 1997). One possibility to do so is to send manipulative data to
the CGI program for processing.^ We wanted to know whether Web experimenters
had faced such problems.
No respondent indicated having observed a hacker attack. Of course, this
does not mean that there were no such attacks. Although one may assume that
successful attempts to damage the experiment would have been noticed, the
possibility that a hidden hacker went undetected cannot be excluded.
Which demographic questions did you ask your participants?
To know which denizens of the Web chose to participate in their online
experiments, investigators have asked a variety of demographic questions. The
variables Web experimenters gathered most often from their participants were
age (32 out of 35), sex (32), occupation (16), language (ll), nationality (9),
insider knowledge concerning the field of research (7), and education (6).
To prevent this, experimenters can set up their CGI program to restrict acceptance to
data from only certain referers.
chapter 3 Brief History of Web Experimenting 79
Several researchers asked respondents to provide additional data re-
garding
• Their handedness (3)
• Their marital status (2)
• Their dialect (2)
• Their circumstances of housing or job
• Their religion
• Their socioeconomic status
• Their status as student or nonstudent
• Their college major
• Their class (for credit assignment)
• Their name
• The Web browser they used
• The size and color (yes or no) of their monitor display
• The speed of their Internet connection
Did you provide the participants with the possibility
to contact you via e-mail?
The vast majority of researchers (94%) offered an e-mail link, thus
allowing participants to ask questions, to make comments, and to point out
errors. The chance to obtain this kind of feedback is a notable advantage of
Web experiments (Welch & Krantz, 1996).
Did your experimental material contain any graphics or sounds?
Advanced facilities for viewing graphics are available in the HTML
language and 16 out of 35 experiments contained graphics in some form.
A modality less frequently used than graphics in a typical experiment is
sound. Only two experiments were based on acoustical stimuli. This might
well be the result of the technical difficulties associated with the large
differences in the hardware and software used to interpret and play sound files,
and in the audio formats they can handle (cf. Welch & Krantz, 1996).
Did you measure any reaction times (below 1 second)?
In spite of the technical difficulties associated with reaction time measurement
on the World Wide Web, eight experiments made use of such
measurements. This surprisingly high number might be due to the fact that
many Web servers automatically log access time in fractions of a second.
Did you measure any time intervals (above 1 second)?
80 Jochcn Musch and Ulf-Dietrich Reips
The measurement of time intervals provides valuable information about
the participant's behavior during the experiment, and 14 out of the 34
experiments recorded time intervals.
Monetary rewards can be expected to increase both the motivation and
the number of participants.
Did you offer a monetary reward to each participant?
Three out of 34 experiments offered a monetary reward to each participant.
In these experiments, the amount every respondent received was $15 (on
average), $13, and DM 10.- (approximately $6).
Did you offer monetary or other prizes in a lottery? If yes, what was the
total amount of money (the value of your prizes) in dollars?
Ten experiments offered a lottery. The total value of the prizes was
$1224, $750, $500, $192, $150, $100, $100, $100, $85, and $11, respectively. One
experiment offered a 1% chance to actually play (with real money) one of the
attractive gambles that were part of the experiment. To investigate whether
financial incentives have a beneficial effect, we conducted some additional
analyses. We found that if some kind of monetary compensation (individual
payments or lottery prizes) was offered, the percentage of participants who
completed the experiment was significantly higher (86%, N = 13) than if no
rewards were at stake (55%, N = 7; t = 2.61, rf/= 18, p < .05). Although the
sample sizes are very small, this observation might indicate that financial
incentives can help to reduce the number of dropouts. There was a similar
(though nonsignificant) trend for a higher number of participants per week
when a financial incentive was offered (49 versus 37 participants per week).
On average, how much time did it take to participate
in your experiment (in minutes)?
The average duration of the experiments was 22 minutes ( N = 33), with
a median of 15 minutes. The minimum duration that was reported was 5
minutes, the maximum was 90 minutes.
Did you inform your participants in advance about the
duration of the experiment?
The 29 experiments which informed potential participants in advance
about how much time they would need to complete it attracted a higher
number of participants (mean N = 496) than the 5 experiments that did not
provide this information (mean N = 230). However, because of the small
chapter 3 Brief History of Web Experimenting 81
sample sizes and the large variance in participation rates, this difference was
not significant.
Did you offer an individual feedback to each participant?
Web experiments, as other computer-based forms of experimenting,
offer the opportunity to provide dynamic and interactive forms of feedback to
the participants (Schmidt, 1997). This feedback can be specifically tailored to
the responses given, and it can provide summary statistics about the results of
other respondents. It is thus possible to give respondents interesting pieces of
information in return for their efforts, probably a powerful motivation to
participate in an experiment. Schmidt (1997) conjectured that if respondents
know that the feedback they receive is about themselves and based on the data
they provide, they are likely to supply accurate and thoughtful responses.
However, apparently because much effort is needed to implement an individualized
feedback, only 9 out of 34 experiments included this feature. Of these 9
experiments, 6 provided feedback immediately after participation, and 3
provided feedback with some temporal delay.
Did you offer feedback concerning the goals and hypotheses of the
experiment to the participants?
Most experiments offered feedback about what was being investigated,
either immediately after participation (lO) or with some temporal delay (16).
In eight experiments, participants were not offered feedback about the goals of
the study.
Did you offer feedback concerning the design of the experiment?
About half of the experiments conveyed the experimental design to the
participants, either immediately after participation (4) or with some temporal
delay (l2). Participants in 18 experiments were not informed about the design
of the study in which they participated.
Did you also run the experiment with non-Internet participants
(for comparison)? If yes, how would you rate the convergence of
Web and lab data? If there was less than complete agreement,
please describe the differences.
To assess the validity of their findings, 18 experimenters conducted a
replication study in a traditional setting. (The results of another 5 replication
studies were not yet available at the time of the survey).
Almost all experimenters observed complete or good agreement between
their Web and lab data. There were no experiments for which a lack of
82 Jochen Musch and Ulf-Dietrich Reips
agreement between lab and Web data was observed. Low agreement between
data collected in a traditional versus an Internet-based setting was found in one
case where an effect of stimulus material was found in the lab but not in the
Web data. The author of this experiment conjectures that the stimulus images
were too small to be seen properly in the Web version of his experiment.
Higher variances in the Web were observed in one experiment for which the
experimenter otherwise observed good agreement between lab and Web data.
Another experimenter also described the agreement between lab and Web
data as good but observed slightly more outliers (in the form of extreme or
illogical judgements) in the Web version of his experiment. The author of an
experiment that showed only partial agreement between the Web and the lab
data assumed that this difference might be due to the small sample size in his
lab experiment. Another researcher observed that Internet participants were
more highly educated than the participants in his lab sample and showed lower
rates of violation of stochastic dominance in their choices.
Overall, how much time (person hours) did you put into the Web
experiment, compared to a similar laboratory experiment?
If you plan to conduct another Web experiment, how much time
do you expect to need, compared to your first Web experiment?
And how much money did you put into the Web experiment,
compared to a similar laboratory experiment?
There was no clear pattern with regard to the question whether first-time
Web experiments need less working time than more traditional approaches;
about one third of participants indicated they needed more time for the Web
experiment than they would have needed for a similar laboratory experiment,
another third needed about the same amount of time, and another third
indicated that they needed less time for the Web experiment. A majority of
researchers stated that in their opinion, much time can be saved when more
than one Web experiment is conducted. This seems consequential regarding
the effort one has to put into learning and implementing a whole new research
method. Naturally, the difficulties of getting the first Web experiment running
will be reduced in subsequent Web experiments, and the computer setup as
well as other materials can be recycled (although rapid changes in technology
may create obsolescence quickly). Experimenters also indicated that Web
experimenting is a quite inexpensive undertaking. The vast majority of respondents
estimated the costs of Web experiments to be smaller or much smaller
than the costs of a traditional laboratory experiment.
Overall, how much space or how many rooms did you need for the
Web experiment, compared to a similar laboratory experiment?
Chapter 3 Brief History of Web Experimenting 83
Another benefit of using the Web for experiments stems from savings of
space and rooms needed for laboratory experiments (Reips, 1995b, 1996a,
1996b, 1997b; chap. 4, this volume). All researchers indicated that they needed
less or much less space for their Web experiment than they would have needed
for a laboratory experiment.
Are you planning to conduct another Web experiment?
The experiences Web experimenters reported seem overwhelmingly
positive. With only one exception, all researchers said they would certainly, or
at least perhaps, conduct another Web experiment.
Did you present your experiment to a scientific conference or is your
presentation accepted for an upcoming conference? If yes, which conference?
More than two thirds of the data of already finished Web experiments
have been presented to a scientific conference or have been accepted for
presentation at an upcoming conference. On the following occasions Web
experiments were (or will be) reported:
• Virtual Reality Modeling Language Conference, 1998
• European Conference on Visual Perception, 1998
• Annual Workshop on Object Perception and Memory, 1998
• Society for Computers in Psychology, 1996, 1998
• British Psychological Society, 1998
• German Online Research, 1998
• Economic Science Association, 1998
• Conference of the Society for Research in Experimental Economics
(Tagung der Gesellschaft fiir Experimentelle Wirtschaftsforschung),
1998
• Annual Convention of the Canadian Psychological Association
• Congress of the German Psychological Society, 1998
• Experimental Psychologists* Conference (Tagung Experimentell Arbeitender
Psychologen), 1998, 1999
• Annual Conference of the Text and Discourse Society
• Annual Conference of the Cognitive Science Society, 1999
• Annual Conference of the German Linguistic Society (jahrestagung
der Deutschen Gesellschaft fiir Sprachwissenschaft), 1999
• International Society for Research in Emotion Conference, 1998
84 Jochen Musch and Ulf-Dietrich Reips
Did you already write a paper on your experiment? If yes, did you
report comparison data from non-Web participants? Which journal
did you submit it to? What is the current status of the first
submission of your paper? If the paper was not accepted when you
first submitted it, what were the stated reasons? If the paper
was not accepted, did you submit it to another journal afterward?
The respondents had written 12 papers on their Web experiments by the
time of the survey, 5 of which reported comparison data from a traditional lab
sample. Several manuscripts were not yet completed. The journals to which
the completed manuscripts were submitted to are:
• Behavior Research Methods, Instruments^ and Computers (three submissions-two
manuscripts were accepted; one was rejected but not
because Web data were reported; the manuscript has not yet been
resubmitted)
• Journal of Economic Behavior and Organization (under review)
• Journal of Behavioral Decision Making (not accepted for reasons other
than the Web data)
• British Journal of Psychology (accepted but wanted it as a short report;
the authors elected to resubmit elsewhere rather than rewrite)
• Psychological Review (under review)
• Psychology and Marketing (not accepted for reasons other than the Web
data)
Four articles and a book by survey participants are already in press or
published (Birnbaum, 1999a, 1999b, in press; Morrow & McKee, 1998; Schmalhofer,
Elst, Aschoff, B'arenfanger, & Bourne, in press). However, the number
of Web experiments that already found their way into scientific journals is still
very low (but see Hanze, Hildebrandt, & Meyer, 1998; Krantz et al., 1997;
Welch & Krantz, 1996). Of course, one reason for this is that the first Web
experiments were started only a few years before our survey. It remains to be
seen whether the editors and reviewers of traditional journals will accept the
growing number of papers reporting data from Web experiments.
Web experiments offer a number of advantages compared to traditional
laboratory experiments. This has mostly been explored theoretically or concluded
from experience with very few Web experiments (Krantz & Dalai,
chap. 2, this volume; Reips, 1995b, 1996a, 1996b, 1997b; chap. 4, this volume)
and, of course, a number of traditional experiments. Our survey shows for the
first time on an empirical basis that many of those advantages hold true in the
impressions of those who have used Web experiments as a method for doing
research.
chapters Brief History of Web Experimenting 85
Although online data collection efforts are undertaken in many areas of
research (American Psychological Society, 1995), it seems that the core areas
for Web experiments are those that deal with cognition. Combined with the
practical advantages and the ease with which Web experiments can be
conducted this might lead to an increase in cognitive research, especially in
light of the integration of online research and online publication (Reips, 1997a,
1998).
Taken together, we feel that the results of this survey give a number of
interesting insights into the experiences of the first generation of Web
experimenters. At the moment, the number of Web experiments is still small,
but a rapid growth can be predicted on the basis of the present results. We
would not be surprised if within the next few years, a fair proportion of
psychological experiments will be conducted on the Web. It will be interesting
to see how the experiences of the early researchers shape this new direction in
psychological research.
REFERENCES
Abbate, J. E. (1994). From Arpanet to Internet: A history of ARPA-sponsored computer networks,
1966—1988, Unpublished doctoral dissertation, University of Pennsylvania.
American Psychological Society. (1995). Psychology experiments on the Net [WWW document].
Available URL: ht tp: // psych. hanover. edu / APS / exponnet. html
American Psychological Society. (1998). [WWW document]. Available URL: h t t p : / /
www.psychologicalscience.org
Batinic, B. (1997). How to make an Internet based survey? In W. Bandilla & F. Faulbaum (Eds.),
SoftStat '97—Advances in statistical software 6 (pp. 125-132). Stuttgart: Lucius & Lucius.
Batinic, B., Graf, L., Werner, A., & Bandilla, W. (1999). Online research. Gottingen: Hogrefe.
Birnbaum, M. H. (1999a). How to show that 9 > 221: Collect judgments in a between-subjects
design. Psychological Methods, 4, 243-249.
Birnbaum, M. H. (l999b). Testing critical properties of decision making on the Internet. Psychological
Science, W, 399-407.
Birnbaum, M. H. (in press). Introduction to behavioral research on the Internet. Upper Saddle River, NJ:
Prentice-Hall.
Booth-Kewley, S., Edwards, J., & Rosenfeld, P. (1992). Impression management, social desirability,
and computer administration of attitude questionnaires: Does the computer make a
di(ierencc7 Journal of Applied Psychology, 11, 562-566.
Buchanan, T., & Smith, J. L. (1999). Using the Internet for psychological research: Personality
testing on the World Wide Web. British Journal of Psychology, 90, 125-144.
Bush, V. (1945). As we may think. Atlantic Monthly, 1, 101-103.
Connes, B. (1972). The use of electronic desk computers in psychological experiments. Journal of
Structural Learning, 3, 51-72.
Flanagan, D. (1998). Javascript: The definitive guide (3rd ed.). Cambridge: 0*Reilly.
Hanze, M., Hildebrandt, M., & Meyer, H. A. (1998). Feldexperimente im World Wide Web: Zur
Verhaltenswirksamkeit des "mere-exposure"-Effekts bei der Informationssuche [Field experi-
86 Jochen Musch and Ulf-Dietrich Reips
ments in the World Wide Web: Effectiveness of the "mere exposure" effect on information
seeking]. Psychologische Beitrage, 40, 363-372.
Hardy, H. E. (1995). A short history of the Net [WWW document]. Available URL: h t t p : / /
www.ocean, i c . n e t / ftp / doc / snethistnew.html
Hewson, C. M., Laurent, D., & Vogel, C. M. (1996). Proper methodologies for psychological and
sociological studies conducted via the Internet. Behavioral Research Methods, bistrumertts, &
Computers, 28, 186-191.
Hoffman, R., & MacDonald, J. (1993). Using HyperCard and Apple events in a network
environment: Collecting data from simultaneous experimental sessions. Behauior Research Methods,
Instnnnents, & Computers, 25, 114-126.
Hoggatt, A. C. (1977). On the uses of computers for experimental control and data acquisition.
American Behavioral Scientist, 20, 347-365.
Kieley, J. M. (1996). CGI scripts: Gateways to World-Wide Web power. Behavior Research Methods,
Instruments, & Computers, 28, 165-169.
Kiesler, S., & Sproull, L. S. (1986). Response effects in the electronic survey. Public Opinion
Quarterly, 50, 402-413.
Klauer, K. C, Musch, J., & Naumer, B. (1999). On belief bias in syllogistic reasoning. Manuscript
submitted for publication.
Krantz, J. H., Ballard, J., & Scher, J. (1997). Comparing the results of laboratory and World-Wide
Web samples on the determinants of female attractiveness. Behavioral Research Methods, Instruments,
& Computers, 29, 264-269.
Martin, C. L., & Nagao, D. H. (1989). Some effects of computerized interviewing on job applicant
responses. Journal of Applied Psychology, 74, 72-80.
Morrow, R., & McKee, J. (1998). CGI scripts: A strategy for between-subjects Web designs.
Behavior Research Methods, Instruments, & Computers, 30, 306-308.
Musch, J. (1997). Die Geschichte des Netzes: Ein historischer AbriB [The Net's history: A
synopsis]. In B. Batinic (Ed.), Internet fur Psychologen [internet for psychologists] (pp. 27-48).
Gottingen: Hogrefe.
Mailagent 1.1 [Computer software]. (1998). Rohnert Park, CA: Netdrams Software.
Ott, R., Kriiger, T., & Funke, J. (1997). Wissenschaftliches Publizieren im Internet [Scientific
publishing on the Internet]. In B. Batinic (Ed.), Internet fur Psychologen [Internet for psychologists]
(pp. 199-220). Gottingen: Hogrefe.
Reips, U.-D. (1995a). The Web*s Experimental Psychology Lab [WWW document]. Available URL:
ht tp: // www. psych. uni zh. ch / genpsy / Ul f / Lab / WebExpPsy Lab .html
Reips, U.-D. (1995b). The Web experiment method [WWW document]. Available URL: h t t p : / /
www.psych.unizh.ch/ genpsy / Ulf / Lab / WWWExpMethod.html
Reips, U.-D. (1996a, April). Experimentieren im World Wide Web [Experimenting in the World
Wide Web] [Abstract]. Experimentelle Psychologie: Tagung experimentell arbeitender Psychologen
[Proceedings of the experimental psychologist's conference], Germany, 38, 256~257.
Reips, U.-D. (1996b, October). Experimenting in the World Wide Web. Paper presented at the 1996
Society for Computers in Psychology conference, Chicago.
Reips, U.-D. (1997a). Forschen im Jahr 2007: Integration von Web-Experimenticren, OnlinePublizieren
und Multimedia-Kommunikation [Science in the year 2007: Integration of Web
experimenting, online publishing, and multimedia communication]. In D. Janetzko, B. Batinic,
D. Schoder, M. Mattingley-Scott, & G. Strube (Eds.), CAW-97. Beitrage zum Workshop
''Cognition & Web" [Articles presented at the workshop "Cognition & Web**]. Freiburg:
IIG-Berichte 1/97.
Reips, U.-D. (1997b). Das psychologische Experimentieren im Internet [Psychological experimenting
on the Internet]. In B. Batinic (Ed.), Internet fur Psychologen [Internet for psychologists] (pp.
245-265). Gottingen: Hogrefe.
Chapter 3 Brief History of Web Experimenting 87
Reips, U.-D. (1998). Forschung in der Zukunft [Future science]. In T. Kruger & J. Funke (Eds.),
Psychologie im Internet: Ein Wegweiser fur psychologisch interessierte User [Psychology in the Internet:
a pathfinder for psychologically interested users] (pp. 115-123). Weinheim: Beltz.
Reips, U.-D. (1999a). Theorie und Techniken des Web-Experimentierens [Theory and techniques
of Web experimenting]. In B. Batinic, L. Graf, A. Werner, & W. Bandilla (Eds.), Online
research. Gottingen: Hogrefe.
Reips, U.-D. (1999b). Online answering biases. Manuscript submitted for publication.
Salus, P. H. (1995). Casting the net: from ARPANET to Internet and beyond. Reading, MA: Addison-
Wesley.
Schmalhofer, F., Elst, L., Aschoff, R., Barenfanger, O., & Bourne, L. (in press). Mentale Modelle
sozialer Interaktionen: Wie Texte iiber Sozialbetriigereien verstanden werden [Mental models
of social interactions: How readers comprehend texts about cheating behaviors]. Zeitschrift filr
Experimentelle Psychologie.
Schmidt, W. (1997). World-Wide Web survey research: Benefits, potential problems, and solutions.
Behavior Research Methods, Instruments, & Computers, 29, 274~279.
Schmidt, W. Hoffman, R., & MacDonald, J. (1997). Operate your own World-Wide Web server.
Behavior Research Methods, Instruments, & Computers, 29, 189-193.
Smith, M., & Leigh, B. (1997). Virtual subjects: Using the Internet as an alternative source of
subjects and research environment. Behavior Research Methods, Instruments, and Computers, 29,
496-505.
Weigend, A. (1995). Music recognition experiments. [WWW document]. Available URL: h t t p : / /
www.es.Colorado.edu/~ andreas / Teaching/ Music / Experiinents.html
Welch, N. (1995). Demonstrations in auditory perception. [WWW document]. Available URLs:
h t t p : / / w w w . m u s i c . m c g i l l . c a / a u d i t o r y / A u d i t o r y . h t i n l and h t t p : / /
g l l 5 . b i o . tu-darmstadt .de / a u d i t o r y / auditory.html
Welch, N., & Krantz, J. (1996). The World-Wide Web as a medium for psychoacoustical
demonstrations and experiments: Experience and results. Behavior Research Methods, Instruments,
& Computers, 28, 192-196.