Theories, methods and current research on emotions in library and
information science, information retrieval and human–computer
interaction
Irene Lopatovska a
, Ioannis Arapakis b,⇑
a
Pratt SILS, 144 w. 14th street, 6th floor, New York, NY 10011-7301, United States
b
Department of Computing Science, University of Glasgow, 18 Lilybank Gardens, G12 8QQ Glasgow, United Kingdom
a r t i c l e i n f o
Article history:
Received 8 June 2009
Received in revised form 22 June 2010
Accepted 6 September 2010
Available online xxxx
Keywords:
Affect
Affective computing
Affective feedback
Emotion
Emotion research
Feelings
Methods for studying affect
Subjective variables
Theories of emotion
a b s t r a c t
Emotions are an integral component of all human activities, including human–computer
interactions. This article reviews literature on the theories of emotions, methods for studying
emotions, and their role in human information behaviour. It also examines current
research on emotions in library and information science, information retrieval and
human–computer interaction, and outlines some of the challenges and directions for future
work.
Ó 2010 Elsevier Ltd. All rights reserved.
1. Introduction
Emotions play an essential role in social interactions (Russell, Bachorowski, & Fernandez-Dols, 2003; Scherer, 2003; Sander
et al., 2005), perform important regulatory and utilitarian functions within human body and brain, and facilitate rational
decision making and perception (Damasio, 1994). Evidence from recent neurological studies underlines the importance of
emotions in human cognition and perception (Picard, 2001). Yet, the disciplines that study human–computer interaction
have only recently started to investigate this phenomenon and gain understanding of its causes and effects (Julien, McKechnie,
& Hart, 2005; Picard, 1997; Nahl & Bilal, 2007). Terms like ‘‘affective computing” (Picard, 1997) and ‘‘emotional design”
(Norman, 2004) refer to the integration of emotions in the design of computer systems in an attempt to make them more
natural for humans to understand and use (Picard, 2003). Some progress has been made in developing ‘‘affective systems”
that are capable of recognising and appropriately responding to human emotions, and ultimately making human–computer
interaction experiences more effective and pleasurable.
This article aims to aid current and future studies of emotions in human–computer interactions by reviewing the definitions
and theories of emotions, methods for studying emotions and, surveying current state of emotion research in library
and information science (LIS), information retrieval (IR) and human–computer interaction (HCI).
0306-4573/$ - see front matter Ó 2010 Elsevier Ltd. All rights reserved.
doi:10.1016/j.ipm.2010.09.001
⇑ Corresponding author. Tel.: +30 695 1000 266.
E-mail addresses: ilopatov@pratt.edu (I. Lopatovska), arapakis@dcs.gla.ac.uk (I. Arapakis).
Information Processing and Management xxx (2010) xxx–xxx
Contents lists available at ScienceDirect
Information Processing and Management
journal homepage: www.elsevier.com/locate/infoproman
Please cite this article in press as: Lopatovska, I., & Arapakis, I. Theories, methods and current research on emotions in library and information
science, information retrieval and human–computer interaction. Information Processing and Management (2010), doi:10.1016/
j.ipm.2010.09.001
After reviewing and carefully evaluating the theories and methods employed in the field of emotion research, we observed
varying strengths and weaknesses in all of them and settled on a neutral stance in our review. The field of emotion
research is young and rapidly developing. In our opinion, the reviewed methods have had a similar level of exposure and
validation by our peers. Our approach has been to present the existing methodologies to our readers in a way that will enable
them to evaluate these approaches against their specific research objectives.
2. Theories of emotions
Despite the long history of inquiry into the nature of emotion there is an apparent lack of consensus and uniformity within
the scientific community on what emotions are and how we can represent them. Kleinginna et al. (2005) collected more
than 90 definitions of emotions. Emotions have been defined as states of emotional feeling (Johnson-Laird et al., 1989), as
feeling states involving positive or negative affective valence (Ortony, Clore, & Collins, 1988), as states of automatic arousal
(Schachter & Singer, 1962), or changes in the activation of action dispositions (Frijda, 1986). Moreover, the indiscriminate
application of the term ‘‘emotion” has led to the vague differentiation between the terms ‘‘emotion”, ‘‘feeling”, ‘‘mood”, ‘‘attitude”,
and others.
There is no agreement about the nature of emotion and its relationship to the emotion stimuli. Theories of emotion can be
grouped into two main categories. The first category invokes cognition as a necessary element of emotion and tries to explain
the subjective manifestations of emotional experiences. The cognitive theories of emotion argue that the cognitive activity
can be conscious or unconscious, intentional or unintentional and take a form of a judgement or a thought. This activity is
also known as cognitive appraisal (Folkman, Lazarus, Gruen, & DeLongis, 1986) and refers to the evaluation of a particular
encounter with the environment, as well as the determination of its relevance to one’s well-being. The major proponent
of the cognitive theory of emotion was Lazarus (1984), who stressed the importance of cognitive evaluations in establishing
the meaning of stimuli and the way of coping with it. Another example of a cognitive approach is the work of Frijda (1994),
who defined emotion as reaction to affectively important event that consist of affect, awareness of an emotional object and
further appraisal of that object, action readiness and automatic arousal. The componential theory of emotion (Scherer, 2005)
is another example of the modern cognitive theory that treats emotion as a synchronisation of many different bodily, perceptual
and cognitive processes.
The second category of emotion theories emphasises somatic factors and seeks to describe emotional expressions and
perceptions of emotional expressions (Zajonc, 1984). Somatic theories argue that bodily responses, and not cognitive judgements,
cause emotional reactions. Major proponents of the somatic approach include Silvan Tomkins, Robert Plutchik and
Paul Ekman. Tomkins (1984) views affect system as the primary motivation system that can amplify other physical and bodily
functions (e.g., interference with breathing causes terror that leads to the struggle for air). Plutchik (1980) stresses the
evolutionary link of emotion with instinctive behaviour in animals. Ekman (1984), who shares a similar viewpoint, regards
emotions as psychosomatic states that have evolved over time due to their adaptive value in dealing with prototypical life
tasks. Ekman suggests that emotions’ primary function is to mobilise an organism to respond quickly to prototypical events,
similar to those that were encountered in the past.
Both categories of emotion theories are implicitly or explicitly used in the studies of affective aspects of information use.
For example, studies that require participants to explain their feelings assume existence of the evaluative component of
emotional reaction (Bilal & Bachir, 2007; Gwizdka & Lopatovska, 2009; Kuhlthau, 1991; Lopatovska & Mokros, 2008), while
studies that measure spontaneous bodily responses to emotional stimuli follow the rationale of somatic theories of emotion
(Arapakis, Konstas, & Jose, 2009; Arapakis et al., 2009; Mooney, Scully, Jones, & Smeaton, 2006; Ren, 2009; Soleymani, Chanel,
Kierkels, & Pun, 2008a, Soleymani, Chanel, Kierkels, & Pun, 2008b; Smeaton & Rothwell, 2009).
There is lack of consensus regarding the structure and manifestations of emotion. The two dominant views on emotions’
structure are the discrete and continuous approaches. Discrete emotion theorists, following Darwin’s work, suggest the existence
of six or more basic emotions (happiness, sadness, anger, fear, disgust, and surprise), which are universally displayed
and recognised (Darwin, 2005; Ekman, 1992; Ekman, 1999a, chap. 6). The arguments for the existence of basic emotions include
cross-cultural universals for facial expressions and antecedent events, and presence of these emotions in other primates.
Experiments in many countries, including countries isolated from media, show that people express and recognise basic
emotions the same way (Ekman & Friesen, 1975). There is no agreement on which emotions qualify as basic, but the list typically
includes fear, anger, disgust, happiness, sadness, and surprise (Ekman, 1992; Plutchik, 1980). Other emotions are seen
as combinations of these basic emotions or as socially learned variants of these emotions, e.g., grief, guilt and loneliness are
all variants of basic sadness, (Bower, 1992). In the context of HCI research, the theory of basic emotion implies that: (i) emotional
experiences can be measured on all stages of human interaction with systems, (ii) accurate translations of emotional
expressions and predictions based on these translations can be made, and (iii) systems can incorporate characters depicting
basic emotions that users can recognise accurately.
The continuous approach assumes the existence of two or more dimensions that describe and distinguish between different
emotions (Barrett et al., 1999; Russell, 1994; Russell & Mehrabian, 1977; Russell & Steiger, 1982). Support for the
dimensional emotion theories comes from physiological correlates of emotional stimuli, such as heart rate and skin conductance
levels. The first dimensional model was developed by Wundt (1904), who applied both introspective and experimental
methods to study subjective experiences. The model has been supported by other research, which revealed that people tend
2 I. Lopatovska, I. Arapakis / Information Processing and Management xxx (2010) xxx–xxx
Please cite this article in press as: Lopatovska, I., & Arapakis, I. Theories, methods and current research on emotions in library and information
science, information retrieval and human–computer interaction. Information Processing and Management (2010), doi:10.1016/
j.ipm.2010.09.001
to perceive all kind of meaning in terms of valence (positive vs. negative) and activation (active vs. passive) (Scherer, 2002).
Russell (1994) proposed the use of independent bipolar dimensions of pleasure–displeasure, arousal, and dominance–submissiveness,
rather than a small number of discrete emotion categories. In a dimensional taxonomy all emotion categories
vary quantitatively (Russell & Steiger, 1982) and are mapped within a bipolar dimensional space.
Examples of studies that rely on the continuous approach include the work of Chan et al. (2005), Hanjalic and Xu (2005),
and Soleymani et al. (2008a). Chan et al. (2005) propose an approach towards the annotation of the emotional dimension of
multimedia content using low-level feature analysis. Affective features extracted from multimedia audio content were analysed
in terms of arousal and valence, and they were combined in a two-dimensional frame to form the affective curve. Similarly,
Soleymani et al. (2008a) present an approach to affective ranking of movies based on the emotions experienced by the
viewers and affective characterisation of the content. Using a range of sensory data, in combination with audio- and videobased
analysis, the levels of arousal and valence were estimated for each scene and cross-checked with the viewers selfassessments.
Finally, Hanjalic and Xu (2005) suggest a framework of video content representation that follows the dimensional
approach. A two-dimensional emotion space of arousal and valence was used to map the affective video content, using
low-level feature analysis.
Examples of studies that follow the discrete approach are Arapakis, Jose, and Gray (2008), Arapakis et al. (2009),
Lopatovska and Cool (2008), Smeaton and Rothwell (2009), and Ren (2009). Arapakis et al. (2008) examined the effects of
search task difficulty on users emotions. The study employed questionnaire as well as affective data, derived from automatic
facial expression analysis. Both sources of evidence were presented using the discrete approach. In Lopatovska and Cool
(2008) facial expression analysis was used to study digital libraries search. The observed emotions were studied using the
discrete approach. Smeaton and Rothwell (2009) recorded people’s physiological reactions as they view films in a controlled,
cinema-like environment. The annotation and affective tagging of the films was performed using emotion categories, such as
fear, anger, shame, happiness, and other. Arapakis et al. (2009) applied automatic facial expression analysis to determine the
topical relevance of viewed videos, based on users affective responses. The accumulated data was interpreted in terms of the
seven basic emotion categories discussed earlier. Ren (2009) developed general-purpose agents that can recognise human
emotion and create machine emotion. The user emotions were observed using a range of sensory information (brain waves,
speech analysis, facial expressions, etc.). The affective information derived from the facial expressions was again analysed in
terms of seven basic emotion categories.
It is our belief that none of the reviewed theories stands out as significantly better or worse than the others. The theories
describe different aspects of emotion, and attempt to explain its different aspects. Since all theories have their strengths and
weaknesses and none have been recognized as dominant in their native field, we leave it up to the reader to pass the judgment
on their individual merits and applicability to the information research domain.
The next section will review some of the methods of emotion inquiry that were developed based on the theories described
above.
3. Methods of emotion research
Considering the countless definitions of emotion, one should not expect a single standard method of emotion measurement.
This section reviews methods that are currently used by the disciplines that study emotions.
Scherer (2005) suggests that due to the component nature of the phenomenon only the assessment of all components
involved can offer a comprehensive and accurate depiction of an emotion episode. This suggestion entails that in the ideal
study, all of the following emotion components should be measured: (i) changes in the appraisal processes (at all levels of
the nervous system), (ii) responses produced in the neuroendocrine, autonomic, and somatic nervous system, (iii) motivational
changes brought by the appraisal process, (iv) facial, vocal and bodily indications, and (v) nature of the subjectively
experienced emotional state that relates to the above component changes. Though such accurate measurement of emotion
has not been accomplished yet, significant progress has been made in measuring its individual components. We will review
neuro-physiological signal processing, observer and self-report methods that are used in the studies of emotional experiences
(Larsen & Fredrickson, 1999).
3.1. Neuro-physiological signal processing methods
One of the emotion components is physiological arousal, which corresponds to the physiological changes (e.g., respiratory
and cardiovascular accelerations and decelerations, muscle spasms, etc.) that often occur during emotion episodes. Neurophysiological
methods involve monitoring body responses to emotional stimuli. Researchers can infer the presence of emotion
by collecting brain activity images, pulse rate, blood pressure or skin conductance readings. The procedures for collecting
neuro-physiological measures vary between a simple sensor on a finger for monitoring pulse rate and skin conductance
to more invasive sensors, such as electrocardiograph (ECG), blood pressure monitoring and electroencephalogram (EEG).
Proponents of the neuro-physiological methods argue that, while this approach requires the use of physical sensors, the sensors
do not invade user’s privacy and can capture short-term changes not measurable by other means (Scheirer, Fernandez,
Klein, & Picard, 2002). Another benefit of the neuro-physiological measures is the detection of responses that cannot be captured
by any other sensory channels (Bamidis, Papadelis, Kourtidou-Papadeli, Pappas, & Vivas, 2004).
I. Lopatovska, I. Arapakis / Information Processing and Management xxx (2010) xxx–xxx 3
Please cite this article in press as: Lopatovska, I., & Arapakis, I. Theories, methods and current research on emotions in library and information
science, information retrieval and human–computer interaction. Information Processing and Management (2010), doi:10.1016/
j.ipm.2010.09.001
The method is criticised for limiting participants’ mobility and causing distraction of emotional reactions. Natural ageing
and unanticipated changes in physiological characteristics (due to accidents or surgery) can introduce noise in the measurement
of neuro-physiological signals (Chandra & Calderon, 2005). Additional limitations include the inability to map neurophysiological
data to specific emotions (e.g., frustration), difficulties in translating temporal micro-resolutions (milliseconds)
to temporal units relevant to emotional responses and reliance on non-transparent measurement instruments (e.g., sensors
that constrain movements) (Bamidis et al., 2004). In addition, such methods require special expertise and the use of special,
often expensive, equipment.
An example of a neuro-physiological instrument for detecting emotions is a LifeShirt sensor system that can be used for
monitoring cardiovascular, respiratory, metabolic and other physiological effects of physical or emotional stress (Wilhelm,
Pfaltz, & Grossman, 2006). The system collects comprehensive set of physiological measures, is wearable and relatively
unobtrusive. The system might help to identify specific emotion signatures and accurately infer affective states from physiological
signals (especially when the system is individually calibrated). Picard, Vyzas, and Healey (2001) used a combination
of neuro-physiological and other methods to classify a single subject’s emotional reactions to eight emotional stimuli over
the period of time based on facial muscle tension, blood volume pressure, skin conductance, and respiration data. Partala,
Surakka, and Vanhala (2006) detected positive and negative emotions by using electrodes to capture activity of two facial
muscles. Partala and Surakka (2003) investigated pupil size dilation during auditory emotional stimulation. The findings
indicated that positive and negative sounds caused participants’ pupils to dilate, while neutral sounds did not impact the
pupil size. Scheirer et al. (2002) investigated user’s physiological and behavioural changes associated with frustration and
found that blood volume pressure decreased, while skin conductivity and number of mouse clicks increased during frustrating
episodes. Mooney et al. (2006) used a range of peripheral physiological metrics (galvanic skin response, skin temperature,
and other) to examine the role of searchers’ emotional states in the process of data indexing during the search
process. The study provided evidence in favour of using physiological data processing for studying searchers’ emotions. More
specifically, it demonstrated that we can observe measurable features in response to events in movies and within computer
mediated tasks.
3.2. Observer methods
In addition to neuro-physiological measures, observer methods offer a way to study emotions through the observation of
facial, vocal and gesture cues to emotional stimuli.
Research indicates that emotions are primarily communicated through facial expressions rather than bodily gestures (Ekman
& Friesen, 1975). Facial cues (smiles, chuckles, smirks, frowns, etc.) are an essential aspect of social interaction (Russell
et al., 2003); help to clarify the focus of attention (Pantic et al., 2000a) and regulate human interactions with the surrounding
environment (Fig. 1). Most of the studies done in the area of emotion recognition from facial expressions have been largely
inspired by Darwin’s pioneering work (Darwin, 2005). More recent research on recognising emotions through facial expressions
was conducted by Paul Ekman. Ekman’s classification system of discrete emotion categories (Ekman, 1999a, chap. 6;
Ekman, 1999b, chap. 16; Rosenberg & Ekman, 1993), as well as his work on the Facial Action Coding System (FACS), have
Fig. 1. Sources of facial expressions (Fasel & Luettin, 2003).
4 I. Lopatovska, I. Arapakis / Information Processing and Management xxx (2010) xxx–xxx
Please cite this article in press as: Lopatovska, I., & Arapakis, I. Theories, methods and current research on emotions in library and information
science, information retrieval and human–computer interaction. Information Processing and Management (2010), doi:10.1016/
j.ipm.2010.09.001
provided a good foundation for automatic extraction and validation of emotional cues through the analysis of users’ facial
expressions.
Facial expressions are the result of facial muscle contractions, which induce movements of the facial skin and temporary
deformations of the facial features, such as eyebrows, nose, and mouth. FACS is based on recognising facial expressions of six
universally distinguished emotions: fear, surprise, sadness, happiness, anger, disgust, and their combinations. The intensity
of the emotion can be determined indirectly by the presence and degree of changes in all facial regions associated with it. For
example, sadness is usually expressed through the brow, eye and mouth areas. In sadness, the inner corners of brows are
drawn up, skin below the eyebrow is triangulated with the inner corner up, upper eyelid inner corner is raised, corners of
lips are down or the lip is trembling (Ekman, 2003).
The benefits of the FACS method include: (i) high reading accuracy rates, (ii) use of non-obtrusive and common laboratory
equipment, such as video camera, and (iii) high validity that is confirmed by correlations with physiological measures (e.g.,
increased heart rate that coincides with surprise and disgust). Another benefit of FACS is the fact that it can be programmed
into computer systems to automatically recognise user emotions (Cohn & Kanade, 2006; Picard, 1997).
Most of the automatic facial expression analysis systems directly interpret the observed facial expressions and classify
them into 6 basic emotion categories (Essa & Pentland, 1997; Hong, Neven, & von der Malsburg, 1998; Kimura & Yachida,
1997; Lanitis, Taylor, & Cootes, 1997; Wang, Iwai, & Yachida, 1998; Zhang, Lyons, Schuster, & Akamatsu, 1998). Some systems
describe facial expressions by noting facial muscles relative positions (vector space) without assigning labels (Black
& Yacoob, 1997; Pantic et al., 2000b). Since most of the automatic emotion analysers have been designed using static images
of faces without facial hair or glasses, taken under good illumination conditions during extreme emotional episodes, they
cannot draw accurate inferences on observed emotions that are expressed during longer episodes, which occur in more naturalistic
settings. Additional limitations include the inability to perform a context-dependent interpretation of the observed
facial expressions (Jaimes & Sebe, 2007). This suggests that, unless the emotion stimuli or current focus of attention is identified
(e.g., by the use of eye-tracking), it is difficult to interpret the context in which the facial behaviour occurs. Finally, Fasel
and Luettin (2003) argue that facial expression recognition should not be confused with human emotion recognition since
facial expressions can also be influenced by non-emotional mental and physiological activities. Furthermore, the face is one
of the many channels (e.g., voice, gesture, gaze) that reveals emotion. The authors emphasize that emotion recognition requires
interpretive analysis of the context and understanding of the situation in which emotion occurs. For a more detailed
discussion on the facial expression analysis methods see Fasel and Luettin (2003), Jaimes and Sebe (2007), and Pantic and
Rothkrantz (2003).
Several studies that investigated the role of emotions in online search behaviour used facial expression analysis method
for obtaining emotion readings. In Arapakis et al. (2009) the authors applied real-time facial expression analysis to inform
the design of the video recommender system and suggest meaningful recommendations of unseen videos. While investigating
relationships between search task difficulty and emotions, Arapakis et al. (2009) used facial expression analysis, among
other sensory channels, to aggregate information on user affective behaviour and develop models capable of determining
topical relevance of documents and videos, without the aid of explicit judgments. Automatic facial expression analysis
was also applied in a study of Google search (Lopatovska, 2009a). The author found that during the search, surprise was
the most frequently expressed emotion, followed by neutral, sad, fear and happy. The study also found that specific search
behaviours, such as left mouse clicks or wheel scrolls up, were associated with unique patterns of emotional expressions preceding
and following these behaviours. Facial expression analysis was used in a study of digital libraries’ search Lopatovska
and Cool (2008). The authors found that during the search most of the positive emotions corresponded with the time when
an assistant entered the room. The study also found a wide variation in individual levels of emotional expressivity (e.g., one
subject’s face expressed 57 intense emotions, while other subject’s face expressed only 9 emotions during the same period of
search time).
Observer methods for studying emotions also include analysis of verbal communication. The empirical investigation of
the effect of emotions on voice begun in the early 20th century (Mendoza & Carballo, 1999). Since then, scientific interest
in the vocal attributes of moods, affective and cognitive states has increased (Scherer, 2003). In recent years, research on vocal
expression patterns of the naturally occurring emotions has produced a substantial number of theories and models of
speech communication. Brunswik’s functional lens model of perception is an example of one such model (Mitroff, 1974).
The model suggests that vocal communication of emotion is initiated with an encoding, or expression, of an emotional state
by certain voice and speech characteristics, which are susceptible to objective signal measurement. The assumption made
here is that that there are some acoustic correlates of emotion in the acoustic parameters (e.g., respiration, phonation) that
can provide insightful cues about the speaker’s affect state.
According to Pantic and Rothkrantz (2003), the auditory features that are most often extracted from the speech signal are:
(i) pitch, (ii) intensity, (iii) speech rate, (iv) pitch contour, and (v) phonetic features. Pitch corresponds to the rate at which
vocal cords vibrate and determines the frequency of the acoustic signal, while intensity refers to vocal energy. Variations in
voice pitch and intensity usually have a linguistic function, such as over-stressing or under-stressing certain words (Cowie
et al., 2001). When, for example, a person is experiencing anger, fear or joy, the sympathetic nervous system becomes
aroused, resulting in a heart rate and blood pressure increase that produces mouth dryness and occasional muscle tremors.
Speech is then characterised by loudness, increased speech rate and strong, high frequency energy (Breazeal, 2001). Speech
rate represents the number of spoken words within a time interval. Finally, pitch contour corresponds to pitch variations
described in terms of geometric patterns, and phonetic features of all types of sounds involved in a speech (e.g., vowels,
I. Lopatovska, I. Arapakis / Information Processing and Management xxx (2010) xxx–xxx 5
Please cite this article in press as: Lopatovska, I., & Arapakis, I. Theories, methods and current research on emotions in library and information
science, information retrieval and human–computer interaction. Information Processing and Management (2010), doi:10.1016/
j.ipm.2010.09.001
consonants and their pronunciation). Table 1 illustrates correlations between emotion and acoustic parameters from the
Murray et al. (1993) review.
Systems that analyse speech, along with other representations of affect, are described in Corradini, Mehta, Bernsen, and
Martin (2003), Go, Kwak, Lee, and Chun (2003), Schapira and Sharma (2001), Schuller, Lang, and Rigoll (2002), Song, Bu,
Chen, and Li (2004), Sebe, Bakker, Cohen, Gevers, and Huang (2005), Schuller, Arsic, Wallhoff, and Rigoll (2006), Yoshitomi,
Kim, Kawano, and Kilazoe (2000), and Zeng et al. (2006). These systems achieved an accuracy rate of 72–85% when detecting
one or more basic emotions from noise-free audiovisual input (Jaimes & Sebe, 2007). These accuracy rates outperform the
equivalent human emotion recognition skill that achieves an accuracy rate of 55–70% in neutral content speech (Pantic &
Rothkrantz, 2003).
Despite the high accuracy rates in detecting emotions from the vocal stream, the method is not extensively used due to a
number of limitations. Jaimes and Sebe (2007) list four limitations of the existing vocal affect analysers: (i) singular classification
of input audio signals into a few discrete emotion categories, (ii) context-independent analysis of the input audio
signal, (iii) analysis of vocal expression information only on short time scales (thus inferences about moods and attitudes
are almost impossible to obtain), and (iv) assumptions about the quality of the test data (noise-free recordings, short sentences
with intermediary pauses, clear speech, etc.).
Examples of the studies that used audio data to infer presence and quality of emotions can be found in Chan et al. (2005),
and Hanjalic and Xu (2005). In Chan et al. (2005), a set of affective features was extracted from multimedia audio content and
was annotated using a set of labels with predetermined affective semantics. The audio features that consisted of speech, music,
special effects and silence, were analysed in terms of the affective dimensions of arousal and valence. Similarly, in Hanjalic
and Xu (2005) the authors modelled video content using a selection of low level audio (signal energy, speech rate,
inflection, rhythm duration, voice quality) and visual features (motion). The framework used in the study was based on
the dimensional approach to emotion, where video content represented a set of points in a two-dimensional affect space
(arousal and valence) and reliably depicted expected transitions of viewers emotional perception during the video.
Whether body movements or gestures are indicative of specific emotions is a subject under debate. Some studies suggest
that the latter are indicative of the intensity of emotion, but not its type. Other studies (Boone & Cunningham, 1998; de Meijer,
2005; Wallbott, 1998) provide evidence that associate certain body movements with specific emotions. This approach
follows Darwin’s view of human beings’ genetic predisposition to exhibit certain patterns of bodily movements during
the expression of affect states (Darwin, 2005). Body movements, and specifically hand gestures (Chen, Fu, & Huang, 2003;
Castellano, Villalba, & Camurri, 2007; Caridakis et al., 2007), have recently attracted the attention of the HCI community
(Ambady & Rosenthal, 1992). Gunes and Piccardi (2007) fused facial expressions and body gestures for bimodal emotion recognition.
In their study they provided a list of expressive gestures and their correlations to the emotion categories. A list of
emotions recognised by the changes that occur in the body are presented in Table 2.
Most of the recent work in hand gesture recognition can be grouped into: (i) glove-based, and (ii) vision-based (Chen
et al., 2003). Glove-based gesture recognition requires the user to wear an unobtrusive hand glove device, which communicates
gestures to a computer system through a set of wires. This approach is based on the 3-D spatial description of hands.
Table 1
Emotions and Speech Parameters (Murray et al., 1993).
Anger Happiness Sadness Fear Disgust
Rate Slightly faster Faster or slower Slightly slower Much faster Very much faster
Pitch Average Very much higher Much higher Slightly lower Very much higher Very much lower
Pitch Range Much wider Much wider Slightly narrower Much wider Slightly wider
Intensity Higher Higher Lower Normal Lower
Voice Quality Breathy, chest Breathy, blaring tone Resonant Irregular voicing Grumble chest tone
Pitch Changes Abrupt on stressed Smooth, upward inflections Downward inflections Normal Wide, downward terminal inflects
Articulation Tense Normal Slurring Precise Normal
Table 2
List of bodily emotions and the accompanying changes that occur on the
body when they are displayed (Gunes & Piccardi, 2007).
Anxiety Hands close to the table surface; fingers moving;
fingers tapping on the table
Anger Body extended; hands on the waist; hands made
into fists and kept low, close to the table surface
Disgust Body backing; left/right hand touching the neck
or face
Fear Body contracted; body backing; hands high up,
trying to cover bodily parts
Happiness Body extended; hands kept high; hands made
into fists and kept high
Uncertainty Shoulder shrug; palms up
6 I. Lopatovska, I. Arapakis / Information Processing and Management xxx (2010) xxx–xxx
Please cite this article in press as: Lopatovska, I., & Arapakis, I. Theories, methods and current research on emotions in library and information
science, information retrieval and human–computer interaction. Information Processing and Management (2010), doi:10.1016/
j.ipm.2010.09.001
Vision-based gesture recognition method relies on appearance of hands in images and applies appearance-based techniques,
while glove-based recognition applies model-based techniques for gesture analysis. McAllister, McKenna, and Ricketts
(2002b) outlined the major difficulties in using gesture recognition techniques, which are related to the following factors:
(i) the hand’s jointed physical structure often results in self-occlusion, which makes it harder to model, (ii) many gesture
recognition applications cannot track hands under poorly controlled and varying lighting conditions, (iii) tracking both
hands at the same time demands a solution to the temporal matching (data association) problem, and a method for dealing
with the temporary occlusion of one hand by the other. Finally, gesture recognition can be hindered by clothing or other
objects. Examples of the studies that used gesture recognition include Brewster, Lumsden, Bell, Hall, and Tasker (2003), Chen
et al. (2003), Camurri, Lagerlöf, and Volpe (2003), Gunes and Piccardi (2007), Isbister, Höök, Sharp, and Laaksolahti (2006),
Kapoor, Picard, and Ivanov (2004), Kaliouby and Robinson (2004), Kirishima, Sato, and Chihara (2005), Licsár and Szirányi
(2005), McAllister, McKenna, and Ricketts (2002a), Morency and Darrell (2006), Ng et al. (2002), Morency, Sidner, Lee,
and Darrell (2007), Tsalakanidou, Malassiotis, and Strintzis (2007).
HCI and IR research offers a unique observer method of inferring emotions from interactive behaviours captured in computer
log files. Kapoor, Burleson, and Picard (2007) used log data to predict frustration. Fox, Karnawat, Mydland, Dumais, and
White (2005) found a correlation between certain search behaviours, such as the time spent on the search result page and
number of result pages visited, with searchers satisfaction and dissatisfaction. Lopatovska (2009a) correlated various search
behaviours with emotion and mood data and found association between certain types of clicks, number of search activities,
search session duration, and happy expressions. Use of a log file data for inferring emotional states can be useful in designing
systems that are capable of recognising, and possible, responding to emotions. However, we feel that this research area is still
in its infancy and more work is needed to verify prior findings and find new correlations between information behaviours
and emotions.
3.3. Self-report methods
While physiological response patterns and expressive behaviour can be observed and used to infer the affective state of a
person, self-report methods rely on simply asking participants to describe the nature of their experience. The self-report
methods rely on the assumption that individuals are able and willing to recognise and report their emotions. The reliability
and validity of the measures are evident from the high correlations of self-reports with the quality of the physical stimuli and
neurological activities of the brain (Kahneman, 2000). Momentary reports are considered the most accurate; however, the
accuracy of retrospective reports can also be improved by the use of special techniques. While they may be subject to participant’s
bias, self-report methods are efficient and easy techniques for obtaining emotion data. We will review several selfreport
methods in more detail.
The two major self-report methods are based on the discrete and dimensional approaches described in the previous
section. The discrete approach relies on the semantics-based categories that correspond to unique emotion patterns. In
most cases, a list of emotion terms is provided to a respondent who must determine which term better describes his/
her emotional experience, rate the intensity of emotion and, finally, state how long that emotion has been experienced.
While ensuring efficiency and standardisation of data collection, discrete emotion self-reports have several disadvantages,
including: (i) the possibility that one or several response alternatives may bias the respondent to choose them, (ii) the
situation when a respondent wishes to refer to a category that is not provided on the list, or (iii) the situation when a
respondent may be unfamiliar with the labels chosen by a researcher (Scherer, 2005). Russell and Steiger (1982) argue
that, when using natural language, people classify emotions by means of a taxonomy, but cannot explicitly describe it.
Thus, the taxonomy is implicit in the same sense that the syntactic rules of language are implicit. Lastly, while the information
obtained from this approach appears to be intuitive and easily interpretable, there are issues of comparability of
results between studies that employed different lists of emotion labels. Examples of the studies that rely on the discrete
emotion include Klein, Moon, and Picard (1999), and Scheirer et al. (2002) who investigated the effects and manifestations
of frustration, a discrete emotion.
The use of dimensional approach for describing emotional states, was established by Wilhelm Wundt (Scherer, 2005;
Sander, Grandjean, & Scherer, 2005) who suggested the use of a three-dimensional space formed by the valence (positive–
negative), arousal (calm–excited), and tension (tense–relaxed) dimensions to describe emotion. Given this approach, a
respondent can report his/her subjective experience by simply indicating emotion’s coordinates in the three-dimensional
space. Due to the difficulty of consistently identifying a third dimension from arousal or excitation, researchers often apply
only two of the three dimensions, thus forming a two-dimensional surface (arousal–valence space). This approach is quite
straightforward, simple and provides interval data that can be readily used in statistical processing (Russell & Steiger,
1982). However, the results lack the intuitiveness of the discrete emotions approach and are limited to degrees of positive
or negative valence or arousal (Fig. 2). Another shortcoming of the dimensional approach is ambiguity. For example, it is often
not clear whether a valence judgement is indicative of the appraisal of the emotional stimulus or a feeling induced by it. Most
importantly it is very difficult, if not impossible, to distinguish the intensity of an emotion from bodily excitation. As a result,
extremely intense happiness may be characterised by high arousal, while intense sadness may be accompanied by very low
arousal. Peter and Herbon (2006) advocate the use of dimensional approach in HCI research and investigate emotions in at
least 2-dimensional spaces: arousal and valence. The authors suggest adopting this view for the use in HCI, since it allows
an automated classification of different emotional states within arousal–valence space without labelling them. Example of
I. Lopatovska, I. Arapakis / Information Processing and Management xxx (2010) xxx–xxx 7
Please cite this article in press as: Lopatovska, I., & Arapakis, I. Theories, methods and current research on emotions in library and information
science, information retrieval and human–computer interaction. Information Processing and Management (2010), doi:10.1016/
j.ipm.2010.09.001
the study that used dimensional approach can be found in Partala and Surakka (2004), who investigated effects of pre-programmed
positive and negative interventions in human–computer interaction.
To mitigate disadvantages of self-report methods, researchers often choose to use a free-response report, which allows
participants to express experienced emotions using words or expressions that best represent their experiences. This technique
provides a high level of specificity, which can be useful in studies where accuracy and explicitness are considered
important. However, according to Scherer (2005) it is difficult to analyse free-response data in a quantitative, statistical manner.
Such data collection techniques, including journals, think-aloud protocols and interviews are popular in LIS research.
One of the earliest studies that identified searchers’ emotional states during the information search process was Kuhlthau
(1991) research on students’ information seeking behaviour. The emotion data was collected by asking participants to record
their feelings and thoughts related to information seeking in a journal. Analysis of the data collected through journals and
questionnaires led to development of the six stage information seeking model that identified relationships between the
search process, participants’ feelings, thoughts, and actions. Meghabghab (1995) observed inexperienced school librarians
learning to search online databases. Subjects were asked to document their actions, feelings and thoughts on log sheets
and work activity sheets. In the study that examined developmental steps of acquiring expertise with the search engine
(Nahl, 1998), participants were asked to keep logs of their cumulative searches and provide weekly self-ratings on satisfaction
scales. James and Nahl (1996) examined the semester-long affective development of senior college students who were
learning to use the internet. The students were asked to record their cognitive and affective information processing activities
in ‘‘self-witnessing” reports. Affective states experienced by children and graduate students’ during online search were compared
in a study by Bilal and Kirby (2002). Self-report data was collected through journals (for students) and interviews (for
children).
Think-aloud methods were used in several LIS studies that investigated the role of affective variables in search behaviour.
Nahl and Tenopir (1996) studied searching behaviour of novice database users by recording their think-aloud reports,
including interactions with the study monitor, and using screen logging software to record their search activities. Wang
and Soergel (1998) examined document selection criteria, including evaluations of document’s emotional values. Participants,
25 self-selected faculty members, were asked to think aloud while selecting documents. Tenopir, Wang, Zhang, Simmons,
and Pollard (2008) observed how academic users interact with the ScienceDirect system by collecting think-aloud
protocols capturing participants’ affective and cognitive verbalisations.
Another popular technique of studying affect in LIS is interview. In most of the reviewed above studies, interviews with
participants were conducted before and after participants’ engagement in a search activity. While most of the studies conFig.
2. Alternative dimensional structures for the semantic space for emotions (Scherer, 2005).
8 I. Lopatovska, I. Arapakis / Information Processing and Management xxx (2010) xxx–xxx
Please cite this article in press as: Lopatovska, I., & Arapakis, I. Theories, methods and current research on emotions in library and information
science, information retrieval and human–computer interaction. Information Processing and Management (2010), doi:10.1016/
j.ipm.2010.09.001
ducted one-on-one interviews, a few studies used group interviews to collect data on users’ emotional experiences. In the
longitudinal study of uncertainty involved in information seeking, Wilson, Ford, Ellis, Foster, and Spink (2002) administered
pre- and post-search interviews. Bilal and Bachir (2007) conducted individual pre-search interviews to generate children’s
profiles, including their demographic information, prior experience, reading habits and preferences. Participants’ affective
states were captured in the exit one-on-one interviews in several studies of children’s use of a search engine (Bilal, 2000;
Bilal, 2002; Bilal & Kirby, 2002). Julien (2007) examined library customers’ experiences with internet public stations using
interview data. In a study of anxiety and perception of research, Kracker (2002) used the critical incident technique that required
students to recall specific research assignments and describe their feeling and thoughts associated with the research
process.
Other self-report methods for investigating affective variables include questionnaires. Pre- and post-search questionnaires
about users’ affective states are frequently used in LIS research in conjunction with other methods in the above-mentioned
research of students’ information seeking behaviour (Kuhlthau, 1991), participants completed questionnaires about
their perception of the six areas of library use in addition to keeping journals of their information seeking experience. Mentis
(2007) examined memories of frustrated search experiences by administering open-ended online questionnaire were participants
were free to define and describe their frustrated experiences.
Standardised tests for measuring affect are also frequently used in LIS research. Lopatovska (2009b) reported using Positive
Affect and Negative Affect Scale (PANAS) to measure searchers’ affect between search tasks. The PANAS (Watson, Clark,
& Tellegen, 1988) comprises of two 10-item scales that measure positive affect (extent to which a person feels enthusiastic,
active, alert, etc.) and negative affect (extent to which a person experiences subjective distress, including anger, contempt,
disgust, guilt, fear, nervousness, etc.). The study showed that affect did not change significantly during the course of the
search and is not significantly influenced by the search process. A study that investigated subjective variables of the information
search process found that better mood before the search and during the search correlates with better mood after the
search, but also correlates with the worse search outcomes and lower satisfaction, suggesting that, perhaps, it pays off to feel
some ‘‘pain” during the search in order to ‘‘gain” quality outcomes (Gwizdka & Lopatovska, 2009).
In a comprehensive study of web use Wang, Hawk, and Tenopir (2000) asked 24 graduate students to fill out a pre-search
questionnaire identifying their web experience, the State Trait Anxiety Inventory (STAI, forms Y1 and Y2) to measure affective
states and the Embedded Figure Test to measure cognitive styles. STAI consists of two forms: S-anxiety, which measures
individual general tendency of feelings, and T-anxiety, which measures individual’s current feelings. High scores on the tests
indicate high levels of anxiety, the scores range from a minimum of 20 to a maximum of 80. Kracker (2002) researched student
anxiety and perceptions of research using State Trait Anxiety Inventory test (STAI Y-1) and the critical incident technique
that required students to recall specific research assignment and describe their feelings and thoughts associated
with the process. Form STAI Y-1 was used to clarify statements given by participants about their most memorable or the
most recent research assignments.
In a study that examined relationships between search performance, anxiety levels and research achievement
(Onwuegbuzie et al., 2004) student participants were asked to fill out several questionnaires prior to engaging in the search
task. The questionnaires were designed to collect data about participants’ emotional states, and included Library Anxiety
Scale, Hope Scale, Procrastination Assessment Scale, Multidimensional Perfectionist Scale and others. In the study of effects
of emotion control on the web search behaviour and performance (Kim, 2006), participants were asked to take a ProblemSolving
Inventory test prior to engaging in the search tasks.
In a study of affect measurement tools for the design of haptic interfaces, Swindells, MacLean, Booth, and Meitner (2006)
presented the results from two experiments, where self-reports and physiological measurement techniques were compared
and contrasted. Self-reports produced better results than the biometric measures. However, the authors argue that this could
be attributed to a very subtle changes in affect experienced by users during the studies.
This section outlined various methods of emotion research. It also provided examples of the studies that used described
techniques to investigate the role of emotion in various information use contexts. All the reviewed methods have advantages
and disadvantages (see Table 3), and should be chosen based on the study objectives. For example, studies that are mainly
interested in the users’ perceptions and explanations of the felt emotions should consider self-report methods; studies that
are interested in affect-based systems that are capable of capturing human emotions should consider neurophysiological or
observer methods that can be detected automatically. A researcher should always consider using several methods in order
to: (i) ensure the accuracy and consistency of the collected data, (ii) increase reliability of findings, and (iii) create a comprehensive
representation of the users’ affective states.
4. Information systems and emotions
A lot of what we know about emotions in HCI is attributed to the research done under the umbrella of ‘‘affective computing”.
Rosalind Picard (1997, p. 3) defined affective computing as ‘‘computing that relates to, arises from, or deliberately influences
emotions”. The affective computing agenda includes giving a computer the ability to recognise and intelligently
respond to humans emotions (Hudlicka, 2003; Picard & Klein, 2002; Picard, 2003).
A number of HCI articles describe efforts to design ‘‘affective computers”, or systems capable of recognising and responding
to users’ feelings. Klein, Moon, and Picard (2002), and Scheirer et al. (2002) proposed prototypes of interactive computer
I. Lopatovska, I. Arapakis / Information Processing and Management xxx (2010) xxx–xxx 9
Please cite this article in press as: Lopatovska, I., & Arapakis, I. Theories, methods and current research on emotions in library and information
science, information retrieval and human–computer interaction. Information Processing and Management (2010), doi:10.1016/
j.ipm.2010.09.001
systems that can detect and respond to certain human emotional expressions. The design of such interactive systems is
based on the assumptions of the Media Equation theory (Reeves & Nass (1996)), as cited in Klein et al. (2002). The theory
suggests that people exhibit a propensity for interacting with machines as if they were other people, and respond to
computer’s praise and criticism in the same manner they respond to praise and criticism of other people.
Not everyone agrees with the idea of giving computers the power to respond to human emotions. Several studies outline
the benefits of giving users means of changing computer interfaces in order to improve emotions. Blom and Monk (2003)
suggests that the reasons users personalise their computers include emotional aspects, such as improved feelings of control,
ownership, fun, and relieve from boredom. Tractinsky (2004) reviewed the studies that showed the link between personaTable
3
Methods of emotion Research.
Method Modality Advantages Disadvantages Examples of studies
Neuro-physiological Body responses:
brain activity,
pulse rate, blood
pressure, skin
conductance, etc.
Can detect short-term changes
not measurable by other
means; neuro-physiological
changes cannot be easily faked
Reliance on non-transparent,
invasive sensors; can reduce
people’s mobility, causing
distraction of emotional reactions;
prone to noise due to
unanticipated changes in
physiological characteristics;
inability to map data to specific
emotions; require expertise and
the use of special, often expensive,
equipment
Picard et al. (2001), Scheirer et al.
(2002), Partala and Surakka
(2003), Bamidis et al. (2004),
Chandra and Calderon (2005),
Mooney et al. (2006), Partala et al.
(2006), and Wilhelm et al. (2006)
Observer Facial
expressions;
speech; gestures
Use of unobtrusive techniques
for measuring emotion; crosscultural
universals
Cannot perform contextdependent
interpretation of
sensory data; highly dependent on
environmental conditions
(illumination, noise, etc.); some
responses can be faked;
recognises the presence of
emotional expressions, not
necessarily emotions
Black and Yacoob (1997), Essa and
Pentland (1997), Kimura and
Yachida (1997), Boone and
Cunningham (1998), Hong et al.
(1998), Wallbott (1998), Wang
et al. (1998), Zhang et al. (1998),
Pantic et al. (2000b), Yoshitomi
et al. (2000), Breazeal (2001),
Cowie et al. (2001), Schapira and
Sharma (2001), McAllister et al.
(2002a), Schuller et al. (2002), Ng
et al. (2002), Brewster et al.
(2003), Chen et al. (2003),
Corradini et al. (2003), Go et al.
(2003), Kapoor et al. (2004),
Kaliouby and Robinson (2004),
Song et al. (2004), Chan et al.
(2005), Hanjalic and Xu (2005),
Kirishima et al. (2005), Licsár and
Szirányi (2005), de Meijer (2005),
Sebe et al. (2005), Cohn and
Kanade (2006), Isbister et al.
(2006), Morency and Darrell
(2006), Schuller et al. (2006), Zeng
et al. (2006), Castellano et al.
(2007), Caridakis et al. (2007),
Gunes and Piccardi (2007),
Morency et al. (2007),
Tsalakanidou et al. (2007),
Lopatovska and Cool (2008),
Arapakis et al. (2009), Lopatovska
(2009a), and Arapakis et al. (2009)
Self-report Diary; interview;
questionnaire
High correlation to neurophysiological
evidence;
unobtrusive; straightforward
and simple – do not require
the use of special equipment
Rely on the assumption that
people are aware of and willing to
report their emotions; subject to
the respondent’s bias; results of
different studies might not be
directly comparable
Watson et al. (1988), Kuhlthau
(1991), Meghabghab (1995),
James and Nahl (1996), Nahl and
Tenopir (1996), Nahl (1998),
Tenopir et al. (2008), Wang and
Soergel (1998), Bilal (2000), Wang
et al. (2000), Bilal (2002), Bilal and
Kirby (2002), Kracker (2002),
Wilson et al. (2002), Onwuegbuzie
et al. (2004), Kim (2006), Bilal and
Bachir (2007), Julien (2007),
Mentis (2007), Gwizdka and
Lopatovska (2009), and
Lopatovska (2009b)
10 I. Lopatovska, I. Arapakis / Information Processing and Management xxx (2010) xxx–xxx
Please cite this article in press as: Lopatovska, I., & Arapakis, I. Theories, methods and current research on emotions in library and information
science, information retrieval and human–computer interaction. Information Processing and Management (2010), doi:10.1016/
j.ipm.2010.09.001
lised aesthetic interfaces and improved perceptions of usability and satisfaction. However, the link between personalisable
interfaces and improved emotions can be questioned in light of findings reported by Ward and Marsden (2004). The authors
did not find statistically significant differences between user’s physiological reactions on well or poorly designed websites,
which might suggest that aesthetic differences between systems might not sufficiently impact user emotions.
Klein et al. (2002) described an experiment that simulated a computer game in which users were intentionally frustrated.
The experiment showed that participants who received supportive messages from the system were able to better recover
from negative emotional states and chose to stay in the game longer. The authors suggest that while it is not always possible
to build systems that do not cause frustration, it is possible to design computers that try to mitigate the effects of frustration.
Role of frustration in HCI was also investigated by Scheirer et al. (2002) who frustrated users by occasionally ‘‘freezing”
mouse movements. The study found correlation between blood volume pressure, skin conductivity, number of mouse clicks
and frustrating episodes. The findings suggest practical ways of designing systems capable of recognising user’s affective
states.
Klein et al. (2002), and Scheirer et al. (2002) research is discussed in several follow up articles. Lindgaard (2004) suggests
that addressing immediate negative emotions might be detrimental to achieving long-term goals. The author outlined the
difficulties in determining causal relationships from behavioural observations (participants who received sympathetic messages
from a system in Klein et al. (2002) study might have chosen to work with a system longer out of curiosity, and not
because of a sympathetic message).
Muller (2004) also critiques Klein et al. (2002), and Scheirer et al. (2002) studies for the misuse of hypothesis-testing
methods. In Scheirer et al. (2002), the authors suggest that the change of physiological measures was attributed to changing
levels of frustration, while in fact it can only point to the changing levels of arousal. The author suggests expanding affective
computing research to include ethnographic research methods and draws parallels between HCI and human emotional
attachments to other social objects (e.g., cars and boats). Oatley (2004) suggests moving from system’s acknowledgement
of user frustration to system’s attempt to repair frustration–causing problems. One of the hypothetical ways of doing that
would be soliciting user’s feedback, improving systems and giving improved products back to users for free.
Several studies have tested systems that communicate affect by incorporating emotionally expressive computer agents
(e.g., emoticons and avatars). Tzeng (2004) examined effects of apologies for the failed computer game expressed through
text and emoticons. Apologetic feedback and emoticons were found to enhance the aesthetics of game interaction and shorten
the psychological distance between the game and the player. Apologetic (sad) emoticons were found to communicate
emotions more effectively than pure text.
Brave, Nass, and Hutchinson (2005) examined effects of emotionally expressive computer agents on user’s perception of
computer game. The study used the game of black jack where dealers were represented by a human photograph and a blob
text. The nature of the photograph and a text represented neutral, sad and happy emotions directed towards participants’
wins or looses. Empathetic agents were perceived as likeable, carrying, trustworthy, supportive, and overall had positive effect
on participants. The authors suggest modelling affective computers after people in service roles who are trained to express
happiness and empathy regardless of their actual feelings.
Kleinsmith, De Silva, and Bianchi-Berthouze (2006) examined cross-cultural differences in recognising affect from computer
animated avatars. Avatars expressing fear, anger, happiness and sadness were designed based on the body postures of
predominantly Japanese and a few Sri Lanka and Caucasian American models. Participants recruited from the same ethnic
groups were asked to recognise emotional expressions from the avatar postures. Three culture groups indicated medium level
of agreement in judging emotions from avatar postures. Significant differences were found between groups’ judgements
of emotional intensity. The findings point to the challenges in designing universally recognised computer representations of
emotions.
A number of studies focused on the use of emotion data for improving information systems. In Arapakis et al. (2009), the
authors presented a novel video search environment that applies real-time facial expression analysis to aggregate information
on users’ affective behaviour. The collected information was used to classify the topical relevance of the perused videos
and, additionally, enrich the user profiles. The benefits of such system include combination of different modalities (facial
expression data, interaction data, etc.), integration of affective features into the employed profiling techniques, and, finally,
facilitation of meaningful recommendations of unseen videos.
Mooney et al. (2006) examined the role of searchers’ emotional states in an attempt to improve data indexing for and
within the search process. Users physiological responses to emotional stimuli were recorded using a range of biometric measurements
(GSR, skin temperature, etc.). The study provides initial evidence that users exhibit measurable biometric behaviour
when watching movies and engaging in interactive tasks. It also examines how this data can be used for affective
indexing of data within the search process.
In Soleymani et al. (2008a) the authors proposed an approach to affective ranking of movie scenes, which is based on
viewers’ affective responses and content-based features. The study found that important events that characterise every scene
are correlated with the viewers’ self-assessed arousal and valence. Furthermore, the provided evidence suggests that peripheral
physiological signals can be used to characterise and rank video content.
A number of IR studies are focusing on emotions as descriptors of information and are forming a new area of emotional
information retrieval research. Certain information objects, such as music and images, induce emotions that help to describe
and ultimately retrieve these objects. Lee and Neal (2007) developed a system for tagging emotions and their intensity induced
by music. Schmidt and Stock (2009) used similar method for tagging images using basic emotions’ tags: anger, disgust,
I. Lopatovska, I. Arapakis / Information Processing and Management xxx (2010) xxx–xxx 11
Please cite this article in press as: Lopatovska, I., & Arapakis, I. Theories, methods and current research on emotions in library and information
science, information retrieval and human–computer interaction. Information Processing and Management (2010), doi:10.1016/
j.ipm.2010.09.001
fear, happiness and sadness. The study findings suggest that collective emotion tagging can provide a new way to describe
and retrieve information.
Reviews of additional studies related to the design of affective computers can be found in Brave and Nass (2003), Hudlicka
(2003), Picard (2003), Pantic, Sebe, Cohn, and Huang (2005), and Eckhardt et al. (2009).
4.1. Emotions in online searching
This section describes several studies where emotions were examined in the context of online searching. The included
studies landed themselves into the two categories: (i) studies that investigated causes of various emotions experienced during
the search, and (ii) studies that investigated effects of emotions on search behaviours.
A number of studies that focused on exploring the causes of certain emotions experienced during the online search found
that positive emotions were usually associated with satisfactory search results (Tenopir et al., 2008), successful search completion
(Bilal & Kirby, 2002), use of online systems (Bilal, 2000; Bilal & Bachir, 2007), easier tasks (Arapakis et al., 2008), interest
in the process and documents (Kracker, 2002; Kracker & Wang, 2002; Lopatovska & Mokros, 2008), or documents’
stylistic features (Lopatovska & Mokros, 2008). Negative emotions were associated with frustrating aspects of systems,
uncertain search tasks and confusing search strategies (Tenopir et al., 2008), software failures (Bilal, 2000), uncertainty prior
to the search (Bilal & Bachir, 2007), difficulties in finding the answer and inadequate knowledge of a system (Bilal & Kirby,
2002; Meghabghab, 1995). We will focus on a few studies that investigated the causes of positive and negative emotions in
more detail.
While examining relationships between affective and cognitive variables during the online search, Tenopir et al. (2008)
showed that positive feelings were reported more frequently than negative feelings, and were associated with the thoughts
about search results. Negative feelings co-occurred more often with the thoughts related to system, search strategy and task.
Similar findings were reported in the study of children’s use of a search engine (Bilal, 2000). The study found that young participants
identified more positive than negative feelings during the search. Positive feelings were associated with the use of a
system in general, and ease of use, keyword search option, availability of graphics and fun in particular. Negative feelings of
confusion and frustration were associated with software failures. However, negative feelings did not have significant impact
on children’s persistence and patience in web searching.
An investigation into the role of emotions in the information seeking process (Arapakis et al., 2008) has provided evidence
of the effect of the search task on users’ emotions. More specifically, the findings indicated a progressive transition from positive
to negative valence as the degree of task difficulty increases.
In the study of children’s interaction with the International Children’s Digital Library, Bilal and Bachir (2007) discovered
that positive feelings were associated with the use of the digital library in general and easiness of use and effective navigation
in particular. Negative feelings were associated with the limited size of collection and uncertainty prior to searching the
system. Bilal and Kirby (2002) compared the internet search behaviour of adults and children and showed that while adults
(graduate students) performed the search tasks more effectively and efficiently, they experienced the same feelings as young
searchers. Both groups experienced satisfaction and comfort with the successful completion of task, and frustration due to
difficulties in finding the answer and inadequate knowledge of a system.
Kracker (2002), and Kracker and Wang (2002) focused on the effects of educating students about Kuhlthau’s Information
Search Process (ISP) model. Authors discovered that emotions related to anxiety, uncertainty and difficulty were mentioned
more frequently than positive emotions related to confidence and positive perceptions of the process. Positive emotions
were mostly correlated with the interest in the process and documents. The findings of this study are not consistent with
Tenopir et al. (2008), and Bilal (2000), who found that participants reported more positive than negative feelings. Inconsistencies
in findings might point to the lack of comparable definitions and methods as well as the need for further research.
Lopatovska and Mokros (2008) asked searchers to rate their feelings about individual web documents they reviewed. The
authors found that positive and negative feelings were caused primarily by documents’ stylistic properties (e.g., easy to
read), followed by the personal interest in the information contained in a document.
Studies that examined effects of emotions on various online search variables have shown relationships between emotions/affect
and search strategies (Nahl & Tenopir, 1996), performance (Nahl & Meer, 1997; Nahl, 1998; Wang et al.,
2000) and satisfaction (Nahl, 2004a; Nahl, 2004b). Early studies on the effects of emotions in information seeking were performed
in a library environment. While investigating the effects of the library anxiety, Mellon (1988) found that negative
emotions impeded information seeking and learning. The author suggested mitigating the negative effects of library anxiety
by offering library instruction programs that are attuned to students’ emotional needs. More recently, Onwuegbuzie et al.
(2004) also examined college students’ library anxiety and showed that it had a negative effect on research paper quality.
A series of studies by Diane Nahl, Carol Tenopir, Dania Bilal and others identified emotions experienced during the online
search and investigated their effects on search behaviour. Nahl and Tenopir (1996) explored affective and cognitive aspects
of searching behaviour of novice users and found that hesitation, desire to confirm, fear, surprise and other feelings affected
search strategies. Wang et al. (2000) also examined cognitive and affective aspects of the search behaviour on the web and
found reciprocal relationships between affect and search performance. The study findings showed that positive feelings supported
subsequent interactions while negative feelings hindered the search. The findings also indicated that successful
search performance reduced negative feelings, such as anxiety.
12 I. Lopatovska, I. Arapakis / Information Processing and Management xxx (2010) xxx–xxx
Please cite this article in press as: Lopatovska, I., & Arapakis, I. Theories, methods and current research on emotions in library and information
science, information retrieval and human–computer interaction. Information Processing and Management (2010), doi:10.1016/
j.ipm.2010.09.001
Nahl (2004b) investigated the effects of affective variables on search behaviour and found that self-efficacy and optimism
counteracted the effects of negative emotions (e.g., irritation and frustration associated with uncertainty and time pressure),
and increased user support and acceptance of the system. In a study of affective motivation during on-line information
search, Nahl (2004a) found positive correlation between self-efficacy and optimism, and motivation for completing the task.
The author found that higher self-efficacy and optimism were associated with higher satisfaction. The effects of self-efficacy
on search behaviour were also studied by Nahl and Meer (1997). The authors found positive correlation between students’
self-efficacy and search performance.
Kim (2008) examined relationships between search tasks, user emotion control and performance during the web search.
The experiment involved completion of different search tasks of varying scope (specific task vs. general task), and reporting
users’ expectations of problem solving. The study results indicated that both tasks and emotion control impact users’ search
behaviour. The author suggested ways of releasing cognitive and affective burden on the searcher by offering information
literacy education and improving interface design.
Nahl (1998) reviewed information behaviour literature covering cognitive and affective components of searching and
found evidence of the effect of affective variables on search motivation, performance and satisfaction. In conclusion, the reTable
4
Causes, effects and correlates of emotions.
Context Causes/correlates Emotions/feelings Effects/correlates Studies
Human–computer
interaction
Personalisable interface features Positive feelings (feelings
of control, ownership, fun,
relieve from boredom)
Tractinsky
(2004), and
Blom and Monk
(2003)
Computer game Supportive messages from computer Quicker recovery from
negative feelings; players
stayed in the game longer
Klein et al.
(2002)
Computer game Computer mouse not functioning properly Frustration Changes in blood volume
pressure, skin
conductivity, increased
mouse clicks
Scheirer et al.
(2002)
Computer game Apologies for the failed computer game expressed
through text and emoticons
Enhanced aesthetics;
positive feelings
Shortened psychological
distance between the
game and the player
Tzeng (2004),
and Brave et al.
(2005)
Children’s use of
search engine
use of a system in general; ease of use; keyword
search option; availability of graphics and fun;
software failures
positive feelings; negative
feelings of confusion and
frustration
No impact on persistence
and patience in web
searching
Bilal (2000)
Childrens use of
digital library
Use of the digital library in general; ease of use
and effective navigation; uncertainty prior to
searching the system and limited size of collection
positive and negative
feelings
Bilal and Bachir
(2007)
On-line search by
children and
adults
Successful completion of task; difficulties with
finding the answer and inadequate knowledge of a
system
Satisfaction; comfort;
frustration
Bilal and Kirby
(2002)
On-line search Interest in the process and documents Positive emotions Kracker (2002),
and Kracker
and Wang
(2002)
On-line search Search task difficulty levels Changes in positive/
negative valence
Arapakis et al.
(2008)
On-line search Documents’ stylistic properties (e.g., easy to read),
personal interest in the information contained in a
document
Positive and negative
feelings
Lopatovska and
Mokros (2008)
On-line search Hesitation, desire to
confirm, fear, surprise
Search strategies Nahl and
Tenopir (1996)
On-line search Successful search performance Positive feelings; reduced
negative feelings, such as
anxiety
Support of subsequent
interactions
Wang et al.
(2000)
On-line search Self-efficacy and
optimism
Increased user support
and acceptance of a
system
Nahl (2004b)
On-line search Self-efficacy, optimism Motivation for completing
the task; higher
satisfaction
Nahl (2004a)
On-line search Self-efficacy Search performance Nahl and Meer
(1997)
On-line search Emotion control Search behaviour Kim (2008)
Multiple contexts
(literature
review)
Affective variables Search motivation;
performance and
satisfaction
Nahl (1998)
I. Lopatovska, I. Arapakis / Information Processing and Management xxx (2010) xxx–xxx 13
Please cite this article in press as: Lopatovska, I., & Arapakis, I. Theories, methods and current research on emotions in library and information
science, information retrieval and human–computer interaction. Information Processing and Management (2010), doi:10.1016/
j.ipm.2010.09.001
viewed studies found that affect influences search strategies, performance, satisfaction, motivation to continue search,
acceptance and support of a system.
Table 4 summarises causes, effects and correlates of emotions examined in the mentioned above studies.
5. Discussion and conclusions
Recent scientific findings suggest that emotion is a ubiquitous element of any human–computer interaction (Brave et al.,
2005) and should be considered when designing usable and intelligent systems (Karat, 2003). Emotions not only regulate our
social encounters but also influence our cognition, perception and decision-making through a series of interactions with our
intentions and motivations (Damasio, 1994; Scherer, 2001). This article reviewed theories of emotions and illustrated how
emotions have been studied in the context of computer-related tasks.
We examined the two major categories of classical emotion theories: (i) cognitive, which stresses the importance of cognitive
evaluation (appraisal) in establishing the meaning of stimuli and ways of copying with it, and (ii) somatic, which
emphasises somatic factors and describe expressions and perceptions of emotional experiences in terms of bodily responses.
We, furthermore, discussed the two dominant views on emotion structure, namely: the discrete and dimensional. The former
supports the existence of six or more basic emotion categories, which are universally displayed and recognised, while
the latter suggests the representation of emotions in terms of a multi-dimensional space of arousal, valence and other.
Even though no accurate and holistic measurement of emotion exists, we presented several approaches for measuring
individual components of this phenomenon. The neuro-physiological methods examine the physiological changes that occur
during an emotion episode. These methods require the use of special equipment and are considered accurate at detecting
short-term changes that cannot be captured by other means. However, they are considered obtrusive and are prone to noise
introduced by unanticipated changes in the physiological characteristics. The observer methods (facial expression and
speech analysis, gesture recognition, etc.) are considered less obtrusive and do not require the use of special laboratory
equipment. They can be used for the analysis of temporal changes but are also noise-prone. Finally, self-report methods rely
on the assumption that a person can accurately assess and report his/her emotional experiences. This category of methods
offers insights into the subjective experiences, however it has its own limitations, such as reliance on ‘‘imperfect” language to
communicate emotional experiences. It is up to the individual researcher to weight all the advantages and disadvantages of
particular methods and select one that best addresses the study objectives.
We provided examples of the studies that used various methods to investigate the role of emotions in human–computer
interaction, including online searching. We have also reviewed several studies that integrated emotion data in the information
systems design. It is the authors’ opinion that the state of the knowledge about the role of emotions in HCI is still in its
infancy. Very often, studies that investigate affective variables, such as emotions, feelings, affect or mood, do not define these
concepts, which leads to the misuse of the terms and the presentation of incomparable findings. We also found lack of interdisciplinary
cross-referencing in the reviewed studies, which points to the lack of awareness about relevant research in the
adjacent disciplines that study emotions. Our review tried to educate the reader about multiple emotion theories and methods
that have been developed to aid emotion inquiries; and survey the work that has been done to date. We feel that it is
time for the emerging affective paradigm to become broader and richer, and to encourage research that reaches across the
boundaries of the narrowly defined fields.
Acknowledgements
Part of the research leading to this paper was supported by the European commission, under the contract FP6-033715
(MIAUCE Project). We would like to thank Nicholas J. Belkin and Joemon M. Jose for the fruitful discussions and encouragement,
and our anonymous reviewers for their constructive critique and suggestions.
References
Ambady, N., & Rosenthal, R. (1992). Thin slices of expressive behavior as predictors of interpersonal consequences: A meta-analysis. Psychological Bulletin,
111(2), 256–274.
Arapakis, I., Moshfeghi, Y., Joho, H., Ren, R., Hannah, D., & Jose, J. M. (2009). Enriching user profiling with affective features for the improvement of a
multimodal recommender system. In Proceeding of the ACM International Conference on Image and Video Retrieval (pp. 1–8). New York, NY, USA: ACM.
Arapakis, I., Jose, J. M., & Gray, P. D. (2008). Affective feedback: An investigation into the role of emotions in the information seeking process. In SIGIR ’08:
Proceedings of the 31st annual international ACM SIGIR conference on research and development in information retrieval (pp. 395–402). New York, NY: ACM.
Arapakis, I., Konstas, I., & Jose, J. M. (2009). Using facial expressions and peripheral physiological signals as implicit indicators of topical relevance. In
Proceedings of the seventeen ACM international conference on multimedia (pp. 461–470). USA: Springer.
Bamidis, P. D., Papadelis, C., Kourtidou-Papadeli, C., Pappas, C., & Vivas, A. B. (2004). Affective computing in the era of contemporary neurophysiology and
health informatics. Interacting with Computers, 16(4), 715–721.
Barrett, L. F., & Russell, J. A. (1999). The structure of current affect: Controversies and emerging consensus. Current Directions in Psychological Science, 8(1),
10–14.
Bilal, D. (2000). Children’s use of the yahooligans! web search engine: 1. Cognitive, physical, and affective behaviors on fact-based search tasks. Journal of the
American Society for Information Science, 51(7), 646–665.
Bilal, D. (2002). Children’s use of the yahooligans! web search engine. iii. Cognitive and physical behaviors on fully self-generated search tasks. Journal of the
American Society for Information Science and Technology, 53(13), 1170–1183.
Bilal, D., & Bachir, I. (2007). Children’s interaction with cross-cultural and multilingual digital libraries ii: Information seeking, success, and affective
experience. Information Processing and Management: An International Journal, 43(1), 65–80.
14 I. Lopatovska, I. Arapakis / Information Processing and Management xxx (2010) xxx–xxx
Please cite this article in press as: Lopatovska, I., & Arapakis, I. Theories, methods and current research on emotions in library and information
science, information retrieval and human–computer interaction. Information Processing and Management (2010), doi:10.1016/
j.ipm.2010.09.001
Bilal, D., & Kirby, J. (2002). Differences and similarities in information seeking: Children and adults as web users. Information Processing and Management: An
International Journal, 38(5), 649–670.
Black, M. J., & Yacoob, Y. (1997). Recognizing facial expressions in image sequences using local parameterized models of image motion. International Journal
of Computer Vision, 25(1), 23–48.
Blom, J. O., & Monk, A. F. (2003). Theory of personalization of appearance: Why users personalize their pcs and mobile phones. Human–Computer Interaction,
18(3), 193–228.
Boone, R. T., & Cunningham, J. G. (1998). Children’s decoding of emotion in expressive body movement: The development of cue attunement. Developmental
Psychology, 34(5), 1007–1016.
Bower, G. H. (1992). How might emotions affect learning? The handbook of cognition and emotion. Hillsdale, New Jersey: Lawrence Erlbaum Associates (pp.
3–31).
Brave, S., Nass, C., & Hutchinson, K. (2005). Computers that care: Investigating the effects of orientation of emotion exhibited by an embodied computer
agent. International Journal of Human–Computer Studies, 62 (2), 161–178. <http://www.sciencedirect.com/science/article/B6WGR-4F5SB91-1/2/
c3731ef2fb0432046a13222809708ed9>.
Brave, S., & Nass, C. (2003). Emotion in Human–Computer Interaction, 81–96.
Breazeal, C. (2001). Designing social robots. Cambridge, MA: MIT Press.
Brewster, S., Lumsden, J., Bell, M., Hall, M., & Tasker, S. (2003). Multimodal ‘eyes-free’ interaction techniques for wearable devices. In Proceedings of the
SIGCHI conference on human factors in computing systems (pp. 473–480). New York, NY, USA: ACM.
Camurri, A., Lagerlöf, I., & Volpe, G. (2003). Recognizing emotion from dance movement: Comparison of spectator recognition and automated techniques.
International Journal of Human–Computer Studies, 59(1–2), 213–225.
Caridakis, G., Castellano, G., Kessous, L., Raouzaiou, A., Malatesta, L., Asteriadis, S., et al (2007). Multimodal emotion recognition from expressive faces, body
gestures and speech. Artificial intelligence and innovations 2007: From theory to applications (Vol. 247, pp. 375–388). Berlin, Heidelberg: Springer-Verlag.
Castellano, G., Villalba, S. D., & Camurri, A. (2007). Recognising human emotions from body movement and gesture dynamics. In Proceedings of the 2nd
international conference on affective computing and intelligent interaction (pp. 71–82). Berlin, Heidelberg: Springer-Verlag.
Chandra, A., & Calderon, T. (2005). Challenges and constraints to the diffusion of biometrics in information systems. Communications of the ACM, 48(12),
101–106.
Chan, C. H., & Jones, G. J. F. (2005). Affect-based indexing and retrieval of films. In Proceedings of the 13th annual ACM international conference on multimedia
(pp. 427–430). New York, NY, USA: ACM.
Chen, F.-S., Fu, C.-M., & Huang, C.-L. (2003). Hand gesture recognition using a real-time tracking method and hidden markov models. Image and Vision
Computing, 21(8), 745–758. <http://www.sciencedirect.com/science/article/B6V09-48NC790-1/2/8f12375d2a82de6de563284fd02d3f23>.
Cohn, J., & Kanade, T. (2006). Use of automated facial image analysis for measurement of emotion expression. In J. A. Coan & J. B. Allen (Eds.), The handbook of
emotion elicitation and assessment. Oxford University Press Series in Affective Science.
Corradini, A., Mehta, M., Bernsen, N., & Martin, J. C. (2003). Multimodal input fusion in human–computer interaction. In NATO-ASI conference on data fusion
for situation monitoring, incident detection, alert, and response management.
Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W., et al (2001). Emotion recognition in human–computer interaction. Signal
Processing Magazine, IEEE, 18(1), 32–80.
Damasio, A. R. (1994). Descartes error: Emotion, reason, and the human brain. New York, NY, USA: Putnam/Grosset Press.
Darwin, C. (2005). The expression of the emotions in man and animals. Kessinger Publishing.
de Meijer, M. (2005). The contribution of general features of body movement to the attribution of emotions. Journal of Nonverbal Behavior, 13(4), 247–268.
Eckhardt, M., & Picard, R. W. (2009). A more effective way to label affective expressions. In Proceedings of the 2009 international conference on affective
computing and intelligent interaction.
Ekman, P. (1984). Expression and the nature of emotion. Approaches to emotion. Hillsdale, New Jersey: Lawrence Erlbaum Associates (pp. 319–344).
Ekman, P. (1992). An argument for basic emotions. Cognition and Emotion, 6, 169–200.
Ekman, P. (1999a). Basic emotions. The handbook of cognition and emotion. UK: John Wiley & Sons, Ltd. (pp. 45–60).
Ekman, P. (1999b). Facial expressions. The handbook of cognition and emotion. UK: John Wiley & Sons, Ltd. (pp. 301–320).
Ekman, P. (2003). Emotions revealed: Recognizing faces and feelings to improve communication and emotional life. New York: Times Books.
Ekman, P., & Friesen, W. V. (1975). Unmasking the face. A guide to recognizing emotions from facial clues. Englewood Cliffs, New Jersey: Prentice-Hall.
Essa, I. A., & Pentland, A. P. (1997). Coding, analysis, interpretation, and recognition of facial expressions. IEEE Transactions on Pattern Analysis and Machine
Intelligence, 19(7), 757–763.
Fasel, B., & Luettin, J. (2003). Automatic facial expression analysis: A survey. Pattern Recognition, 36(1), 259–275 (iDIAP-RR 99-19).
Folkman, S., Lazarus, R. S., Gruen, R. J., & DeLongis, A. (1986). Appraisal, coping, health status, and psychological symptoms. Journal of Personality and Social
Psychology, 50(3), 571–579.
Fox, S., Karnawat, K., Mydland, M., Dumais, S., & White, T. (2005). Evaluating implicit measures to improve web search. ACM Transactions on Information
Systems (TOIS), 23(2), 147–168.
Frijda, N. H. (1986). The emotions. Paris, France, EU: Cambridge University Press.
Frijda, N. H. (1994). Varieties of affect: Emotions and episodes, moods, and sentiments. The nature of emotion. New York: Oxford University Press (pp. 59–
67).
Go, H.-J., Kwak, K.-C., Lee, D.-J., & Chun, M.-G. (2003). Emotion Recognition from the Facial Image and Speech Signal, 3, 2890–2895.
Gunes, H., & Piccardi, M. (2007). Bi-modal emotion recognition from expressive face and body gestures. Journal of Network and Computer Applications, 30(4),
1334–1345.
Gwizdka, J., & Lopatovska, I. (2009). The role of subjective factors in the information search process. Journal of the American Society for Information Science and
Technology, 60(12), 2452–2464.
Hanjalic, A., & Xu, L.-Q. (2005). Affective video content representation and modeling. Multimedia, IEEE Transactions on, 7(1), 143–154.
Hong, H., Neven, H., & von der Malsburg, C. (1998). Online facial expression recognition based on personalized galleries. In Proceedings of the 3rd
international conference on face & gesture recognition (p. 354). Washington, DC, USA: IEEE Computer Society.
Hudlicka, E. (2003). To feel or not to feel: The role of affect in human–computer interaction. International Journal of Human–Computer Studies, 59(1–2), 1–32.
Isbister, K., Höök, K., Sharp, M., & Laaksolahti, J. (2006). The sensual evaluation instrument: Developing an affective evaluation tool. In Proceedings of the
SIGCHI conference on human factors in computing systems (pp. 1163–1172). New York, NY, USA: ACM.
Jaimes, A., & Sebe, N. (2007). Multimodal human–computer interaction: A survey. Computer Vision and Image Understanding, 108(1–2), 116–134.
James, L., & Nahl, D. (1996). Achieving focus, engagement, and acceptance: Three phases of adapting to internet use. Electronic Journal on Virtual Culture, 4
(1). <http://www2.hawaii.edu/nahl/articles/ejvc.html>.
Johnson-Laird, P. N., & Oatley, K. (1989). The language of emotions: An analysis of a semantic field. Cognition and Emotion, 3, 81–123.
Julien, H. (2007). Experiencing information literacy affectively. Information and emotion: The emergent paradigm in information behavior research and theory.
Medford, NJ: Information Today (pp. 243–254).
Julien, H., McKechnie, L. E., & Hart, S. (2005). Affective issues in library and information science systems work: A content analysis. Library & Information
Science Research, 27(4), 453–466.
Kahneman, D. (2000). Experienced utility and objective happiness: A moment-based approach. Choices, values, and frames. New York, NY, USA: Cambridge
University Press (pp. 673–692).
Kaliouby, R.E., & Robinson, P. (2004). Real-time inference of complex mental states from facial expressions and head gestures. In Proceedings of the 2004
conference on computer vision and pattern recognition workshop (CVPRW’04) (Vol. 10, p. 154). Washington, DC, USA: IEEE Computer Society.
I. Lopatovska, I. Arapakis / Information Processing and Management xxx (2010) xxx–xxx 15
Please cite this article in press as: Lopatovska, I., & Arapakis, I. Theories, methods and current research on emotions in library and information
science, information retrieval and human–computer interaction. Information Processing and Management (2010), doi:10.1016/
j.ipm.2010.09.001
Kapoor, A., Burleson, W., & Picard, R. W. (2007). Automatic prediction of frustration. International Journal of Human–Computer Studies, 65(8), 724–736.
Kapoor, A., Picard, R.W., & Ivanov, Y. (2004). Probabilistic combination of multiple modalities to detect interest. In Proceedings of the pattern recognition, 17th
international conference on (ICPR’04) (Vol. 3, pp. 969–972). Washington, DC, USA: IEEE Computer Society.
Karat, J. (2003). Beyond task completion: Evaluation of affective components of use. Hillsdale, NJ, USA: L. Erlbaum Associates Inc. (pp. 1152–1164).
Kim, J. (2006). Task difficulty as a predictor and indicator of web searching interaction. In CHI ’06 extended abstracts on human factors in computing systems
(pp. 959–964). New York, NY, USA: ACM.
Kim, K.-S. (2008). Effects of emotion control and task on web searching behavior. Information Processing and Management: An International Journal, 44(1),
373–385.
Kimura, S., & Yachida, M. (1997). Facial expression recognition and its degree estimation. In Proceedings of the 1997 conference on computer vision and pattern
recognition (CVPR ’97) (p. 295). Washington, DC, USA: IEEE Computer Society.
Kirishima, T., Sato, K., & Chihara, K. (2005). Real-time gesture recognition by learning and selective control of visual interest points. IEEE Transactions on
Pattern Analysis and Machine Intelligence, 27(3), 351–364.
Klein, J., Moon, Y., & Picard, R. W. (1999). This computer responds to user frustration. In CHI ’99 extended abstracts on human factors in computing systems (pp.
242–243). New York, NY, USA: ACM.
Kleinginna, P., & Kleinginna, A. (2005). A categorized list of motivation definitions, with a suggestion for a consensual definition. Motivation and Emotion,
5(3), 263–291.
Klein, J., Moon, Y., & Picard, R. W. (2002). This computer responds to user frustration: Theory, design and results. Interacting with Computers, 14(2), 119–140.
Kleinsmith, A., De Silva, P., & Bianchi-Berthouze, N. (2006). Cross-cultural differences in recognizing affect from body posture. Interacting with Computers,
18(6), 1371–1389.
Kracker, J. (2002). Research anxiety and students’ perceptions of research: An experiment. Part I: Effect of teaching kuhlthau’s ISP model. Journal of the
American Society for Information Science and Technology, 53(4), 282–294.
Kracker, J., & Wang, P. (2002). Research anxiety and students’ perceptions of research: An experiment. Part II: Content analysis of their writings on two
experiences. Journal of the American Society for Information Science and Technology, 53(4), 295–307.
Kuhlthau, C. C. (1991). Inside the search process: Information seeking from the user’s perspective. Journal of the American Society for Information Science,
42(5), 361–371.
Lanitis, A., Taylor, C., & Cootes, T. (1997). Automatic interpretation and coding of face images using flexible models. IEEE Transactions on Pattern Analysis and
Machine Intelligence, 19(7), 743–756.
Larsen, R. J., & Fredrickson, B. L. (1999). Measurement issues in emotion research. Well-being: The foundations of hedonic psychology. New York: Russell Sage
Foundation (pp. 40–60).
Lazarus, R. S. (1984). Thoughts on the relations between emotion and cognition. Approaches to emotion. Hillsdale, New Jersey: Lawrence Erlbaum Associates
(pp. 247–259).
Lee, H. J., & Neal, D. (2007). Towards web 2.0 music information retrieval: Utilizing emotion-based, user-assigned descriptions. In Proceedings of the 70th
annual meeting of the American society of information science and technology. Joining research and practice: Social computing and information science (Vol.
45, pp. 732–741).
Licsár, A., & Szirányi, T. (2005). User-adaptive hand gesture recognition system with interactive training. Image and Vision Computing, 23(12), 1102–1114.
Lindgaard, G. (2004). Adventurers versus nit-pickers on affective computing. Interacting with Computers, 16(4), 723–728. <http://www.sciencedirect.com/
science/article/B6V0D-4CYR0BP-1/2/a5f30b5517b7f20158280770bac1299f> (human–computer interaction in Latin America).
Lopatovska, I. (2009a). Emotional aspects of the online information retrieval process. Ph.D. Thesis. Rutgers: The State University of New Jersey.
Lopatovska, I. (2009b). Searching for good mood: Examining relationships between search task and mood. In Proceedings of the 72th annual meeting of the
American society for information science and technology, in press.
Lopatovska, I., & Cool, C. (2008). Online search: Uncovering affective characteristics of information retrieval experience. Presented at the 2008 ALISE annual
conference, Philadelphia, PA.
Lopatovska, I., & Mokros, H. B. (2008). Willingness to pay and experienced utility as measures of affective value of information objects: Users’ accounts.
Information Processing and Management: An International Journal, 44(1), 92–104.
McAllister, G., McKenna, S. J., & Ricketts, I. W. (2002a). Hand tracking for behaviour understanding. Image and Vision Computing, 20(12), 827–840. <http://
www.sciencedirect.com/science/article/B6V09-46SFPJF-2/2/8b8cdc3f055133ade27ac09bd8b3fa56>.
McAllister, G., McKenna, S. J., & Ricketts, I. W. (2002b). Hand tracking for behaviour understanding. Image and Vision Computing, 20(12), 827–840. <http://
www.sciencedirect.com/science/article/B6V09-46SFPJF-2/2/8b8cdc3f055133ade27ac09bd8b3fa56>.
Meghabghab, D. B. (1995). Cd-rom vs. online vs. internet: Search strategies and evaluation from the user’s perspective. In Proceedings of the 16th national
online meeting (pp. 295–307).
Mellon, C. A. (1988). Attitudes: The forgotten dimension in library instruction. Library Journal, 113(14), 137–139.
Mendoza, E., & Carballo, G. (1999). Vocal tremor and psychological stress. Journal of Voice, 13(1), 105–112. <http://www.sciencedirect.com/science/article/
B7585-4GM9P0N-D/2/01036b570cbfb0ab303f793bc474217a>.
Mentis, H. M. (2007). Memory of frustrating experiences. Information and emotion: The emergent paradigm in information behavior research and theory.
Medford, NJ: Information Today (pp. 197–210).
Mitroff, I. I. (1974). A brunswik lens model of dialectical inquiring systems. Theory and Decision, 5(1), 45–67.
Mooney, C., Scully, M., Jones, G. J., & Smeaton, A. F. (2006). Investigating biometric response for information retrieval applications. Lecture Notes in Computer
Science, 570–574.
Morency, L.-P., & Darrell, T. (2006). Head gesture recognition in intelligent interfaces: The role of context in improving recognition. In Proceedings of the 11th
international conference on Intelligent user interfaces (pp. 32–38). New York, NY, USA: ACM.
Morency, L.-P., Sidner, C., Lee, C., & Darrell, T. (2007). Head gestures for perceptual interfaces: The role of context in improving recognition. Artificial
Intelligence, 171(8–9), 568–585.
Muller, M. (2004). Multiple paradigms in affective computing. Interacting with Computers, 16(4), 759–768 (human–computer interaction in Latin America).
<http://www.sciencedirect.com/science/article/B6V0D-4DKK9F6-6/2/733dc4edeea63dee300301e91d8f3fb4>.
Murray, I. R., & Arnott, J. L. (1993). Toward the simulation of emotion in synthetic speech: A review of the literature on human vocal emotion. Journal of the
Acoustic Society of America, 93(2), 1097–1108.
Nahl, D. (1998). Learning the internet and the structure of information behavior. Journal of the American Society for Information Science, 49(11), 1017–1023.
Nahl, D. (2004a). Affective and cognitive information behavior: Interaction effects in internet use. In Proceedings of the 68th ASIST annual Meeting (Vol. 42).
Medford, NJ: Information Today.
Nahl, D. (2004b). Measuring the affective information environment of web searchers. In Proceedings of the American society for information science and
technology (Vol. 41, pp. 191–197).
Nahl, D., & Meer, M. P. (1997). User-centered assessment of two web browsers: Errors, perceived self-efficacy, and success. In Proceedings of the ASIS annual
meeting (Vol. 34, pp. 89–97).
Nahl, D., & Bilal, D. (Eds.). (2007). Information and emotion: The emergent affective paradigm in information behavior research and theory. SMedford, New
Jersey: Information Today, Inc.
Nahl, D., & Tenopir, C. (1996). Affective and cognitive searching behavior of novice end-users of a full-text database. Journal of the American Society for
Information Science, 47(4), 276–286.
Ng, C. W., & Ranganath, S. (2002). Real-time gesture recognition system and application. Image and Vision Computing, 20(13–14), 993–1007.
Norman, D. A. (2004). Emotional design. Ubiquity, 4(45), 1.
16 I. Lopatovska, I. Arapakis / Information Processing and Management xxx (2010) xxx–xxx
Please cite this article in press as: Lopatovska, I., & Arapakis, I. Theories, methods and current research on emotions in library and information
science, information retrieval and human–computer interaction. Information Processing and Management (2010), doi:10.1016/
j.ipm.2010.09.001
Oatley, K. (2004). The bug in the salad: The uses of emotions in computer interfaces. Interacting with Computers, 16(4), 693–696 (human–computer
interaction in Latin America). <http://www.sciencedirect.com/science/article/B6V0D-4CTTR8B-2/2/6dc649fe1fd9f22302f3e02dd96f10a4>.
Onwuegbuzie, A. J., & Jiao, Q. G. (2004). Information search performance and research achievement: An empirical test of the anxiety-expectation mediation
model of library anxiety. Journal of the American Society for Information Science and Technology, 55(1), 41–54.
Ortony, A., Clore, G. L., & Collins, A. (1988). The cognitive structure of emotions. New York, NY, USA: Cambridge University Press (pp. 10011–4211).
Pantic, M., Sebe, N., Cohn, J. F., & Huang, T. (2005). Affective multimodal human–computer interaction. In Proceedings of the 13th annual ACM international
conference on multimedia (pp. 669–676). New York, NY, USA: ACM.
Pantic, M., & Rothkrantz, L. J. (2000a). Automatic analysis of facial expressions: The state of the art. IEEE Transactions on Pattern Analysis and Machine
Intelligence, 22(12), 1424–1445.
Pantic, M., & Rothkrantz, L. (2000b). Expert system for automatic analysis of facial expression. Image and Vision Computing Journal, 18(11), 881–905. <http://
pubs.doc.ic.ac.uk/Pantic-IVCJ00/>.
Pantic, M., & Rothkrantz, L. J. (2003). Toward an affect-sensitive multimodal human–computer interaction. Proceedings of the IEEE, 91(9), 1370–1390.
Partala, T., & Surakka, V. (2003). Pupil size variation as an indication of affective processing. International Journal of Human–Computer Studies, 59(1–2),
185–198.
Partala, T., & Surakka, V. (2004). The effects of affective interventions in human–computer interaction. Interacting with Computers, 16(2), 295–309.
Partala, T., Surakka, V., & Vanhala, T. (2006). Real-time estimation of emotional experiences from facial expressions. Interacting with Computers, 18(2),
208–226.
Peter, C., & Herbon, A. (2006). Emotion representation and physiology assignments in digital systems. Interacting with Computers, 18(2), 139–170.
Picard, R. W. (1997). Affective computing. Cambridge, MA, USA: MIT Press.
Picard, R. W. (2001). Building hal: Computers that sense, recognize, and respond to human emotion. In Society of photo-optical instrumentation engineers.
Human vision and electronic imaging VI (Vol. 4299, pp. 518–523).
Picard, R. W. (2003). Affective computing: Challenges. International Journal of Human–Computer Studies, 59(1–2), 55–64.
Picard, R. W., & Klein, J. (2002). Computers that recognise and respond to user emotion: Theoretical and practical implications. Interacting with Computers,
14(2), 141–169.
Picard, R. W., Vyzas, E., & Healey, J. (2001). Toward machine emotional intelligence: Analysis of affective physiological state. IEEE Transactions on Pattern
Analysis and Machine Intelligence, 23(10), 1175–1191.
Plutchik, R. (1980). A general psychoevolutionary theory of emotion. Emotion: Theory, research and experience. Theories of emotion (Vol. 1). New York:
Academic (pp. 3–33).
Reeves, B., & Nass, C. I. (1996). The media equation: How people treat computers. Television and new media like real people and places. Cambridge, England:
Cambridge University Press (Book Review).
Ren, F. (2009). Affective information processing and recognizing human emotion. Electronic Notes in Theoretical Computer Science (ENTCS), 225, 39–50.
Rosenberg, E. L., Ekman, P. (1993). Facial expression and emotion. Neuroscience year: Supplement 3 to the encyclopedia of neuroscience. Birkhäuser, pp. 51–
52.
Russell, J. A. (1994). Is there universal recognition of emotion from facial expression? Psychological Bulletin, 115, 102–141.
Russell, J. A., Bachorowski, J., & Fernandez-Dols, J. (2003). Facial and vocal expressions of emotion. Annual Review of Psychology, 54, 329–349.
Russell, J. A., & Mehrabian, A. (1977). Evidence for a three-factor theory of emotions. Journal of Research in Personality, 11(3), 273–294.
Russell, J. A., & Steiger, J. H. (1982). The structure in persons’ implicit taxonomy of emotions. Research in Personality, 16, 447–469.
Sander, D., Grandjean, D., Pourtois, G., Schwartz, S., Seghier, M. L., Scherer, K. R., et al. (2005). Emotion and attention interactions in social cognition: Brain
regions involved in processing anger prosody. NeuroImage, 28 (4), 848–858 (special Section: Social Cognitive Neuroscience).
Sander, D., Grandjean, D., & Scherer, K. R. (2005). A systems approach to appraisal mechanisms in emotion. Neural Networks, 18(4), 317–352.
Schachter, S., & Singer, J. (1962). Cognitive, social and physiological determinants of emotional state. Psychological Review, 69, 379–399.
Schapira, E., & Sharma, R. (2001). Experimental evaluation of vision and speech based multimodal interfaces. In Proceedings of the 2001 workshop on
perceptive user interfaces (pp. 1–9). New York, NY, USA: ACM.
Scheirer, J., Fernandez, R., Klein, J., & Picard, R. W. (2002). Frustrating the user on purpose: A step toward building an affective computer. Interacting with
Computers, 14(2), 93–118. <http://www.sciencedirect.com/science/article/B6V0D-44GF457-1/2/9601b14ed27badfe069784931ea7cc31>.
Scherer, K. R. (2001). Appraisal considered as a process of multi-level sequential checking. In K. R. Schorr, A. Scherer, & T. Johnstone (Eds.), Appraisal processes
in emotion: Theory, methods, research. New York, Oxford: Oxford University Press.
Scherer, K. R. (2002). Emotion, the psychological structure of emotions. International encyclopedia of the social & behavioral sciences. Oxford: Harvard Libraries.
Scherer, K. R. (2003). Vocal communication of emotion: A review of research paradigms. Speech Communication, 40(1–2), 227–256.
Scherer, K. R. (2005). What are emotions? And how can they be measured? Social Science Information, 44(4), 695–729.
Schmidt, S., & Stock, W. G. (2009). Collective indexing of emotions in images. A study in emotional information retrieval. Journal of the American Society for
Information Science and Technology, 60(5), 863–876. <http://dx.doi.org/10.1002/asi.21043>.
Schuller, B., Lang, M., & Rigoll, G. (2002). Multimodal emotion recognition in audiovisual communication, 1, 745–748.
Schuller, B., Arsic, D., Wallhoff, F., & Rigoll, G. (2006). Emotion recognition in the noise applying large acoustic feature sets. In Proceedings of speech prosody.
International symposium on computer architecture, Dresden.
Sebe, N., Bakker, E., Cohen, I., Gevers, T., & Huang, T. S. (2005). Bimodal emotion recognition. In 5th International conference on methods and techniques in
behavioral research. Wageningen, Netherlands.
Smeaton, A. F., & Rothwell, S. (2009). Biometric responses to music-rich segments in films: The cdvplex. In Seventh international workshop on content-based
multimedia indexing (pp. 162–168).
Soleymani, M., Chanel, G., Kierkels, J. J., & Pun, T. (2008a). Affective ranking of movie scenes using physiological signals and content analysis. In Proceeding of
the 2nd ACM workshop on multimedia semantics (pp. 32–39). New York, NY, USA: ACM.
Soleymani, M., Chanel, G., Kierkels, J. J., & Pun, T. (2008b). Affective characterization of movie scenes based on multimedia content analysis and user’s
physiological emotional responses. In Proceedings of the 2008 tenth IEEE international symposium on multimedia (pp. 228–235). Washington, DC, USA:
IEEE Computer Society.
Song, M., Bu, J., Chen, C., & Li, N. (2004). Audio-visual based emotion recognition – A new approach. 2 (June-2 July), 1020–1025.
Swindells, C., MacLean, K. E., Booth, K. S., & Meitner, M. (2006). A case-study of affect measurement tools for physical user interface design. In Proceedings of
graphics interface 2006 (pp. 243–250). Toronto, Ont., Canada, Canada: Canadian Information Processing Society.
Tenopir, C., Wang, P., Zhang, Y., Simmons, B., & Pollard, R. (2008). Academic users’ interactions with sciencedirect in search tasks: Affective and cognitive
behaviors. Information Processing and Management: An International Journal, 44(1), 105–121.
Tomkins, S. S. (1984). Affect theory. Approaches to emotion. Hillsdale, New Jersey: Lawrence Erlbaum Associates (pp. 163–197).
Tractinsky, N. (2004). Tools over solutions? Comments on interacting with computers special issue on affective computing. Interacting with Computers,
16(4), 751–757 (human–computer interaction in Latin America). <http://www.sciencedirect.com/science/article/B6V0D-4CYNT51-1/2/
aa4ca242421dfdac1f9dc1c91d0206ef>.
Tsalakanidou, F., Malassiotis, S., & Strintzis, M. G. (2007). A 3d face and hand biometric system for robust user-friendly authentication. Pattern Recognition
Letters, 28(16), 2238–2249.
Tzeng, J.-I. (2004). Toward a more civilized design: Studying the effects of computers that apologize. International Journal of Human–Computer Studies, 61(3),
319–345. <http://www.sciencedirect.com/science/article/B6WGR-4BRPJP0-6/2/47b1d8dc22f6bf870a0a469d6db96900>.
Wallbott, H. G. (1998). Bodily expression of emotion. European Journal of Social Psychology, 28(6).
I. Lopatovska, I. Arapakis / Information Processing and Management xxx (2010) xxx–xxx 17
Please cite this article in press as: Lopatovska, I., & Arapakis, I. Theories, methods and current research on emotions in library and information
science, information retrieval and human–computer interaction. Information Processing and Management (2010), doi:10.1016/
j.ipm.2010.09.001
Wang, M., Iwai, Y., & Yachida, M. (1998). Expression recognition from time-sequential facial images by use of expression change model. In Proceedings of the
3rd international conference on face & gesture recognition (p. 324). Washington, DC, USA: IEEE Computer Society.
Wang, P., Hawk, W. B., & Tenopir, C. (2000). Users’ interaction with world wide web resources: An exploratory study using a holistic approach. Information
Processing and Management: An International Journal, 36(2), 229–251.
Wang, P., & Soergel, D. (1998). A cognitive model of document use during a research project. Study i. Document selection. Journal of the American Society for
Information Science, 49(2), 115–133.
Ward, R. D., & Marsden, P. H. (2004). Affective computing: Problems, reactions and intentions. Interacting with Computers, 16(4), 707–713. (human–
computer interaction in Latin America). <http://www.sciencedirect.com/science/article/B6V0D-4CTTR8B-1/2/4901d0cfad70855cd0a1ecd4febfce35>.
Watson, D., Clark, L. A., & Tellegen, A. (1988). Development and validation of brief measures of positive and negative affect: The panas scales. Journal of
Personality and Social Psychology, 54(6), 1063–1070.
Wilhelm, F. H., Pfaltz, M. C., & Grossman, P. (2006). Continuous electronic data capture of physiology, behavior and experience in real life: Towards
ecological momentary assessment of emotion. Interacting with Computers, 18(2), 171–186.
Wilson, T. D., Ford, N., Ellis, D., Foster, A., & Spink, A. (2002). Information seeking and mediated searching. Part 2: Uncertainty and its correlates. Journal of the
American Society for Information Science and Technology, 53(9), 704–715.
Wundt, W. (1904). Principles of physiological psychology. London: Swan Sonnenschein.
Yoshitomi, Y., Kim, S.-I., Kawano, T., & Kilazoe, T. (2000). Effect of sensor fusion for recognition of emotional states using voice, face image and thermal image
of face. In Proceedings of the 9th IEEE international workshop on robot and human interactive communication (pp. 178–183).
Zajonc, R. B. (1984). Affect theory. Approaches to emotion. Hillsdale, New Jersey: Lawrence Erlbaum Associates (pp. 239–246).
Zeng, Z., Hu, Y., Fu, Y., Huang, T. S., Roisman, G. I., & Wen, Z. (2006). Audio-visual emotion recognition in adult attachment interview. In Proceedings of the 8th
international conference on multimodal interfaces (pp. 139–145). New York, NY, USA: ACM.
Zhang, Z., Lyons, M., Schuster, M., & Akamatsu, S. (1998). Comparison between geometry-based and gabor-wavelets-based facial expression recognition
using multi-layer perceptron. In Proceedings of the 3rd. international conference on face & gesture recognition (p. 454). Washington, DC, USA: IEEE
Computer Society.
18 I. Lopatovska, I. Arapakis / Information Processing and Management xxx (2010) xxx–xxx
Please cite this article in press as: Lopatovska, I., & Arapakis, I. Theories, methods and current research on emotions in library and information
science, information retrieval and human–computer interaction. Information Processing and Management (2010), doi:10.1016/
j.ipm.2010.09.001