User-Centered Evaluation of Visualization
Video from MD SHAHRIAR ISLAM / Pixabay
Simone Kriglstein
kriglstein@mail.muni.cz
Image from https://extremepresentation.typepad.com/files/choosing-a-good-chart-09.pdf
Success of a visualization depends strongly on
the target users
• their expectations and needs
• how well they can interpret the data with the
visualization
• understandable/ correct interpretation
• which visualization approach is better?
• why does the approach not work?
• what can we change?
• how can we improve it?
• …Photo by fotografierende from Pexels
What will be evaluated?
Photo by vjapratama from Pexels
Prototyping
There exist many types of prototypes of varying
degrees of fidelity, e.g.:
• single page vs. multipage such that the user can complete a
task
• realistic and detailed vs. hand-sketched
• interactive (clickable) vs. static (a person simulates the
system)
Level of fidelity -> the level of detail and functionality
• Low-fidelity
• High-fidelity
Fidelity of a prototype can vary:
• Visual design
• Content
• Interactivity
Source: https://www.nngroup.com/articles/ux-prototype-hi-lo-fidelity/
https://blog.adobe.com/en/publish/2017/11/29/prototyping-difference-low-fidelity-
high-fidelity-prototypes-use.html#gs.qcml9j
Bors, C., Gschwandtner, T., & Miksch, S. (2014). Qualityflow: Provenance
generation from data quality. In Poster Proceedings of the EuroGraphics
Conference on Visualization.
Understanding the impact of operations on the data’s
quality and structural changes
Who will be evaluated?
Photo by vjapratama from Pexels
Expert evaluation
Review of the visualization approach
Good if you
• have limited access to users
• need to conduct an extremely fast review ->
do not have time to recruit participants
But experts are often not the user!!!
Recommended before a user study to find large
number of basic problems
Ideally, experts in HCI and the domain (no or
little involvement in the project)
• rarely covered by one person -> 2-3 experts with
different backgrounds
Photo by Sora Shimazaki from Pexels
Wilson, C. (2009). User experience re-mastered: your guide to getting the right design. Morgan Kaufmann.
Rubin, J., & Chisnell, D. (2008). Handbook of usability testing: how to plan, design and conduct effective
tests. John Wiley & Sons.
User Study
“Real” Target user
Goal
• Feedback on strengths and weaknesses of the
design
• Identify usability problems
• Identify improvement possibilities
• Comparison between different approaches
Two approaches:
• Attitudinal – you listen to users’ words (e.g., in
interviews)
• Behavioral – you watch their actions through
observational studies
Rubin, J., & Chisnell, D. (2008). Handbook of usability testing: how to plan, design and conduct effective
tests. John Wiley & Sons.
Source: https://www.interaction-design.org/literature/topics/user-research
Photo by Sora Shimazaki from Pexels
Where will be evaluated?
Photo by vjapratama from Pexels
User Study
Testing Environment
• Lab
• User’s place (field study)
Rubin, J., & Chisnell, D. (2008). Handbook of usability testing: how to plan, design and conduct effective
tests. John Wiley & Sons.
Photo by Sora Shimazaki from Pexels
User Study
Testing Environment
• Lab
• Simulate different testing
setups/environments (e.g., office setting)
• Same conditions -> better to compare
• Controllable
• Leaves the user and moderator alone
• Observers are in a separate room
• If you don’t have a lab -> portable lab
equipment and a quiet room
• Laptop for the user
• Laptop for the moderator (e.g., to take notes)
• Webcam to record the user
• Input device
Rubin, J., & Chisnell, D. (2008). Handbook of usability testing: how to plan, design and conduct
effective tests. John Wiley & Sons.
Source: https://www.nngroup.com/articles/traveling-usability-lab/
Image from: Rubin, J., & Chisnell, D. (2008). Handbook of usability testing: how
to plan, design and conduct effective tests. John Wiley & Sons. (page 102)
User Study
Testing Environment
• User’s site (field study)
• Test in real context and location
-> important if the environment can influence
the design
• Need special planning -> logistics are not
trivial -> portable lab equipment
• Decide where to best observe people in action
• Be sensitive and considerate -> person’s
personal space (workplace or home)
Photo by Sora Shimazaki from Pexels
Rubin, J., & Chisnell, D. (2008). Handbook of usability testing: how to plan, design and conduct effective
tests. John Wiley & Sons.
Source: https://www.nngroup.com/articles/field-studies/
How will be evaluated?
Photo by vjapratama from Pexels
User Study
Test Preparation
1. Identify and specify the purpose and goals of the test
• Specify the questions what you would like to
answer with this test
2. Identify and define participant characteristics (e.g.,
background, expertise, domain,…) -> Personas can
help
3. Recruit test persons -> needs time!!
4. Method(s) (test design)
5. Specify which data has to be collected
6. Specify the test environment and equipment (e.g.,
audio and/or video recorder, check batteries!!)
7. Specify the tasks for the test
8. Prepare materials (e.g., pen, notepad,…)
9. Prepare the test procedure/test plan
10. Letter of agreement (permission of the participants)
Photo by Sora Shimazaki from Pexels
Rubin, J., & Chisnell, D. (2008). Handbook of usability testing: how to plan, design and conduct effective
tests. John Wiley & Sons.
User Study
Tasks
Representative of typical user activities
But: do not focus too much on a specific feature
-> see how users will use the visualization
Tasks should have the following characteristics:
• Typical kind of things that people will do
• Describe tasks as scenarios -> not too long and easy
to understand
• Tasks should have a specific end goal -> fulfilled/not
fulfilled
• Avoid giving clues -> Step descriptions often contain
hidden clues
Photo by Sora Shimazaki from Pexels
Wilson, C. (2009). User experience re-mastered: your guide to getting the right design. Morgan Kaufmann.
Source: https://www.nngroup.com/articles/task-scenarios-usability-testing/
User Study
Qualitative data/methods
• for answering questions about why or how to fix a
problem
• to get an in-depth understanding of the experiences
and everyday lives of individual users or user groups
• e.g.: observation notes, user explanations and
opinions, non-numeric terms, anecdotes,
transcripts,...
Quantitative data/methods
• for answering how many and how much
• to measure user behavior in a way that can be
quantified and used for statistical analysis
• e.g.: logging of user actions, speed, error rate,
counts of activities
Rubin, J., & Chisnell, D. (2008). Handbook of usability testing: how to plan, design and conduct effective tests. John Wiley & Sons.
Source: https://www.interaction-design.org/literature/article/user-research-what-it-is-and-why-you-should-do-it
https://www.nngroup.com/articles/which-ux-research-methods/
Photo by Sora Shimazaki from Pexels
Purpose and goals of the test
User Study
Examples of methods:
• Observation
• Thinking Aloud
• Interviews and questionnaires
• Log file
Photo by Sora Shimazaki from Pexels
Video von Oleg Gamulinskiy / Pixabay
Use Cases
Gameplay Visualizations
Heatmaps are very popular since they
• are easy to create
• indicate the frequency of occurrence of a variable (e.g., death
locations of players) at a particular location
Examples: Colors show where players died most often in each map
http://www.wired.com/images/article/magazine/1509/ff_
halo5_350.jpg
Halo 3
http://www.polygon.com/2012/11/27/3682896/ho
w-call-of-duty-elite-evolved
Call of Duty Team Fortress 2
http://tf2wiki.net/wiki/File:Valve_stats_cp
_dustbowl_deaths.jpg
Gameplay Visualizations
Analysis of multiple variables (e.g., attackers versus victims) is
difficult with heatmaps
Attackers Victims Attackers + Victims
Gameplay Visualizations
Two visualization techniques were developed as suitable
alternatives to classic heatmaps that
• use clustering to group nearby data points
Enclosure Circle
Test Case
A (team-based) first-person shooter
• Heatmaps are commonly used in this
genre to communicate data about
gameplay back to the player
Team Fortress 2
• Popularity
• Log-files contain information about the
spatial coordinates of players for specific
events Image from http://www.hlportal.de/images/images/original/13682.jpg
Test Case
For the evaluation, we used data from the following two maps:
Gravel Pit (GP) Gold Rush(GR)
Test Case
We extracted
• players’ death locations and weapons which have been used
• locations where players were located when killing another player
The positions (attacker, victim) were represented as small
colored circular nodes
Sample
We selected participants with
• varying gaming experience
• different knowledge of Team Fortress 2
29 participants
• 23 males and 6 females between 22 and 34 years
• played digital games
• were familiar with FPS
Photo by Sora Shimazaki from Pexels
Tasks
Participants were asked to use the different
visualizations to analyze the provided data and to
report their observations
Participants had to find the area on the map where a
single variable occurred most frequently
Participants had to find the areas on the map where
one variable occurred more frequently in relation to
other variables
Photo by fotografierende from Pexels
Methods
Thinking aloud
Observation
Questionnaire
• if the representation was suitable to solve the
task after each round
• which of the three representations they
preferred
• for which tasks the different visualizations are
more suitable
• which representation caused greater
confidence with his/her solutions
• advantages as well as disadvantagesPhoto by fotografierende from Pexels
Sample & Procedure
5 Experts from the game industry
Duration: 2-hour sessions
Introduction about the purpose of the study
They were asked to sign a consent form informing
them about which data is collected and how it will
be used
They played the game for ten minutes to allow them
to familiarize themselves with the game
After that, the visualization system was introduced
to explain the functionality and navigation
Solved tasks with focus on movement, gaze,
emotion, and their combination
Photo by Sora Shimazaki from Pexels
Kepplinger, D. Wallner, G., Kriglstein, S., and Lankes, M.: See, Feel, Move - Player Behaviour Analysis through Combined Visualization of Gaze, Emotions, and Movement. Proc. of
the SIGCHI Conference on Human Factors in Computing Systems, ACM, 2020
Sample & Procedure
Participants were asked to rate the helpfulness of
the tool for the given task on a 5-point Likert-like
scale anchored by not helpful (1) and very helpful (5)
After they finished they were asked, e.g.,
• how helpful the tool was for analyzing gameplay
data
• which features they liked the most
• which aspects of the tool they think need further
refinement
Photo by Sora Shimazaki from Pexels
Kepplinger, D. Wallner, G., Kriglstein, S., and Lankes, M.: See, Feel, Move - Player Behaviour Analysis through Combined Visualization of Gaze, Emotions, and Movement. Proc. of
the SIGCHI Conference on Human Factors in Computing Systems, ACM, 2020
Motivation
• Node-link diagrams and matrix representations are two popular ways to visualize graphs
Images from Wallner, G., & Kriglstein, S. (2014).
PLATO: A visual analytics system for gameplay
data. Computers & Graphics, 38, 341-356.
Node-link diagrams …
• allow users to directly
see the connections
between the nodes as
lines
• can easily lead to
clutter
Matrix representations …
• use space more
efficiently and do not
lead to clutter
• detection of
relationships is not as
intuitive
Often hybrid visualizations are used to combine the
strengths of both representations
Analyzing and interpreting the same data from
different viewpoints can be challenging in order to
see the connections between them
Goal is …
• to get more detailed understanding of the
conceptual problems which may occur when
people have to convert node-link diagrams to
matrices and vice versa
Photo by fotografierende from Pexels
Research Questions
Correctness and Efficiency
• Do participants convert node-link diagrams to
matrix or vice versa more correctly?
Interpretation
• How do participants convert node-link
diagrams to matrices and vice versa?
• How do they transform the connections
between those two representations?
• How do they transform the nodes from a
node-link diagram to a matrix and vice versa?
Photo by fotografierende from Pexels
Method
Hand drawn sketches are an effective method
• offer creative freedom
• help to express, develop, and communicate
concepts with a low entry
• avoid any restrictions or influences resulting from
any software
Materials
2 test cases with the following 4 types of examples were generated
Sample & Procedure
The experimental session was conducted in a quiet
seminar room
The average duration was about 47 minutes
12 participants (average age of 25.16) with at least
basic knowledge with matrix and node-link
visualizations
Photo by Sora Shimazaki from Pexels
Kriglstein, S., Pohl, M., & Haider, J. D. (2018, June). How Users Transform Node-Link Diagrams to Matrices and Vice Versa. In International Conference on Theory and Application of
Diagrams (pp. 526-534). Springer, Cham.
Sample & Procedure
A short introduction and participants had the
possibility to ask questions to clarify any issues with
the study
To counteract learning effects the participants were
divided into two groups
• One group got the four matrices from one test
case and had to draw the corresponding node-link
diagrams
• After they finished they got the four node-link
diagrams from the other test case and had to draw
the corresponding matrices
• For the other group, the order was reversed
Participants were asked if they found it difficult to
convert matrices into the corresponding node-link
diagram and vice versa
Photo by Sora Shimazaki from Pexels
Kriglstein, S., Pohl, M., & Haider, J. D. (2018, June). How Users Transform Node-Link Diagrams to Matrices and Vice Versa. In International Conference on Theory and Application of
Diagrams (pp. 526-534). Springer, Cham.
User-Centered Evaluation of Visualization
Video from MD SHAHRIAR ISLAM / Pixabay
Simone Kriglstein
kriglstein@mail.muni.cz