26/03/2017
1
PA201
Virtual Environments
Lecture 6
Virtual Reality Interaction
Fotis Liarokapis
liarokap@fi.muni.cz
27th March 2017
Introduction
• How should users interact with the virtual
world?
• How should they move about?
• How can they grab and place objects?
• How should they interact with representations
of each other?
• How should they interact with files or the
Internet?
http://vr.cs.uiuc.edu/
Universal Simulation Principle
• Any interaction mechanism
from the real world can be
simulated in VR
– For example, the user might
open a door by turning a knob
and pulling
– As another example, the user
operate a virtual aircraft by
sitting in a mock-up cockpit
– One could even simulate
putting on a VR headset,
leading to an experience that is
comparable to a dream within
a dream!
http://vr.cs.uiuc.edu/
A flight simulator used by the US Air Force (photo by
Javier Garcia)
Terminology
• Interaction Technique (IT)
– Method for accomplishing a task
• 3D application
– System that displays 3D information
• 3D interaction
– Performing user actions in three dimensions
http://courses.cs.vt.edu/cs5754/lectures/interaction_part1.pdf
Relationships
http://courses.cs.vt.edu/cs5754/lectures/interaction_part1.pdf
Virtual-SAP Video Example
https://www.youtube.com/watch?v=Xz_J0EK8LLs
26/03/2017
2
Universal Interaction Tasks
• Navigation
– Travel
• Motor component (see later on)
– Wayfinding
• Cognitive component
• Selection
• Manipulation
• System control
• Symbolic input
http://courses.cs.vt.edu/cs5754/lectures/interaction_part1.pdf
Selection & Manipulation
• Selection
– Specifying one or more objects from a set
• Manipulation
– Modifying object properties
• i.e. position, orientation, scale, shape, color, texture,
behavior, etc.
http://courses.cs.vt.edu/cs5754/lectures/interaction_part1.pdf
Goals of Selection
• Indicate action on object
• Query object
• Make object active
• Travel to object location
• Set up manipulation
http://courses.cs.vt.edu/cs5754/lectures/interaction_part1.pdf
Selection Performance
• Variables affecting user performance
– Object distance from user
– Object size
– Density of objects in area
– Occluders
http://courses.cs.vt.edu/cs5754/lectures/interaction_part1.pdf
Common Selection Techniques
• Touching with virtual hand
• Ray/cone casting
• Occlusion / framing
• Naming
• Indirect selection
http://courses.cs.vt.edu/cs5754/lectures/interaction_part1.pdf
Go-Go Interaction Technique
• The Go-Go immersive interaction technique
uses the metaphor of interactively growing
the user’s arm and non-linear mapping for
reaching and manipulating distant objects
– Unlike others, this technique allows for seamless
direct manipulation of both nearby objects and
those at a distance
http://www.ivanpoupyrev.com/e-library/1998_1996/uist96.pdf
26/03/2017
3
Go-Go Description
• Having an arm to grow at will is a compelling
prospect
• However, implementing this metaphor in VR
poses three major challenges:
– How to enable users to tell the system when they
want to expand their virtual arm
– How users can control their virtual arm length
– How to make the implementation of this
metaphor intuitive, seamless and easy to use
http://www.ivanpoupyrev.com/e-library/1998_1996/uist96.pdf
Mapping Go-Go
• A mapping function F divides the space around
the user into two parts:
– Linear mapping
– Non-linear mapping
http://www.ivanpoupyrev.com/e-library/1998_1996/uist96.pdf
Mapping function F (k = 1/6) consists of linear
(Rr < D) and non-linear parts (Rr > D)
Position of real hand is defined by vector Rr ; position
of virtual hand is defined by vector Rv
Go-Go Examples
With traditional VR manipulation techniques the user
cannot reach the vase; reach is limited by arm length
The Go-Go technique allows users to expand their
reach. The white cube shows the real hand position
http://www.ivanpoupyrev.com/e-library/1998_1996/uist96.pdf
Selection Classification
http://courses.cs.vt.edu/cs5754/lectures/interaction_part1.pdf
Implementation Issues for Selection
Techniques
• How to indicate selection event
• Object intersections
• Feedback
– Graphical
– Aural
– Tactile
• Virtual hand avatar
• List of selectable objects
http://courses.cs.vt.edu/cs5754/lectures/interaction_part1.pdf
Goals of Manipulation
• Object placement
– Design
– Layout
– Grouping
• Tool usage
• Travel
http://courses.cs.vt.edu/cs5754/lectures/interaction_part1.pdf
26/03/2017
4
Manipulation Metaphors
• Simple virtual hand
– Natural but limited
• Ray casting
– Little effort required
– Exact positioning and orienting very difficult
• Lever arm effect
• Hand position mapping
– Natural, easy placement
– Limited reach, fatiguing, overshoot
• Indirect depth mapping
– Infinite reach, not tiring
– Not natural, separates DOFs
http://courses.cs.vt.edu/cs5754/lectures/interaction_part1.pdf
HOMER Technique
• HOMER (Hand-Centered,
Object, Manipulation,
Extending, Ray-Casting)
– Select: ray-casting
– Manipulate: hand
http://courses.cs.vt.edu/cs5754/lectures/interaction_part1.pdf
HOMER Metaphors
• HOMER (ray-casting + arm-extension)
– Easy selection & manipulation
– Expressive over range of distances
– Hard to move objects away from you
http://courses.cs.vt.edu/cs5754/lectures/interaction_part1.pdf
Motor Programs and
Remapping
Motor Programs
• Throughout our lives, we develop fine motor skills to
accomplish many specific tasks
– i.e. writing text, tying shoelaces, throwing a ball, and riding a
bicycle
• These are often called motor programs
– Learned through repetitive trials, with gradual improvements
in precision and ease as the amount of practice increases
• Eventually, we produce the motions without even having
to pay attention to them
– Most people can drive a car without paying attention to
particular operations of the steering wheel, brakes, and
accelerator
• The same applies to user interfaces
– Some devices are easier to learn than others
http://vr.cs.uiuc.edu/
Design Considerations
• Effectiveness for the task
– In terms of achieving the required speed, accuracy, and
motion range (if applicable)
• Difficulty of learning the new motor programs
– Ideally, the user should not be expected to spend many
months mastering a new mechanism
• Ease of use in terms of cognitive load
– The interaction mechanism should require little or no
focused attention after some practice
• Overall comfort during use over extended periods
– The user should not develop muscle fatigue, unless the
task is to get some physical exercise
http://vr.cs.uiuc.edu/
26/03/2017
5
Design Considerations .
• To design and evaluate new interaction
mechanisms, it is helpful to start by
understanding the physiology and psychology of
acquiring the motor skills and programs
• Must consider the corresponding parts for
generating output in the form of body motions in
the physical world
– In this case, the brain sends motor signals to the
muscles, causing them to move, while at the same
time incorporating sensory feedback by utilizing the
perceptual processes
http://vr.cs.uiuc.edu/
Design Considerations ..
(a) Part of the cerebral cortex is devoted to motion. (b) Many other parts interact with the cortex to produce and execute
motions, including the thalamus, spinal cord, basal ganglion, brain stem, and cerebellum. (Figures provided by The Brain from
Top to Bottom, McGill University
http://vr.cs.uiuc.edu/
Neurophysiology of Movement
• Consider the neural hardware
involved in learning, control, and
execution of voluntary movements
• Some parts of the cerebral cortex
are devoted to motion
– The primary motor cortex is the main
source of neural signals that control
movement
– The premotor cortex and
supplementary motor area appear to
be involved in the preparation and
planning of movement
http://vr.cs.uiuc.edu/
Atari 2600 Paddle controller
Neurophysiology of Movement .
• Many more parts are involved in
motion and communicate
through neural signals
• The most interesting part is the
cerebellum, meaning ``little
brain'‘
– Located at the back of the skull
• It seems to be a special
processing unit that is mostly
devoted to motion, but is also
involved in functions such as
attention and language
http://vr.cs.uiuc.edu/
The Atari Breakout game, in which the
bottom line segment is a virtual paddle that
allows the ball to bounce to the top and
eliminate bricks upon contacts
Neurophysiology of Movement ..
• Damage to the cerebellum has been widely seen to
affect fine motor control and learning of new motor
programs
– It has been estimated to contain around 101 billion
neurons, which is far more than the entire cerebral
cortex, which contains around 20 billion
• Even though the cerebellum is much smaller, a
large number is achieved through smaller, densely
packed cells
• In addition to coordinating fine movements, it
appears to be the storage center for motor
programshttp://vr.cs.uiuc.edu/
Neurophysiology of Movement ...
• One of the most relevant uses of the cerebellum for
VR is in learning sensorimotor relationships, which
become encoded into a motor program
• All body motions involve some kind of sensory
feedback
– The most common example is hand-eyecoordination
• Developing a tight connection between motor
control signals and sensory and perceptual signals
is crucial to many tasks
– This is also widely known in engineered systems, in
which sensor-feedback and motor control are combined
in applications such as robotics and aircraft stabilization;
the subject that deals with this is called control systems
http://vr.cs.uiuc.edu/
26/03/2017
6
Neurophysiology of Movement ....
• One of the most important factors is how long it takes to learn
a motor program
– Great variation across humans
• A key concept is neuroplasticity
– Which is the potential of the brain to reorganize its neural structures
and form new pathways to adapt to new stimuli
• Toddlers (children 12 to 36 months old) have a high level of
neuroplasticity,
– Which becomes greatly reduced over time through the process of
synaptic pruning
• This causes healthy adults to have about half as many synapses
per neuron than a child of age two or three
– Unfortunately, the result is that adults have a harder time acquiring
new skills such as learning a new language or learning how to use a
complicated interface
http://vr.cs.uiuc.edu/
Learning Motor Programs
• Typical example is the
computer mouse
– The sensorimotor mapping
seems a bit complicated
– Young children seem to
immediately learn how to use
the mouse, whereas older adults
require some practice
Motor Programs for VR
• A perceptual experience is controlled by body movement
that is sensed through a hardware device
– Using the universal simulation principle, any of these and
more could be brought into a VR system
– The physical interaction part might be identical or it could be
simulated through another controller
– Using the tracking methods the position and orientation of
body parts could be reliably estimated and brought into VR
– For the case of head tracking, it is essential to accurately
maintain the viewpoint with high accuracy and zero effective
latency
• Otherwise, the VR experience is significantly degraded
• The motion of the sense organ must be matched by a
tracking system
http://vr.cs.uiuc.edu/
Remapping
• For the motions of other body parts, this perfect matching
is not critical
• Our neural systems can instead learn associations that are
preferable in terms of comfort
– In the same way as the mouse, and keyboard work in the real
world
• Thus, we want to do remapping, which involves learning a
sensorimotor mapping that produces different results in a
virtual world than one would expect from the real world
– The keyboard example above is one of the most common
examples of remapping
• The process of pushing a pencil across paper to produce a letter has
been replaced by pressing a key
• The term remapping is even used with keyboards to mean the
assignment of one or more keys to another key
http://vr.cs.uiuc.edu/
Remapping .
• Remapping is natural for VR
– For example, rather than reaching out to grab a
virtual door knob, one could press a button to open
the door
– For a simpler case, consider holding a controller for
which the pose is tracked through space, as allowed
by the HTC Vive system
• A scaling parameter could be set so that one centimeter
of hand displacement in the real world corresponds to
two centimeters of displacement in the virtual world
• This is similar to the scaling parameter for the mouse
http://vr.cs.uiuc.edu/
Locomotion
26/03/2017
7
Introduction to Locomotion
• Suppose that the virtual world covers a much larger
area than the part of the real world that is tracked
– The matched zone is small relative to the virtual world
• Some form of interaction mechanism is needed to
move the user in the virtual world while she remains
fixed within the tracked area in the real world
• An interaction mechanism that moves the user in
this way is called locomotion
– It is as if the user is riding in a virtual vehicle that is
steered through the virtual world
http://vr.cs.uiuc.edu/
Locomotion Spectrum
• Moving from left to right, the amount of
viewpoint mismatch between real and virtual
motions increases
http://vr.cs.uiuc.edu/
Redirected Walking
• Redirected walking is a technique where a user
is tracked through a very large space, and it is
possible to make the user think that is walking in
straight lines for kilometers while in fact walking
in circles
– Walking along a straight line over long distances
without visual cues is virtually impossible for humans
(and robots!) because in the real world it is
impossible to achieve perfect symmetry
– One direction will tend to dominate through an
imbalance in motor strength and sensory signals,
causing people to travel in circleshttp://vr.cs.uiuc.edu/
Redirected Walking .
• Imagine a VR experience in which a virtual city contains
long, straight streets
– As the user walks down the street, the yaw direction of the
viewpoint can be gradually varied
– This represents a small amount of mismatch between the real
and virtual worlds, and it causes the user to walk along circular
arcs
• The main trouble with this technique is that the user has
free will and might decide to walk to the edge of the
matched zone in the real world, even if he cannot directly
perceive it
• In this case, an unfortunate, disruptive warning might
appear, suggesting that he must rotate to reset the yaw
orientation
http://vr.cs.uiuc.edu/
Locomotion Implementation
• Consider of sitting down
and wearing a headset
– Middle cases from
Locomotion Spectrum
• Locomotion can then be
simply achieved by moving
the viewpoint with a
controller
– Think of the matched zone
as a controllable cart that
moves across the ground of
the virtual environment
http://vr.cs.uiuc.edu/
Locomotion along a horizontal terrain can be modeled as
steering a cart through the virtual world. A top-down view is
shown. The yellow region is the matched zone, in which the
user's viewpoint is tracked. The values of xt, zt, and θ are
changed by using a controller
Locomotion Implementation .
• First consider the simple case in which the
ground is a horizontal plane
• Let Ttrack denote the homogeneous transform
that represents the tracked position and
orientation of the cyclopean (center) eye in
the physical world
http://vr.cs.uiuc.edu/
26/03/2017
8
Locomotion Implementation ..
• The position and orientation of the cart is
determined by a controller
• The homogeneous matrix:
– encodes the position (xt, zt) and orientation θ of the cart
(as a yaw rotation)
• The height is set at yt = 0 so that it does not change
the height determined by tracking or other systems
http://vr.cs.uiuc.edu/
Locomotion Implementation ...
• The eye transform is obtained by chaining Ttrack
and Tcart to obtain:
• To move the viewpoint for a fixed direction θ,
the xt and zt components are obtained by
integrating a differential equation:
• The variable s is the forward speed
http://vr.cs.uiuc.edu/
Locomotion Implementation ....
• Integrating over a time step Δt, the position update
appears as:
• The average human walking speed is about 1.4 meters
per second
– The virtual cart can be moved forward by pressing a button
or key that sets s = 1.4
– Another button can be used to assign s = -1.4, which would
result in backward motion
– If no key or button is held down, then s = 0, which causes
the cart to remain stopped
http://vr.cs.uiuc.edu/
Locomotion Implementation .....
• An alternative control scheme is to use the two
buttons to increase or decrease the speed, until
some maximum limit is reached
– In this case, motion is sustained without holding
down a key
• Keys could also be used to provide lateral
motion, in addition to forward/backward
motion
– This is called strafing in video games and should be
avoided (if possible)
http://vr.cs.uiuc.edu/
Changing Direction Issues
• Consider the orientation θ and to move in a
different direction, it needs to be reassigned
– The assignment could be made based on the user's
head yaw direction
– This becomes convenient and comfortable when the
user is sitting in a swivel chair and looking forward
• By rotating the swivel chair, the direction can be set
• However, this could become a problem for a wired headset
because the cable could wrap around the user
http://vr.cs.uiuc.edu/
Changing Direction Issues .
• In a fixed chair, it may become frustrating to control
θ because the comfortable head yaw range is limited
to only $ 60$ degrees in each direction
– In this case, buttons can be used to change θ by small
increments in clockwise or counterclockwise directions
• Unfortunately, changing θ according to constant
angular velocity causes yaw vection, which is
nauseating to many people
– Some users prefer to tap a button to instantly yaw about
10 degrees each time
– If the increments are too small, then vection appears
again, and if the increments are too large, then users
become confused about their orientation
http://vr.cs.uiuc.edu/
26/03/2017
9
Changing Direction Issues ..
• Another issue is where to locate
the center of rotation
– What happens when the user
moves his head away from the
center of the chair in the real
world?
– Should the center of rotation be
about the original head center or
the new head center?
• If it is chosen as the original
center, then the user will
perceive a large translation as θ is
changed
– However, this would also happen
in the real world if the user were
leaning over while riding in a cart
On the right the yaw rotation axis is centered on the head, for a
user who is upright in the chair. On the left, the user is leaning
over in the chair. Should the rotation axis remain fixed, or move
with the user?
http://vr.cs.uiuc.edu/
Vection Reduction Strategies
• The main problem with locomotion is vection,
which leads to VR sickness
– Six different kinds of vection occur, one for each DOF
– Numerous factors affect the sensitivity to vection
• Reducing the intensity of these factors should
reduce vection
– And hopefully VR sickness
• Several strategies for reducing vection-based VR
sickness exist
http://vr.cs.uiuc.edu/
Vection Reduction Strategies .
• 1) If the field of view for the optical flow is reduced,
then the vection is weakened
– A common example is to make a cockpit or car interior
that blocks most of the optical flow
• 2) If the viewpoint is too close to the ground, then
the magnitudes of velocity and acceleration vectors
of moving features are higher
– This is why you might feel as if you are traveling faster in
a small car that is low to the ground in comparison to
riding at the same speed in a truck or minivan
http://vr.cs.uiuc.edu/
Vection Reduction Strategies ..
• 3) Surprisingly, a larger mismatch for a short
period of time may be preferable to a smaller
mismatch over a long period of time
http://vr.cs.uiuc.edu/
(a) Applying constant acceleration over a time interval to bring
the stopped avatar up to a speed limit. The upper plot shows the
speed over time. The lower plot shows the acceleration. The
interval of time over which there is nonzero acceleration
corresponds to a mismatchwith the vestibular sense. (b) In this
case, an acceleration impulse is applied, resulting in the desired
speed limit being immediately achieved. In this case, the
mismatchoccurs over a time interval that is effectively zero
length. In practice, the perceived speed changes in a single pair
of consecutive frames. Surprisingly, case (b) is much more
comfortable than (a). It seems the brain prefers an outlier for a
very short time interval, as supposed to a smaller, sustained
mismatchover a longer time interval (such as 5 seconds)
Vection Reduction Strategies ...
• 4) Having high spatial frequency will yield more
features for the human vision system to track
– Therefore, if the passing environment is smoother, with
less detail, then vection should be reduced
• 5) Reducing contrast
– Such as making the world seem hazy or foggy while
accelerating, may help.
• 6) Providing other sensory cues such as blowing
wind or moving audio sources might provide
stronger evidence of motion
– Including vestibular stimulation in the form of a rumble
or vibration may also help lower the confidence of the
vestibular signal
http://vr.cs.uiuc.edu/
Vection Reduction Strategies ....
• 7) If the world is supposed to be moving, rather than
the user, then making it clear through cues or special
instructions can help
• 8) Providing specific tasks, such as firing a laser at
flying insects, may provide enough distraction from
the vestibular conflict
– If the user is instead focused entirely on the motion, then
she might become sick more quickly
• 9) The adverse effects of vection may decrease
through repeated practice
– People who regularly play FPS games in front of a large
screen already seem to have reduced sensitivity to
vection in VR
http://vr.cs.uiuc.edu/
26/03/2017
10
Non-planar Locomotion
• Consider more complicated locomotion cases
– If the user is walking over a terrain, then the y
component can be simply increased or decreased to
reflect the change in altitude
• In the case of moving through a 3D medium, all six
forms of vection become enabled
– Common settings include a virtual spacecraft, aircraft, or
scuba diver
– Yaw, pitch, and roll vection can be easily generated.
• Adding special effects that move the viewpoint will
cause further difficulty with vection
– For example, making an avatar jump up and down will
cause vertical vection
http://vr.cs.uiuc.edu/
Specialized Hardware
• Many kinds of hardware have been developed to
support locomotion
– One of the oldest examples is to create an entire
cockpit for aircraft flight simulation
http://vr.cs.uiuc.edu/
(a) An omnidirectional treadmill used in a CAVE system by the US Army for training. (b) A home-brew bicycle
riding system connected to a VR headset, developed by Paul Dyan
Teleportation
• In VR we move in ways that are physical implausible
– The user is immediately transported to another location.
• How is the desired location determined?
• One simple mechanism is a virtual laser pointer (or
3D mouse)
– which is accomplished by the user holding a controller
that is similar in shape to a laser pointer in the real world
• A smart phone could even be used
– The user rotates the controller to move a laser dot in the
virtual world
– This requires performing a ray casting operation to find
the nearest visible triangle, along the ray that
corresponds to the laser light
http://vr.cs.uiuc.edu/
Teleportation .
• To select a location where the
user would prefer to stand simply
point the virtual laser and press a
key to be instantly teleported
• To make pointing at the floor
easier, the beam could actually
be a parabolic arc that follows
gravity
– Places that are not visible can be
selected using: pop-up maps, textbased
searches or voice
commands
• One method, involves showing
the user a virtual small-scale
version of the environment
– This is effectively a 3D map
http://vr.cs.uiuc.edu/
A virtual ``laser pointer'' that follows a parabolic arc so that a
destination for teleportation can be easily specified as a
point on the floor. (Image from the Budget Cuts game on the
HTC Vive platform.)
Wayfinding
• The cognitive problem of learning a spatial representation and using it
to navigate is called wayfinding
• One trouble with locomotion systems that are not familiar in the real
world is that users might not learn the spatial arrangement of the world
around them
– Would your brain still form place cells for an environment in the real world
if you were able to teleport from place to place?
– We widely observe this phenomenon with people who learn to navigate a
city using only GPS or taxi services, rather than doing their own wayfinding
• The teleportation mechanism reduces vection, and therefore VR
sickness
– However, it may come at the cost of reduced learning of the spatial
arrangement of the environment
• When performing teleportation, it is important not to change the yaw
orientation of the viewpoint
– Otherwise, the user may become eve more disoriented
http://vr.cs.uiuc.edu/
Manipulation
26/03/2017
11
Introduction
• The virtual world does not
have to follow the complicated
physics of manipulation
– It is instead preferable to make
operations such as selecting,
grasping, manipulating,
carrying,and placing an object
as fast and easy as possible
– Extensive reaching or other
forms of muscle strain should
be avoided, unless the VR
experience is designed to
provide exercise
http://vr.cs.uiuc.edu/
Tom Cruise moving windows around on a holographic display
in the 2002 movie Minority Report. It is a great-looking
interaction mechanism for Hollywood, but it is terribly tiring
in reality. The user would quickly experience gorilla arms.
Avoid Gorilla Arms
• One of the most common misconceptions among
the public is that the interface used by Tom
Cruise in the movie Minority Report is desirable
– Previous slide
• In fact, it quickly leads to the well-known
problem of gorilla arms, in which the user quickly
feels fatigue from extended arms
• How long can you hold your arms directly in front
of yourself without becoming fatigued?
http://vr.cs.uiuc.edu/
Selection
• One of the simplest ways to select an object in the virtual
world is with the virtual laser pointer
– Several variations may help to improve the selection process
• With a pointer, the user simply illuminates the object of
interest and presses a button
– If the goal is to retrieve the object, then it can be immediately
placed in the user's virtual hand or inventory
– If the goal is to manipulate the object in a standard, repetitive way,
then pressing the button could cause a virtual motor program to be
executed
• If the object is hard to see, then the selection process may be
complicated
– It might be behind the user's head, which might require
uncomfortable turning
– The object could be so small or far away that it occupies only a few
pixels on the screen, making it difficult to precisely select it
http://vr.cs.uiuc.edu/
Manipulation
• If the user carries an object over a long distance, then it is
not necessary to squeeze or clutch the controller
– This would yield unnecessary fatigue
• In some cases, the user might be expected to carefully
inspect the object while having it in possession
– For example, he might want to move it around in his hand to
determine its 3D structure
– The object orientation could be set to follow exactly the 3D
orientation of a controller that the user holds
– The user could even hold a real object in hand that is tracked
by external cameras, but has a different appearance in the
virtual world
• This enables familiar force feedback to the user
http://vr.cs.uiuc.edu/
Placement
• Consider ungrasping the
object and placing it into
the world
• An easy case for the user is
to press a button and have
the object simply fall into
the right place
• This is accomplished by a
basin of attraction which is
an attractive potential
function defined in a
neighborhood of the target
pose (position and
orientation)
http://vr.cs.uiuc.edu/
To make life easier on the user, a basin of attraction can be
defined around an object so that when the basin in entered,
the dropped object is attracted directly to the target pose
Placement .
• Alternatively, the user may be required to
delicately place the object
• Perhaps the application involves stacking and
balancing objects as high as possible
• In this case, the precision requirements would
be very high
– Placing a burden on both the controller tracking
system and the user
http://vr.cs.uiuc.edu/
26/03/2017
12
Remapping
• The simplest case is the use of the button to select, grasp, and
place objects
– Instead of a button, continuous motions could be generated by the
user and tracked by systems
– Examples include turning a knob, moving a slider bar, moving a
finger over a touch screen, and moving a free-floating body through
space
• Recall that one of the most important aspects of remapping is
easy learnability
– Reducing the number of degrees of freedom that are remapped will
generally ease the learning process
• To avoid gorilla arms and related problems, a scaling factor
could be imposed on the tracked device so that a small amount
of position change in the controller corresponds to a large
motion in the virtual world
http://vr.cs.uiuc.edu/
Current Systems
• The development of interaction mechanisms for manipulation
remains one of the greatest challenges for VR
– Current generation consumer VR headsets either leverage existing
game controllers, as in the bundling of the XBox 360 controller with
the Oculus Rift in 2016, or introduce systems that assume large
hand motions are the norm, as in the HTC Vive headset controller
http://vr.cs.uiuc.edu/
(a) A pair of hand-held controllers that came with the HTC Vive headset in 2016; the device includes side
buttons, a trigger, and a touch pad for the thumb. (b) A user trying the controllers (prototype version)
Current Systems .
• Others are developing gesturing systems that involve no
hardware in the hands, as in the Leap Motion system
• Rapid evolution of methods and technologies for
manipulation can be expected in the coming years, with
increasing emphasis on user comfort and ease of use
http://vr.cs.uiuc.edu/
(a) The hand model used by Leap Motion tracking. (b) The tracked model superimposed in an image of the actual hands
Social Interaction
Introduction
• Communication and social interaction are vast subjects
• Social interaction in VR (or social VR) remains in a stage of
infancy, with substantial experimentation and rethinking
of paradigms occurring
• Connecting humans together is one of the greatest
potentials for VR technology
• Although it might seem isolating to put displays between
ourselves and the world around us, we can also be
brought closer together through successful interaction
mechanisms
• This section highlights several interesting issues with
regard to social interaction, rather than provide a
complete review
http://vr.cs.uiuc.edu/
Shannon-Weaver Model
• An important factor is how many people will be
interacting through the medium
– Start with a pair of people
• One of the most powerful mathematical models
ever developed is the Shannon-Weaver model of
communication
– Used in designing communication systems in engineering
http://vr.cs.uiuc.edu/
The classical Shannon-Weaver model of communication (from 1948). The sender provides a messageto the encoder, which transmits the
messagethrough a channel corrupted by noise. At the other end, a decoder converts the messageinto a suitable format for the receiver
26/03/2017
13
Beyond Shannon-Weaver Model
• This model is powerful in that it mathematically
quantifies human interaction
– But it is also inadequate for covering the kinds of
interactions that are possible in VR
• By once again following the universal simulation
principle, any kind of human interaction that exists
in the real world could be brought into VR
– Simple gestures and mannerisms can provide subtle but
important components of interaction that are not
captured by the classical communication model
http://vr.cs.uiuc.edu/
From Avatars to Visual Capture
• How should others see you in
VR?
– This is one of the most
intriguing questions becauseit
depends on both the social
context and on the
technological limitations
• Many possibilities exist!
• A user may represent himself
through an avatar
– Might not correspond at all to
his visible, audible, and
behavioral characteristics
http://vr.cs.uiuc.edu/
A collection of starter avatars offered by Second Life
From Avatars to Visual Capture .
• At the other extreme, a user
might be captured using
imaging technology and
reproduced in the virtual world
with a highly accurate 3D
representation
• In this case, it may seem as if
the person were teleported
directly from the real world to
the virtual world
– Many other possibilities exist
along this spectrum, and it is
worth considering the tradeoffs
http://vr.cs.uiuc.edu/
Holographic communication research from Microsoft in
2016. A 3D representation of a person is extracted in real
time and superimposed in the world, as seen through
augmented reality glasses (Hololens)
From Avatars to Visual Capture ..
• One major appeal of an avatar is anonymity
– Offers the chance to play a different role or exhibit different
personality traits in a social setting
• In a phenomenon called the Proteus effect, it has been
observed that a person's behavior changes based on the
virtual characteristics of the avatar
– Similar to the way in which people have been known to
behave differently when wearing a uniform or costume
• The user might want to live a fantasy, or try to see the
world from a different perspective
– i.e. people might develop a sense of empathy if they are able
to experience the world from an avatar that appears to be
different in terms of race, gender, height, weight, age, and so
on
http://vr.cs.uiuc.edu/
From Avatars to Visual Capture ...
• Users may also want to experiment with other forms
of embodiment
– For example, a group of children might want to inhabit
the bodies of animals while talking and moving about
– Imagine if you could have people perceive you as if you as
an alien, an insect, an automobile, or even as a talking
block of cheese
• People were delightfully surprised in 1986 when
Pixar brought a desk lamp to life in the animated
short Luxo Jr. Hollywood movies over the past
decades have been filled with animated characters
– And we have the opportunity to embody some of them
while inhabiting a virtual world!
http://vr.cs.uiuc.edu/
Moving Toward Physical Realism
• Based on the current technology, three major kinds
of similarities can be independently considered:
– Visual appearance
• How close does the avatar seem to the actual person in terms of
visible characteristics?
– Auditory appearance
• How much does the sound coming from the avatar match the
voice, language, and speech patterns of the person?
– Behavioral appearance
• How closely do the avatar's motions match the body language,
gait, facial expressions, and other motions of the person?
http://vr.cs.uiuc.edu/
26/03/2017
14
Visual Appearance
• The first kind of similarity could start to match the person
by making a kinematic model in the virtual world that
corresponds in size and mobility to the actual person
• Other simple matching such as hair color, skin tone, and
eye color could be performed
• To further improve realism, texture mapping could be
used to map skin and clothes onto the avatar
– i.e. a picture of the user's face could be texture mapped onto
the avatar face
• Highly accurate matching might also be made by
constructing synthetic models, or combining information
from both imaging and synthetic sources
http://vr.cs.uiuc.edu/
Visual Appearance Example
http://vr.cs.uiuc.edu/
The Digital Emily project from 2009: (a) A real person is imaged. (b) Geometric models are animated along with sophisticated rendering techniques
to produce realistic facial movement
Auditory Appearance
• For the auditory part, users of Second Life and similar
systems have preferred text messaging
– This interaction is treated as if they were talking aloud, in the
sense that text messages can only be seen by avatars that
would have been close enough to hear it at the same
distance in the real world
– Texting helps to ensure anonymity
• Recording and reproducing voice is simple in VR, making
it much simpler to match auditory appearance than
visual appearance
• One must take care to render the audio with proper
localization, so that it appears to others to be coming
from the mouth of the avatar
http://vr.cs.uiuc.edu/
Behavioral Appearance
• The behavioral experience could be matched perfectly,
– While the avatar has a completely different visual appearance
• This is the main motivation for motion capture systems, in
which the movements of a real actor are recorded and
then used to animate an avatar in a motion picture
– Note that movie production is usually a long, off-line process
– Accurate, real-time performance that perfectly matches the
visual and behavioral appearance of a person is currently
unattainable in low-cost VR systems
• Furthermore, capturing the user's face is difficult if part of
it is covered by a headset, although some recent progress
has been made in this area
http://vr.cs.uiuc.edu/
Behavioral Appearance .
• Current tracking systems
can be leveraged to
provide accurately
matched behavioral
appearance:
– i.e. head tracking can be
directly linked to the avatar
head so that others can
know where the head is
turned
– Users can also understand
head nods or gestures, such
as ``yes'' or ``no''
http://vr.cs.uiuc.edu/
Oculus Social Alpha, which was an application for Samsung
Gear VR. Multiple users could meet in a virtual world and
socialize. In this case, they are watching a movie together
in a theater. Their head movements are provided using
head tracking data. They are also able to talk to each
other with localized audio
From one-on-one to Societies
• Consider social interaction on different scales
• One important aspect of one-on-one
communication is whether the relationship between
the two people is symmetrical or complementary
– In a symmetrical relationship the two people are of equal
status
– In a complementary relationship one person is in a
superior position, as in the case of a boss and employee
or a parent and a child
• This greatly affects the style of interaction, particularly in a
targeted activity
http://vr.cs.uiuc.edu/
26/03/2017
15
Additional Interaction
Mechanisms
Media Interaction
• The content of the Internet
can be brought into VR in
numerous ways by following
the universal simulation
principle
– A web browser could appear
on a public display in the
virtual world or on any other
device that is familiar to
users in the real world
– Alternatively, a virtual
screen may float directly in
front of the user, while a
stable, familiar background
is provided
The Valve Steam game app store when viewed in the HTC
Vive headset
http://vr.cs.uiuc.edu/
Text Entry and Editing
• One option is to track a real keyboard and mouse, making
them visible VR
• Tracking of fingertips may also be needed to provide visual
feedback
• This enables a system to be developed that magically
transforms the desk and surrounding environment into
anything
• Much like the use of a background image on a desktop
system, a relaxing panoramic image or video could
envelop the user while she works
• For the actual work part, rather than having one screen in
front of the user, a number of screens or windows could
appear all around and at different depths
http://vr.cs.uiuc.edu/
3D Design and Visualization
• VR offers the ability to interact with and view 3D
versions of a design or data set
– This could be from the outside looking in, perhaps at
the design of a new kitchen utensil
– It could also be from the inside looking out, perhaps
at the design of a new kitchen
• Viewing a design in VR can be considered as a
kind of virtual prototyping, before a physical
prototype is constructed
– This enables rapid, low-cost advances in product
development cycles
http://vr.cs.uiuc.edu/
Review of Interaction
for VR
Taxonomy of Virtual Object
Manipulation Techniques
• Poupyrev, et al. (1999)
developed taxonomy of
the virtual interaction
techniques and performed
empirical evaluation of the
techniques
• Categorized the
techniques into “
– Egocentric
– Exocentric
http://ieomsociety.org/ieom2014/pdfs/360.pdf
Taxonomyof virtual object manipulation techniques
(Poupyrev, et al., 1998)
26/03/2017
16
Technique Classification by
Components
http://courses.cs.vt.edu/cs5754/lectures/interaction_part1.pdf
Taxonomy of Selection/Manipulation
Techniques
http://ieomsociety.org/ieom2014/pdfs/360.pdf
Conclusions
• Many more forms of interaction can be imagined, even by
just applying the universal simulation principle
– Video games have already provided many ideas for interaction
via a standard game controller
• Beyond that, the Nintendo Wii remote has been especially
effective in making virtual versions of sports activities such
as bowling a ball or swinging a tennis racket
• What new interaction mechanisms will be comfortable
and effective for VR?
• If displays are presented to senses other than vision, then
even more possibilities emerge
– i.e. could you give someone a meaningful hug on the other
side of the world if they are wearing a suit that applies the
appropriate forces to the body?
http://vr.cs.uiuc.edu/
Questions