13/03/2017
1
PA201
Virtual Environments
Lecture 4
Virtual Reality Displays
Fotis Liarokapis
liarokap@fi.muni.cz
13th March 2017
VR Immersion Levels
• Fully immersive VR applications
– Can’t experience the surrounding physical and real
environment
• Semi-immersive VR applications
– Certain degree of immersion is gained
• i.e. Stereo projection
• Low immersion
– 2D screen renderings of a conceptually 3D space
Standard vs. Stereoscopic Display
User
C
A
B
Screen
C
A
B
Standard Display Stereoscopic Display
Light & Optics
Introduction to Light
• Knowing how light propagates in the physical
world is crucial to understanding VR
• One reason is the interface between visual
displays and our eyes
• Light is emitted from displays and arrives on
our retinas in a way that convincingly
reproduces how light arrives through normal
vision in the physical world
http://vr.cs.uiuc.edu/
Introduction to Light .
• In the current generation of VR headsets, a
system of both engineered and natural lenses
(parts of our eyes) guide the light
• Another reason to study light propagation is
the construction of virtual worlds
http://vr.cs.uiuc.edu/
13/03/2017
2
Basic Behavior of Light
• Light can be described in three ways:
– Photons: Tiny particles of energy moving through space
at high speeds
• Helpful when considering the amount of light received by a
sensor or receptor
– Waves: Ripples through space that are similar to waves
propagating on the surface of water, but are 3D. The
wavelength is the distance between peaks
• Helpful when considering the spectrum of colors
– Rays: A ray traces the motion of a single hypothetical
photon. The direction is perpendicular to the wavefronts.
• Helpful when explaining lenses and defining the concept of
visibility
http://vr.cs.uiuc.edu/
Basic Behavior of Light
Waves and visibility rays emanating from a point light source
http://vr.cs.uiuc.edu/
Lenses
• Lenses have been made for
thousands of years, with the
oldest known artifact
– It was constructed before 700 BC
in Assyrian Nimrud
– Whether constructed from
transparent materials or from
polished surfaces that act as
mirrors, lenses bend rays of light
so that a focused image is
formed
http://vr.cs.uiuc.edu/
The earliest known artificially
constructed lens
Lenses .
• Over the centuries, their uses
have given rise to several wellknown
devices, such as
eyeglasses, telescopes,
magnifying glasses, binoculars,
cameras, and microscopes
– Optical engineering is therefore
filled with design patterns that
indicate how to optimize the
designs of these well-understood
devices
http://vr.cs.uiuc.edu/
Lenses ..
• VR headsets are unlike classical optical devices,
leading to many new challenges that are outside
of standard patterns that have existed for
centuries
– Thus, the lens design patterns for VR are still being
written
– The first step toward addressing the current
challenges is to understand how simple lenses work
• i.e. Snell's law
– Not covered in this lecture!
http://vr.cs.uiuc.edu/
The Human Eye
• A simplified view of the human eye as an
optical system
http://vr.cs.uiuc.edu/
13/03/2017
3
The Human Eye .
• A ray of light travels through five media before
hitting the retina
• The indices of refraction are indicated
– Considering Snell's law, the greatest bending occurs due
to the transition from air to the cornea
– Note that once the ray enters the eye, it passes through
only liquid or solid materials
http://vr.cs.uiuc.edu/
The Human Eye ..
http://vr.cs.uiuc.edu/
A closer object yields diverging rays,
but with a relaxed lens, the image is
blurry on the retina
Normal eye operation, with relaxed lens
Cameras
• People have built and
used cameras for
hundreds of years,
starting with a camera
obscura that allows light
to pass through a
pinhole and onto a
surface that contains
the real image
A pinhole camera that is recommended
for viewing a solar eclipse
(Figure from TimeAndDate.com)
http://vr.cs.uiuc.edu/
Cameras .
• Across the 19th century, various
chemically based technologies were
developed to etch the image
automatically from the photons
hitting the imaging surface
• Across the 20th century, film was in
widespread use, until digital
cameras avoided the etching
process altogether by electronically
capturing the image using a sensor
• Two popular technologies have
been a Charge-Coupled Device
(CCD) array and a CMOS active-pixel
image sensor
http://vr.cs.uiuc.edu/
A CMOS active-pixel image sensor
Cameras ..
• Such digital technologies
record the amount of light
hitting each pixel location
along the image, which
directly produces a captured
image
– The costs of these devices has
plummeted in recent years,
allowing hobbyists to buy a
camera module
http://vr.cs.uiuc.edu/
A low-cost CMOS camera module
Shutters
• Several practical issues arise when capturing digital
images
• The image is an 2D array of pixels, each of which
having red (R), green (G), and blue (B) values that
typically range from 0 to 255
• Consider the total amount of light energy that hits
the image plane
– For a higher-resolution camera, there will generally be
less photons per pixel because the pixels are smaller
• Each sensing element (one per color per pixel) can
be imagined as a bucket that collects photons,
much like drops of rain
http://vr.cs.uiuc.edu/
13/03/2017
4
Shutters .
• To control the amount of photons, a shutter blocks all the
light, opens for a fixed interval of time, and then closes
again
• For a long interval (low shutter speed), more light is
collected
– However, the drawbacks are that moving objects in the scene
will become blurry and that the sensing elements could
become saturated with too much light
• Photographers must strike a balance when determining
the shutter speed to account for the amount of light in
the scene, the sensitivity of the sensing elements, and the
motion of the camera and objects in the scene
http://vr.cs.uiuc.edu/
Shutters ..
• Also relating to shutters, CMOS
sensors unfortunately work by
sending out the image
information sequentially, line-by-
line
• The sensor is therefore coupled
with a rolling shutter, which
allows light to enter for each line,
just before the information is
sent
• This means that the capture is
not synchronized over the entire
image, which leads to odd
artifacts
The wings of a flying helicopter are
apparently bent backwards due
to the rolling shutter effect
http://vr.cs.uiuc.edu/
Shutters ...
• Image processing algorithms that work with rolling
shutters and motion typically transform the image
to correct for this problem
• CCD sensors grab and send the entire image at
once, resulting in a global shutter
• CCDs have historically been more expensive than
CMOS sensors, which resulted in widespread
appearance of rolling shutter cameras in
smartphones
– However, the cost of global shutter cameras is rapidly
decreasing
http://vr.cs.uiuc.edu/
Aperture
• The optical system also impacts the amount of
light that arrives to the sensor
• Using a pinhole light would fall onto the image
sensor
– But it would not be bright enough for most purposes
• Other than viewing a solar eclipse
• Therefore, a convex lens is used instead so that
multiple rays are converged to the same point in
the image plane
– This generates more photons per sensing element
http://vr.cs.uiuc.edu/
Aperture .
• The main drawback is that the lens sharply
focuses objects at a single depth, while blurring
others
• In the pinhole case, all depths are essentially ``in
focus'', but there might not be enough light
• Photographers therefore want to tune the optical
system to behave more like a pinhole or more like
a full lens, depending on the desired outcome
http://vr.cs.uiuc.edu/
Aperture ..
• The result is a controllable
aperture which appears
behind the lens and sets
the size of the hole
through which the light
rays enter
– A small radius mimics a
pinhole by blocking all but
the center of the lens
– A large radius allows light to
pass through the entire lens
A spectrum of aperture settings, which
control the amount of light that
enters the lens. The values shown are
called the focal ratio or f-stop
http://vr.cs.uiuc.edu/
13/03/2017
5
Physiology of Human
Vision
Physiology of the Human Eye
http://vr.cs.uiuc.edu/
This viewpoint shows how the right eye would appear if sliced horizontally (the nose would be to the left)
Eye Movements
• Eye rotations occur both
voluntarily and involuntarily
– Allows a person to fixate on features
in the world, even the head or target
features are moving
• A reason for eye movement is to
position the feature of interest on
the fovea
– Only the fovea can sense dense,
color images, and it unfortunately
spans a very narrow field of view
• To gain a coherent, detailed view of
a large object, the eyes rapidly scan
over it while fixating on points of
interest
http://vr.cs.uiuc.edu/
The trace of scanning a face using saccades
Eye Movements .
• Another reason for eye movement is that our
photoreceptors are slow to respond to stimuli due to
their chemical nature
– They take up to 10ms to fully respond to stimuli and produce
a response for up to 100ms
• Eye movements help keep the image fixed on the same
set of photoreceptors so that they can fully charge
– This is similar to the image blurring problem that occurs in
cameras at low light levels and slow shutter speeds
• Additional reasons for eye movement are to maintain a
stereoscopic view and to prevent adaptation to a
constant stimulation
– It has been shown experimentally that when eye motions are
completely suppressed, visual perception disappears
completelyhttp://vr.cs.uiuc.edu/
Eye Movements ..
• As movements combine to build a coherent
view, it is difficult for scientists to predict and
explain how people interpret some stimuli
– i.e. Optical illusions
• Next slide
http://vr.cs.uiuc.edu/
Eye Movements ...
http://vr.cs.uiuc.edu/
13/03/2017
6
Eye Muscles
The six muscle tendons attach to the eye so that yaw, pitch, and a small amount of roll become possible
There are six muscles per eye, each of which is capable of pulling the pupil toward its location
http://vr.cs.uiuc.edu/
Quality of the VR Visual Display
• Three crucial factors for the display are:
– Spatial resolution
• How many pixels per square area are needed?
– Intensity resolution and range
• How many intensity values can be produced, and what
are the minimum and maximum intensity values?
– Temporal resolution
• How fast do displays need to change their pixels?
http://vr.cs.uiuc.edu/
HMDs
Introduction to HMDs
• Different categories exist
• Worn on the head or as part
of a helmet
• HMDs have a small display
optic in front of eye(s)
– Monocular HMD
– Binocular HMD
VR Technical Approaches
• Head-Mounted
– Head-mounted wide-view stereo display
• Cave-based
– Walls of a room are rear-projection stereo displays
– The user wears goggles to enable viewing in 3D
• Other-type
– Hand held, or hand moved, display
– Position and orientation are tracked
HMD Depth Perception
• Depth perception requires different images for
the left and right eyes
– Not all HMDs provide depth perception
– Some lower-end modules are essentially bi-ocular
devices where both eyes are presented with the
same image
https://en.wikipedia.org/wiki/Head-mounted_display
13/03/2017
7
HMD Depth Perception .
• Multiple configurations exist:
– Dual video inputs
• Providing a completely separate video signal to each eye
– Time-based multiplexing
• Methods such as frame sequential combine two separate
video signals into one signal by alternating the left and
right images in successive frames
– Side by side or top-bottom multiplexing
• This method allocated half of the image to the left eye and
the other half of the image to the right eye
https://en.wikipedia.org/wiki/Head-mounted_display
HMD Performance Parameters
• Ability to show stereoscopic imagery
– A binocular HMD has the potential to display a
different image to each eye
– This can be used to show stereoscopic images
– This is the distance at which, given the average
human eye rangefinder "baseline" (distance between
the eyes or Interpupillary distance (IPD)) of between
2.5 and 3 inches (6 and 8 cm), the angle of an object
at that distance becomes essentially the same from
each eye
https://en.wikipedia.org/wiki/Head-mounted_display
HMD Performance Parameters .
• Interpupillary distance (IPD)
– This is the distance between the two eyes, measured at the
pupils, and is important in designing HMDs
• Field of view (FOV)
– Humans have an FOV of around 180°, but most HMDs offer far
less
– Typically,a greater field of view results in a greater sense of
immersion and better situational awareness
– Most people do not have a good feel for what a particular
quoted FOV would look like (e.g., 25°) so often manufacturers
will quote an apparent screen size
– Consumer-level HMDs typically offer a FOV of about 30-40°
whereas professional HMDs offer a field of view of 60° to 150°.
https://en.wikipedia.org/wiki/Head-mounted_display
HMD Performance Parameters ..
• Resolution
– HMDs usually mention either the total number of pixels
or the number of pixels per degree
– Pixel density is also used to determine visual acuity
– HMDs typically offer 10 to 20 pixels/°
• Binocular overlap
– Measures the area that is common to both eyes
– Binocular overlap is the basis for the sense of depth and
stereo, allowing humans to sense which objects are near
and which objects are far
• Humans have a binocular overlap of about 100° (50° to the left
of the nose and 50° to the right)
• The larger the binocular overlap offered by an HMD, the greater
the sense of stereo
https://en.wikipedia.org/wiki/Head-mounted_display
HMD Performance Parameters ...
• Distant focus (collimation)
– Optical methods may be used to present the images
at a distant focus
• Seems to improve the realism of images that in the real
world would be at a distance
• On-board processing and operating system
– Some HMDs offer on-board OS (i.e. Android) allowing
applications to run locally on the HMD
• Eliminating the need to be tethered to an external device
to generate video
– These are sometimes referred to as smart goggles
https://en.wikipedia.org/wiki/Head-mounted_display
HMD Peripherals
• The most basic HMDs simply project an image
on a wearer’s visor (or reticle)
– The image is not slaved to the real world
• i.e. the image does not change based on the wearer’s head
position
• More sophisticated HMDs incorporate a
positioning system that tracks the wearer’s head
position and angle
– So that the picture or symbology displayed is
congruent with the outside world using see-through
imagery
https://en.wikipedia.org/wiki/Head-mounted_display
13/03/2017
8
HMD Peripherals .
• Head tracking
– Slaving the imagery
• May also be used with tracking sensors that detect
changes of angle and orientation
• Eye tracking
– Measure the point of gaze, allowing a computer to
sense where the user is looking
• Hand tracking
– Tracking hand movement from the perspective of the
HMD allows natural interaction with content and a
convenient game-play mechanism
https://en.wikipedia.org/wiki/Head-mounted_display
Characteristics of HMDs
• Immersive
– You are inside the computer world
– Can interact with real world (mouse, keyboard,
people)
• Ergonomics
• Resolution and field of view
• Tethered
Modern HMDs Video Head-Mounted Display
• Video head-mounted displays accept video
from a camera and mix it electronically with
computer graphics
– Easier to perform registration and calibration
– Watch a digital representation of the world
• Most popular method until now for AR
TriVisio
• Stereo video input
– PAL resolution cameras
• 2 x SVGA displays
– 30 degree FOV
– User adjustable convergence
• $6,000 USD
http://www.trivisio.com/
Vuzix Display - Wrap 1200DXAR
• 4th generation
• 3 DOF head tracker
• Stereoscopic 3D video
• 16:9 or 4:3 aspect ratio
• 1920 x 1080 resolution
• Weighs less than three ounces
https://www.vuzix.com/
13/03/2017
9
Video See Through Example
https://www.youtube.com/watch?v=nEGtha40PUo
Optical Head-Mounted Display
• Nowadays, see-through displays are lightweight
with high-resolution optical devices
• However, certain inefficiencies remain such as
sufficient:
– Brightness
– Resolution
– Field of view
– Contrast
Optical Head-Mounted Display
• Various techniques have existed for see-through
HMDs and can be summarized into two main
families:
– “Curved Mirror” (or Curved Combiner) based
– “Waveguide” or "Light-guide" based
• The curved mirror technique has been used by
Vuzix in their Star 1200 product, by Olympus, and
by Laster Technologies
• Various waveguide techniques have existed for
some time
– These techniques include diffraction optics, holographic
optics, polarized optics, and reflective optics
https://en.wikipedia.org/wiki/Optical_head-mounted_display
Waveguide Techniques
• Diffractive waveguide
– Slanted diffraction grating elements (nanometric 10E-9)
• Nokia technique now licensed to Vuzix
• Holographic waveguide
– 3 holographic optical elements (HOE) sandwiched
together (RGB)
• Used by Sony and Konica Minolta
• Polarized waveguide
– 6 multilayer coated (25–35) polarized reflectors in glass
sandwich
• Developed by Lumus
https://en.wikipedia.org/wiki/Optical_head-mounted_display
Waveguide Techniques .
• Reflective waveguide
– Thick light guide with single semi reflective mirror
• This technique is used by Epson in their Moverio product
• "Clear-Vu" reflective waveguide
– Thin monolithic molded plastic w/ surface reflectors
and conventional coatings
• Developed by Optinvent and used in their ORA product
• Switchable waveguide
– Developed by SBG Labs
https://en.wikipedia.org/wiki/Optical_head-mounted_display
Google Glass
• Google Glass is based on OHMD
technology
– Displays information in a
smartphone-likehands-free
format
– Wearers communicate with the
Internet via natural language
voice commands
• Available to the public on May
15 2014 for $1,500
– Stopped on January 15 2015
https://en.wikipedia.org/wiki/Google_Glass
13/03/2017
10
Google Glass Videos
• https://www.youtube.com/watch?v=4_X6Eyq
Xa2s
• https://www.youtube.com/watch?v=y3dGVe
W24tc
• https://www.youtube.com/watch?v=JSnB06u
m5r4
Innovega iOptik System
• It comprises a pair of contact lens
which refocus polarized light to
the pupil
• Allows the wearer to focus on an
image that is as near as 1.25 cm
to the eye
• Prototype features a field of view
of 60 degrees or more
– Aiming at 120 degrees FOV
• Designed for military use
– A consumer version coming soon
http://www.innovega-inc.com/index.php
Innovega iOptik System Video
http://www.innovega-inc.com/videos.php
Pinlight Displays
• Wide Field of View Augmented Reality
Eyeglasses using Defocused Point Light
Sources
• Instead of conventional optics
– LCD panel
– An array of point light sources
• Coding allows for miniature see-through
projectors
http://pinlights.info/
Pinlight Displays Video
http://pinlights.info/
Comparison of OHMDs Technologies
Combiner technology Size Eye box FOV Other Example
Flat combiner 45 degrees Thick Medium Medium Traditionaldesign Vuzix, Google Glass
Curved combiner Thick Large Large Classical bug-eye design
Many products (see
through and occlusion)
Phase conjugate material Thick Medium Medium Very bulky OdaLab
Buried Fresnel combiner Thin Large Medium Parasitic diffraction effects
The Technology
Partnership (TTP)
Cascaded prism/mirror
combiner
Variable Medium to Large Medium Louver effects Lumus, Optinvent
Free form TIR combiner Medium Large Medium Bulky glass combiner
Canon, Verizon & Kopin
(see through and
occlusion)
Diffractive combiner with
EPE
Very thin Very large Medium
Haze effects, parasitic
effects, difficult to
replicate
Nokia/ Vuzix
Holographic waveguide
combiner
Very thin Medium to Large in H Medium
Requires volume
holographic materials
Sony
Holographic light guide
combiner
Medium Small in V Medium
Requires volume
holographic materials
Konica Minolta
Combo diffuser/contact
lens
Thin (glasses) Very large Very large
Requires contact lens +
glasses
Innovega & EPFL
Tapered opaque light
guide
Medium Small Small Image can be relocated Olympus
https://en.wikipedia.org/wiki/Optical_head-mounted_display
13/03/2017
11
HMD Market
http://www.credenceresearch.com/report/head-mounted-display-hmd-market
CAVE
CAVE
• Computer Automatic Virtual Environment (CAVE)
– One of the most immersive environment
CAVE Characteristics
• Eyes and hands are
linked and mobile
• Display is fixed
• Shadow effect problem
– When another person
hides walls, or when an
object is supposed to be
between two persons
CAVE Examples CAVE Videos
• https://www.youtube.com/watch?v=STMcWU
tQr1Y
• https://www.youtube.com/watch?v=j59Jxfbvx
Gg
• https://www.youtube.com/watch?v=tlBOr524
_18
13/03/2017
12
Large Screen Projection
• Larger field-of-view than HMD
• Field of regard is smaller than HMD but larger
than a typical CRT Display
• Projectors must be aligned properly
Large Screen Projection Architectures
• Front Projection
– User may be in the way
• Back Projection
– Takes up even more space
• When multiple screens are arranged at or
near 90 degree angles, reflection between
screens may be a problem
Large Screen Projection Pros
• Viewer not isolated from real objects or other
people in the virtual world space
• Less physical gear to wear than HMD
• Potentially better resolution than HMD
• Large field of view compared to HMD or CRT
Large Screen Projection Cons
• Usually one person is head-tracked
• Real objects may occlude virtual objects in
inappropriate ways
• Multiple screens require more computation
• At least one direction is not part of the virtual
world
• Lighting
Large Screens Videos
• https://www.youtube.com/watch?v=VjgqgRThnC
8
• https://www.youtube.com/watch?v=oNAl0RGBL
CA
• https://www.youtube.com/watch?v=RspO9rImJu
w
Other Displays
13/03/2017
13
Small Area Displays
• Small area displays are portable
and thus be suitable for many VR
applications
• The major disadvantages of these
displays are the limited working
area and resolution
– Getting better!
• Small area displays have also
illumination problems
Hand Held VR
• Hand-held computing devices are ubiquitous
– Have become part of our lives
– Also increasingly being equipped with special
sensors and non-traditional displays
Typical Characteristics
• Hands and display are tightly
coupled
– All three are mobile
• Problems
– All of HMD and CAVE systems
Pico Projectors
• Extra small projectors
– Microvision, 3M, Samsung, Philips
http://www.mvis.com/
Head Mounted Projector
• Retro-reflective Material
• Potentially portable
• NVIS P-50 HMPD
– 1280x1024/eye
– Stereoscopic
– 50 degree FOV
http://www.nvis.com/
HMD vs HMP
Head Mounted Display Head Mounted Projector
13/03/2017
14
Conclusions
• Many technologies exist
– Depending on the application different solution
• HMDs are dominating at the moment
– Becoming cheaper
– More immersive
Questions