10/9/2016
1
PA198
Augmented Reality Interfaces
Lecture 4
Augmented Reality Displays
Fotis Liarokapis
liarokap@fi.muni.cz
10th October 2016
The Eye
Basic Eye
Cornea
Crystalline
Lens
Fovea
Retina
Optic
Nerve
Sherman & Craig, pp. 151-159
The Eye
• Accommodation describes the
altering of the curvature of the
crystalline lens by means of the
ciliary muscles
– Expressed in diopters
• Retina is the sensory membrane
that lines the back of the eye
and receives the image formed
by the lens of the eye
• Fovea is the part of the human
retina that possesses the best
spatial resolution or visual acuity
Cornea
Crystalline
Lens
Fovea
Retina
Optic
Nerve
Sherman & Craig, pp. 151-159
Properties of the Eye
• Approximate Field of View
– 120 degrees vertical
– 150 degrees horizontal (one eye)
– 200 degrees horizontal (both eyes)
• Acuity
– 30 cycles per degree (20/20 Snellen acuity)
Sherman & Craig, pp. 151-159
Field of View (FOV)
• The FOV the user can achieve at a given eye location
limited by vignetting of off axis field angles
– Vignet: reduction of an image's brightness or saturation
at the periphery compared to the image center
• This will be limited by the eye relief and the FOV of
the system
10/9/2016
2
FOV limitations
• Eye must rotate to view off axis field angles
– Eye point of rotation located 10mm behind pupil
– Will cause translation of pupil
• As the eye translates out of Eye Box the user will move
eye closer to the system
– Effectively decreasing the Eye Relief
Monocular Field-of-View
• FOV may be measured
horizontally, vertically or
diagonally
• Monocular FOV is the angular
substance of the displayed
image as measured from the
pupil of one eye
– Usually expressed in degrees
Binocular Field-of-View
• Total FOV is the total angular size
of the displayed image visible to
both eyes
• Binocular FOV refers to the part of
the displayed image visible to both
eyes
– Also known as stereoscopic FOV
Focal Length
• The focal length of an optical
system is a measure of how
strongly the system converges
or diverges light
• Represents the distance from
the surface of a lens (or mirror)
at which rays of light converge
The focal point F and focal length f of a
positive (convex) lens, a negative (concave)
lens, a concave mirror, and a convex mirrorhttps://en.wikipedia.org/wiki/Focal_length
Optics Characteristics
Diopter
• The power of a lens is measured in diopters
– Where the number of diopters is equal to:
• 1/(focal length of the lens measured in meters)
• The main benefits over focal length
– The lensmaker's equation has the object distance,
image distance, and focal length all as reciprocals
– When relatively thin lenses are placed close
together their powers approximately add
https://en.wikipedia.org/wiki/Dioptre
10/9/2016
3
Contrast Transfer Function (CTF)
• CTF: measurement of contrast for a given
spatial frequency square wave pattern
Example of square wave pattern displayed by HMD
Contrast frequency( )
Max Min
Max Min
CTF Depends on Pupil Position
• Horizontal and Vertical CTF both measured as
function of pupil position
• Cutoff at 50% of CTF at ideal pupil position
-67
-65
-63
-61
-59
-57
-55
-53
7
6
5
4
3
2
1
0
-1
-2
-3
-4
-5
-6
-7
00.10.20.30.40.50.6
mm
mm
Horizontal CTF vs Pupil Position
0.5-0.6
0.4-0.5
0.3-0.4
0.2-0.3
0.1-0.2
0-0.1
-67 -66 -65 -64 -63 -62 -61 -60 -59 -58 -57 -56 -55 -54
6
5
4
3
2
1
0
-1
-2
-3
-4
-5
-6
mm
mm
Vertical CTF vs Pupil Position
0.5-0.6
0.4-0.5
0.3-0.4
0.2-0.3
0.1-0.2
0-0.1
Ocularity
• Ocularity is another criterion
for categorising HMDs:
– Monocular
• HMD image goes to only one
eye
– Biocular
• Same HMD image to both eyes
– Binocular (stereoscopic)
• Different but matched images
to each eye
Interpupillary Distance (IPD)
• IPD is the distance between the
center of the pupils of the two eyes
• It is critical for the design of
binocular viewing systems
– Because both eye pupils need to be
positionedwithin the exit pupils of the
viewing system
• Viewing systems include
– Binocular microscopes, night vision
devices or goggles (NVGs), and headmounted
displays (HMDs)
https://en.wikipedia.org/wiki/Interpupillary_distance
Vignetting
• In optics, vignetting is a
reduction of an image's
brightness or saturation at
the periphery compared to
the image center
• An unintended and
undesired effect caused by
camera settings or lens
limitations
https://en.wikipedia.org/wiki/Vignetting
Example of vignetting characteristics
dependent on focal length
Eye Relief
• The eye relief of an optical
instrument (i.e. telescope,
microscope, binoculars) is
the distance from the last
surface of an eyepiece at
which the user's eye can
obtain the full viewing angle
• If a viewer's eye is outside
this distance, a reduced FOV
will be obtained
• The calculation is complex
https://en.wikipedia.org/wiki/Eye_relief
Optics showing eye relief and exit pupil.
(1) Real image, (2) Field diaphragm
(3) Eye relief, (4) Exit pupil
10/9/2016
4
Eye Relief Effect on Viewable FOV
Pupil placed at ER resulting
in vignetting of off axis field
angles (lose FOV at edges)
Eye point of rotation placed at ER
resulting in reduced clearance between
user’s eye and the HMD, but vignetting
minimized
Image-Forming Optical System
• An image-forming optical system is a system
capable of being used for imaging
• The diameter of the aperture is a common
criteria for comparison among optical systems
– i.e. Large telescopes
• The traditional systems are:
– Mirror-systems (catoptrics) - has a focal point
– Lens-systems (dioptrics) - has a focal point
– Optical fiber - transfers an image from one plane to
another without an optical focus
https://en.wikipedia.org/wiki/Image-forming_optical_system
Simple Magnifier HMD Design
o
Eyepiece
(one or more lenses) Display
(Image Source)
Eye f Image
Sherman & Craig, pp. 151-159
Thin Lens Equation
• 1/p + 1/q = 1/f
where
• p = object distance
– Distance from image source to eyepiece
• q = image distance
– Distance of image from the lens
• f = focal length of the lens
Thin Lens Equation Conventions
• If the incident light comes from the object, we
say it is a real object, and define the distance
from the lens to it as positive
– Otherwise, it is virtual and the distance is negative
• If the emergent light goes toward the image, we
say it is a real image, and define the distance
from the lens to it as positive
– f = positive for a converging lens
• A light ray through the center of the lens is un-
deflected
Formulas
• Visual Resolution in Cycles per degree
– (Vres) = Number of pixels /2(FoV in degrees)
• Example
– (1024 pixels per line)/(2*40 degrees) = Horizontal
resolution of 12.8 cycles per degree
• To convert to Snellen acuity (as in 20/xx)
– Vres = 600/xx
– So: Vres = 600/12.8 (20/47)
Sherman & Craig, pp. 151-159
10/9/2016
5
Virtual Image
Lens Display
Virtual
Image
f f
Sherman & Craig, pp. 151-159
Large Expanse Extra Perspective (LEEP)
• LEEP provides extreme wide-angle
stereoscopic optics used in photographic and
virtual reality systems
• Higher resolution (more pixels) in the middle
of the field of view, lower resolution on the
periphery
• Pincushion distortion
https://en.wikipedia.org/wiki/Eric_Howlett
LEEP Standard
http://www.leepvr.com/sid1992.php
LEEP Standard
http://www.leepvr.com/sid1992.php
Fresnel Lens
• Lenses of large aperture and short
focal length without the mass and
volume of material
– That would be required by a lens of
conventional design
• Can be made much thinner than a
comparable conventional lens
– i.e. Using a flat sheet
• A Fresnel lens can capture more
oblique light from a light source
– Thus allowing the light to be visible
over greater distances
• More even resolution distribution
• Less distortion
https://en.wikipedia.org/wiki/Fresnel_lens
Relationship Between Angle and
Screen distance
0.00
10.00
20.00
30.00
40.00
50.00
60.00
70.00
80.00
R
2.00
6.00
10.00
14.00
18.00
22.00
26.00
30.00
34.00
Angle in Radians
Distanceinmm
Leep
Fresnel
Sherman & Craig, pp. 151-159
10/9/2016
6
Distortion in LEEP Optics
A rectangle Maps to this
Camera Calibration
To Correct Distortion
• Must pre-distort
image
• This is a pixel-based
distortion
• Graphics rendering
uses linear
interpolation
• Too slow on most
systems
Sherman & Craig, pp. 151-159
Distorted Field of View
• Your computer graphics model assumes some
field of view
• Scan converter may overscan or underscan
– Not all of your graphics image may appear on the
screen
• Problem
– Are the display screens aligned perpendicular to your
optical axis?
Sherman & Craig, pp. 151-159
Distorted Field of View Example
Distance along
z-axis
Sherman & Craig, pp. 151-159
Collimated (o=f)
• Optical collimators can be used to:
– Calibrate other optical devices
– Check if all elements are aligned on the optical axis
– Set elements at proper focus
– Align two or more devices such as binoculars or gun barrels
and gunsights
• If the image source is placed at the focal point of the lens,
then the virtual image appears at optical infinity
– 1/p + 1/q = 1/f q = , if p=f
f
https://en.wikipedia.org/wiki/Collimator
10/9/2016
7
Compound Microscope HMD Design
• Relay lens produces a real image of the display
image source (screen) at some intermediate location
in the optical train
• The eyepiece is then used to produce an observable
virtual image of this intermediate image
Relay Lens
ImageIntermediate
Real Image
Eyepiece
Exit
Pupil
Sherman & Craig, pp. 151-159
Exit Pupil
• The area in back of the optics from which the
entire image can be seen
– Important if IPD not adjustable, mount not secure
• Compound microscope optical systems have a
real exit pupil
• Simple magnifier optical systems do not have an
exit pupil
Sherman & Craig, pp. 151-159
Displays
Visually Coupled Systems
• Integration of the natural visual and motor skills
of a user into the system that is controlling
• Basic components include:
– An immersive visual display
• HMD, large screen projection (CAVE), dome projection, etc
– A means of tracking head and/or eye motion
– A source of visual information that is dependent
on the user's head/eye motion
AR Displays Classification
• The most popular classification of display
technologies can be categorized into:
– Head mounted devices
• Head up display (HUD)
• Head mounted display (HMD)
• Head Mounted Projector (HMP)
– Non-head mounted devices
• Another one is split into (see next slide):
– Indoors
– Outdoors
Another AR Displays Classification
10/9/2016
8
Heads-Up Displays
Heads-Up Displays (HUD)
• HUD is any transparent display
that presents data without
requiring users to look away from
their usual viewpoints
• Origins on military where the
pilot's eyes do not need to
refocus to view the outside after
looking at the optically nearer
instruments
• Applications
– Military aviation, commercial
aircraft, automobiles, etc
https://en.wikipedia.org/wiki/Head-up_display
HUD Generations
• First Generation
– Use a CRT to generate an image on a phosphor screen,
having the disadvantage of the phosphor screen coating
degrading over time
– The majority of HUDs in operation today are of this type
• Second Generation
– Use a solid state light source, for example LED, which is
modulated by an LCD screen to display an image
– These systems do not fade or require the high voltages of
first generation systems
– These systems are on commercial aircraft
https://en.wikipedia.org/wiki/Head-up_display
HUD Generations .
• Third Generation
– Use optical waveguides to produce images directly in
the combiner rather than use a projection system
• Fourth Generation
– Use a scanning laser to display images and even video
imagery on a clear transparent medium
https://en.wikipedia.org/wiki/Head-up_display
HUD for Car Video
https://www.youtube.com/watch?v=EQmd-3dBRXo
Head-Mounted
Displays
10/9/2016
9
Head Mounted Displays
• Optical System
• Image Source
– CRT or Flat Panel (LCD)
• See-through or non see-through
• Mounting Apparatus
• Earphones
• Position Tracker
Characteristics of HMDs
• Immersive
– You are inside the computer world
– Can interact with real world (mouse, keyboard,
people)
• Ergonomics
• Resolution and field of view
• Tethered
Modern HMDs Video Head-Mounted Display
• Video head-mounted displays accept video
from a camera and mix it electronically with
computer graphics
– Easier to perform registration and calibration
– Watch a digital representation of the world
• Most popular method until now for AR
TriVisio
• Stereo video input
– PAL resolution cameras
• 2 x SVGA displays
– 30 degree FOV
– User adjustable convergence
• $6,000 USD
http://www.trivisio.com/
Vuzix Display - Wrap 1200DXAR
• 4th generation
• 3 DOF head tracker
• Stereoscopic 3D video
• 16:9 or 4:3 aspect ratio
• 1920 x 1080 resolution
• Weighs less than three ounces
https://www.vuzix.com/
10/9/2016
10
Video See Through Example
https://www.youtube.com/watch?v=nEGtha40PUo
Optical Head-Mounted Display
• Nowadays, see-through displays are lightweight
with high-resolution optical devices
• However, certain inefficiencies remain such as
sufficient:
– Brightness
– Resolution
– Field of view
– Contrast
Optical Head-Mounted Display
• Various techniques have existed for see-through
HMDs and can be summarized into two main
families:
– “Curved Mirror” (or Curved Combiner) based
– “Waveguide” or "Light-guide" based
• The curved mirror technique has been used by
Vuzix in their Star 1200 product, by Olympus, and
by Laster Technologies
• Various waveguide techniques have existed for
some time
– These techniques include diffraction optics, holographic
optics, polarized optics, and reflective optics
https://en.wikipedia.org/wiki/Optical_head-mounted_display
Waveguide Techniques
• Diffractive waveguide
– Slanted diffraction grating elements (nanometric 10E-9)
• Nokia technique now licensed to Vuzix
• Holographic waveguide
– 3 holographic optical elements (HOE) sandwiched
together (RGB)
• Used by Sony and Konica Minolta
• Polarized waveguide
– 6 multilayer coated (25–35) polarized reflectors in glass
sandwich
• Developed by Lumus
https://en.wikipedia.org/wiki/Optical_head-mounted_display
Waveguide Techniques .
• Reflective waveguide
– Thick light guide with single semi reflective mirror
• This technique is used by Epson in their Moverio product
• "Clear-Vu" reflective waveguide
– Thin monolithic molded plastic w/ surface reflectors
and conventional coatings
• Developed by Optinvent and used in their ORA product
• Switchable waveguide
– Developed by SBG Labs
https://en.wikipedia.org/wiki/Optical_head-mounted_display
Google Glass
• Google Glass is based on OHMD
technology
– Displays information in a
smartphone-likehands-free
format
– Wearers communicate with the
Internet via natural language
voice commands
• Available to the public on May
15 2014 for $1,500
– Stopped on January 15 2015
https://en.wikipedia.org/wiki/Google_Glass
10/9/2016
11
Innovega iOptik System
• It comprises a pair of contact lens
which refocus polarized light to
the pupil
• Allows the wearer to focus on an
image that is as near as 1.25 cm
to the eye
• Prototype features a field of view
of 60 degrees or more
– Aiming at 120 degrees FOV
• Designed for military use
– A consumer version coming soon
http://www.innovega-inc.com/index.php
Microsoft HoloLens
• First fully self-contained, holographic computer
– Uses spatial audio
– Limited FOV
• Features
– Inertial measurement unit (IMU)
• Includes an accelerometer, gyroscope, and a magnetometer)
– Four "environmentunderstanding"sensors (two on each
side)
– Energy-efficient depth camera with a 120°×120° angle of
view
– 2.4-megapixel photographic video camera
– Four-microphone array
– Ambient light sensor
https://www.microsoft.com/microsoft-hololens/en-us
Innovega iOptik System Video
http://www.innovega-inc.com/videos.php
Pinlight Displays
• Wide Field of View Augmented Reality
Eyeglasses using Defocused Point Light
Sources
• Instead of conventional optics
– LCD panel
– An array of point light sources
• Coding allows for miniature see-through
projectors
http://pinlights.info/
Pinlight Displays Video
http://pinlights.info/
Comparison of OHMDs Technologies
Combiner technology Size Eye box FOV Other Example
Flat combiner 45 degrees Thick Medium Medium Traditionaldesign Vuzix, Google Glass
Curved combiner Thick Large Large Classical bug-eye design
Many products (see
through and occlusion)
Phase conjugate material Thick Medium Medium Very bulky OdaLab
Buried Fresnel combiner Thin Large Medium Parasitic diffraction effects
The Technology
Partnership (TTP)
Cascaded prism/mirror
combiner
Variable Medium to Large Medium Louver effects Lumus, Optinvent
Free form TIR combiner Medium Large Medium Bulky glass combiner
Canon, Verizon & Kopin
(see through and
occlusion)
Diffractive combiner with
EPE
Very thin Very large Medium
Haze effects, parasitic
effects, difficult to
replicate
Nokia/ Vuzix
Holographic waveguide
combiner
Very thin Medium to Large in H Medium
Requires volume
holographic materials
Sony
Holographic light guide
combiner
Medium Small in V Medium
Requires volume
holographic materials
Konica Minolta
Combo diffuser/contact
lens
Thin (glasses) Very large Very large
Requires contact lens +
glasses
Innovega & EPFL
Tapered opaque light
guide
Medium Small Small Image can be relocated Olympus
https://en.wikipedia.org/wiki/Optical_head-mounted_display
10/9/2016
12
Bionic Contact Lenses
https://en.wikipedia.org/wiki/Optical_head-mounted_display
Head Mounted
Projector
Head Mounted Projector
• Retro-reflective Material
• Potentially portable
• NVIS P-50 HMPD
– 1280x1024/eye
– Stereoscopic
– 50 degree FOV
http://www.nvis.com/
HMD vs HMP
Head Mounted Display Head Mounted Projector
Non-Heads Mounted
Displays
Non-Head Mounted Devices
• The most common non-head mounted
displays can be categorised as:
– Small Area Displays
– Large Area Displays
– Spatial Displays
10/9/2016
13
Small Area Displays
• Small area displays are portable and thus be
suitable for many VR applications
• The major disadvantages of these displays are
the limited working area and resolution
– Getting better!
• Small area displays have also illumination
problems
Small Area Displays .
Large Area Displays
• Three basic configurations:
– Front projection
– Back projection
– Conventional monitors
• CRTs, LCDs TouchScreens and Plasma
• Users must use 3D glasses or HMDs
• The most significant disadvantage of large
screen displays is the limited area of operation
– i.e. Limitted movement
Large Area Displays
ImmersaDesk CAVE System
AR for Large Screens Video
https://www.youtube.com/watch?v=EtYEtHHHJkc
Spatial Displays
10/9/2016
14
Spatial Displays
• In contrast to small area displays spatial displays isolate
most of the technology from the user and integrate it
into the environment
• Large screens and spatially immersive displays extend
the idea of ImmersaDesk using multiple projection
screens and can be used to create a very effective and
immersive experience
Spatial Displays .
• Nothing to wear and/or carry
• Uses digital projectors to display information
• Marker-based and markerless devices
AR KeyboardAR Phone Keypad
Spatial AR Video
https://www.youtube.com/watch?v=Df7fZAYVAIE
Pico Projectors
• Extra small projectors
– Microvision, 3M, Samsung, Philips
http://www.mvis.com/
MIT Sixth Sense
• Comprised of a pocket
projector, a mirror and a
camera
• $350 to build
http://www.pranavmistry.com/projects/sixthsense/
AR Haptic Workbench
10/9/2016
15
AR Types
Monitor-based AR
• Simplest available
• Treat laptop/PDA/cell phone as a window
through which you can see AR world
Advantages of Monitor-based AR
• Consumer-level equipment
• Most practical
• A lot of current research
aimed here
• Other current active area is a
flip-down optical display
Video See-Through AR
Advantages of Video See-Through AR
• Flexibility in composition strategies
• Real and virtual view delays can be matched
• True occlusion
• Wide FOV is easier to support
Disadvantages of Video See-Through
AR
• Not easy to make ‘good’ quality photorealistic
graphic scenes
• Can be more expensive
10/9/2016
16
Optical See-Through AR Advantages of Optical See-Through AR
• Safety
• Light weight
• Simplicity (cheaper)
• Resolution
• No eye offset
Disadvantages of Optical See-Through
AR
• Prone to lighting conditions
• Registration much harder!
• Optics are not yet there
Other AR Types
Eye Multiplexed AR Approach
• Let users combine the views of the two worlds
mentally in their minds
– The virtual scene is registered to the physical
environment
– The rendered image shows the same view of the
physical scene that the user is looking at
• However, the rendered image is not composited
with the real world view
– Up to the user to mentally combine the two images
in their minds
Billinghurst, M., Clark, A. Lee, G. A Survey of Augmented Reality, Foundations and Trends in Human-Computer Interaction, Vol. 8, No. 2-3 2014
Eye Multiplexed AR Architecture
Billinghurst, M., Clark, A. Lee, G. A Survey of Augmented Reality, Foundations and Trends in Human-Computer Interaction, Vol. 8, No. 2-3 2014
10/9/2016
17
Other Sensory Displays
Billinghurst, M., Clark, A. Lee, G. A Survey of Augmented Reality, Foundations and Trends in Human-Computer Interaction, Vol. 8, No. 2-3 2014
Alternate Displays
Billinghurst, M., Clark, A. Lee, G. A Survey of Augmented Reality, Foundations and Trends in Human-Computer Interaction, Vol. 8, No. 2-3 2014
Virtual Showcase
• Mirrors on a projection table
– Head tracked stereo
– Up to 4 users
– Merges graphic and real objects
– Exhibit/museum applications
• Fraunhofer Institute (2001)
Billinghurst, M., Clark, A. Lee, G. A Survey of Augmented Reality, Foundations and Trends in Human-Computer Interaction, Vol. 8, No. 2-3 2014
Augmented Paleontology
Billinghurst, M., Clark, A. Lee, G. A Survey of Augmented Reality, Foundations and Trends in Human-Computer Interaction, Vol. 8, No. 2-3 2014
Questions Acknowledgements
• Special Thanks to Prof. Mark Billinghurst