20/11/2017
1
PA198
Augmented Reality Interfaces
Lecture 9
Wearable Augmented Reality
Fotis Liarokapis
liarokap@fi.muni.cz
20th November 2017
Introduction
Intro to Wearable Computing (WC)
• Technology which allows for the human and
computer to interact, process data, and perform
tasks as one unit
• The concept of wearable computers attempts to
bridge the ‘interaction gap’ between the
computer and a human
• Wearable computing promotes devices that
should be as natural to the user as wearing
sunglasses or clothes
Conventional Computer
Roadmap: Wearable Computing 2020, Wear it at work.
Today’s Mobile Interaction
• Unusable when interaction with the physical
world needed
Roadmap: Wearable Computing 2020, Wear it at work.
The Wearable Vision
• Non disruptive interaction
• Environment oriented
– Context recognition
– Augmentation
• Physically unobtrusive
• Seamlessly connected
Roadmap: Wearable Computing 2020, Wear it at work.
20/11/2017
2
Wearable vs. Mobile Computing
• Focus on the interaction of user/system
real
world
user
mobile
system
<5%
real
world
user
mobile
system
95%
Interaction mode change
real
worlduser
wearable
system
100 %
What is a Wearable Computer?
• A computer that is
subsumed into the personal
space of the user
• Controlled by the user, and
always with the user – it is
always on and always
accessible
– Operational and interactional
consistency
Wearable Computer Definition
• A wearable computer offers all the features of
a regular computing system, but is also totally
related with the user
Ganguly, K. A Study on Wearable Computing, CS898A - Mobile / Wireless Communications
Fundamentals of Wearable Computing
• Humanistic Intelligence (HI)
• Human-Computer Interaction (HCI)
• Mediated Reality
Ganguly, K. A Study on Wearable Computing, CS898A - Mobile / Wireless Communications
Humanistic Intelligence (HI)
• HI is the intelligence that arises when a human
is part of the feedback loop of a computational
process in which the human and computer are
linked
• This creates a far more powerful entity than the
individual parts
Ganguly, K. A Study on Wearable Computing, CS898A - Mobile / Wireless Communications
Signal flow path theory of HI
"Humanistic intelligence wearcompdef multichannel6only" by Glogger - Own work. Licensed under CC BY-SA 3.0 via Commons -
https://commons.wikimedia.org/wiki/File:Humanistic_intelligence_wearcompdef_multichannel6only.png#/media/File:Humanistic_intelligence_
wearcompdef_multichannel6only.png
20/11/2017
3
HCI
• HCI typically treats the human and computer as 2
separate entities
• Wearable computing extends the HCI concept
– The computer can be regarded as a second brain,
with it’s sensory modalities and additional senses
adding to the wearer’s (paradigm shift)
• Idea is to move the tools of augmented
intelligence and communication directly onto the
human body
Ganguly, K. A Study on Wearable Computing, CS898A - Mobile / Wireless Communications
Mediated Reality
• Refers to the ability to add to,
subtract information from, or
otherwise manipulate one's
perception of reality
– Through the use of a wearable
computer or hand-held device
• Typically, it is the user's visual
perception of the environment
that is mediated
https://en.wikipedia.org/wiki/Computer-mediated_reality
Displays what's really there and then
this allows a computer to be inserted
into the "reality stream" to modify it
Mediated Reality .
• Mixed reality and
augmented reality are
special cases of
mediated reality
https://en.wikipedia.org/wiki/Computer-mediated_reality
Goal of Wearable Computing
• Main goal of the wearable computing paradigm
is to produce a symbiotic relationship between
the human and computer
– Rather than attempting to emulate human
intelligence
• The computer simply performs tasks at which it
is much better and faster at doing
Ganguly, K. A Study on Wearable Computing, CS898A - Mobile / Wireless Communications
Communications
• Wireless communication is an integral part of
wearable computing
– Extremely important!
• Nowadays WC’s use communication protocols
such as:
– 802.11x
– Bluetooth
– Infra-red
Ganguly, K. A Study on Wearable Computing, CS898A - Mobile / Wireless Communications
Hardware
• Small size and light-weight
– Getting better and better
– Innovative design of components
• Functionality is decided by where on the body it
is worn
– Head-mounted are the most common
• Multiple standard connectors
– i.e. USB
• Innovative power use
– Batteries are still a problem
20/11/2017
4
Software
• Common Operating Systems:
– Windows
– Linux (popular)
– MS-DOS
• GUIs are typically minimal
• Installed applications depend on the function of
the device
• Use of Agents is mandatory, not optional
– i.e. Remembrance agent, context-aware agent, etc
Ganguly, K. A Study on Wearable Computing, CS898A - Mobile / Wireless Communications
Why Use Wearables
• Since they are wearable they are always with
you
– Difficult to loose
• Instant access, information anywhere and at
anytime
– Laptops require preparation time
– PDAs require both hands
• Can become very personal items
– Transparent use
Who Uses Wearables
• Researchers
– i.e. Augmented reality
• Field workers
– Access to information given by
remote experts
• Technicians
– Blueprints
• Military
– Soldiers monitoring health and
equipment
Characteristics of Computing Devices
[L. Gorlenko and R. Merrick, No wires attached: Usability challenges in the connected mobile world]
Brief History
1961 1966 1977 1981
1991 1993
1968
1991 1992 1993 1996
1980
Evolution of Wearable Computers
20/11/2017
5
Father of Wearable Computing
https://www.youtube.com/watch?v=jhTchUYEIdU
Wearable Devices
Ganguly, K. A Study on Wearable Computing, CS898A - Mobile / Wireless Communications
Architecture
Key Architecture Question
• What does integration with the outfit mean ?
• Observation:
– Clothing is a heterogeneous, distributed,
dynamically reconfigurable system
• Function
• Technology
• User expectation
• Solution:
– 4 layers of integration reflecting relation between
clothing and electronic
Roadmap: Wearable Computing 2020, Wear it at work.
Layer 1: Functional Textiles
Roadmap: Wearable Computing 2020, Wear it at work.
spacer
fabric
Simple Functions in Textiles
Roadmap: Wearable Computing 2020, Wear it at work.
20/11/2017
6
Smart Shirts
• Wearable Motherboard, Smart Shirt (GATECH)
Roadmap: Wearable Computing 2020, Wear it at work.
Layer 2: Embedded Microsystems
Roadmap: Wearable Computing 2020, Wear it at work.
Miniaturized Sensors
• (Bharatula, Ossevoort, Lukowicz, Tröster, 2004)
Roadmap: Wearable Computing 2020, Wear it at work.
Layer 3: Attachable Peripherals
Roadmap: Wearable Computing 2020, Wear it at work.
Augmented Reality
Roadmap: Wearable Computing 2020, Wear it at work. Billinghurst, M. Designing for Wearables, AWE Asia 2015.
20/11/2017
7
Sensor Based Interfaces
Roadmap: Wearable Computing 2020, Wear it at work.
ETH QBIC: Belt Integrated System
• Buckle as computer housing
• Belt as peripheral bus
– Connectors
– Batteries
– Wireless adapter, storage etc
Roadmap: Wearable Computing 2020, Wear it at work.
QBIC
Roadmap: Wearable Computing 2020, Wear it at work.
ETH QBIC
Roadmap: Wearable Computing 2020, Wear it at work.
Layer 4: Carry On Devices
Roadmap: Wearable Computing 2020, Wear it at work.
Wearable Computer New Scientist
https://www.youtube.com/watch?v=9DNXLAogM7Q
20/11/2017
8
Design Guidelines
How To Design This?
Billinghurst, M. Designing for Wearables, AWE Asia 2015.
Universal Design Principles
• Flexibility
• Equitable use
• Easy to perceive
• Simple and intuitive
• Low physical effort
• High tolerance for error
Billinghurst, M. Designing for Wearables, AWE Asia 2015.
Designing for Wearables
• Wearables are intimate on-body devices, so
interface design for wearables, means:
– Designing for Attention
– Designing for Interruption
– Designing for User Experience
– Designing for Social Interaction
Billinghurst, M. Designing for Wearables, AWE Asia 2015.
Micro-Interactions
• Using mobile phone people split their attention
between the display and the real world
Billinghurst, M. Designing for Wearables, AWE Asia 2015.
Time Looking at Screen
• Oulasvirta, A. (2005). The fragmentation of
attention in mobile interaction, and what to do
with it. interactions, 12(6), 16-18
Billinghurst, M. Designing for Wearables, AWE Asia 2015.
20/11/2017
9
Using Micro Interactions
• Quick micro-interactions reduce divided
attention and allow people to spend more time
in real world
Billinghurst, M. Designing for Wearables, AWE Asia 2015.
Like A Rear View Mirror
• Don't overload the user
• Stick to the absolutely essential
– Avoid long interactions
• Be explicit
Billinghurst, M. Designing for Wearables, AWE Asia 2015.
Make it Glanceable
• Seek to rigorously reduce information density
• Successful designs afford for recognition, not
reading
Billinghurst, M. Designing for Wearables, AWE Asia 2015.
Reduce the Number of Info Chunks
• Designing for recognition, not reading
• Reducing the total # of information chunks will
greatly increase the glance ability of the design
Billinghurst, M. Designing for Wearables, AWE Asia 2015.
Design Single Interactions < 4 sec
Billinghurst, M. Designing for Wearables, AWE Asia 2015.
Test the Glanceability of Your Design
Billinghurst, M. Designing for Wearables, AWE Asia 2015.
20/11/2017
10
Design for Micro-Interactions
• Design interactions less than a few seconds
– Tiny bursts of interaction
– One task per interaction
– One input per interaction
• Benefits
– Use limited input
– Minimize interruptions
– Reduce attention fragmentation
Billinghurst, M. Designing for Wearables, AWE Asia 2015.
Important Note
• Design for limited attention/micro-interactions
• No more than 4 seconds to complete a given
step in the interaction
Billinghurst, M. Designing for Wearables, AWE Asia 2015.
Designing for Interruptions
• Assume user is engaged in critical real world
task
• Use context to filter interruptions
– Is it necessary?
• Interrupt in way that consumes least attention
• Allow user to dismiss interruption with minimal
effort
• Progressively disclose information and increase
interaction
Billinghurst, M. Designing for Wearables, AWE Asia 2015.
Interruptions on Glass Example
• Receiving SMS on Glass
– Gradually increase engagement and attention load
– Respond to user engagement
Billinghurst, M. Designing for Wearables, AWE Asia 2015.
Important Note
• Design carefully for interruption
• Low cognitive load that can be increased as
needed
– i.e. NASA TLX
NASA TLX
• A subjective workload assessment tool
• Allows users to perform subjective workload
assessments on operator(s) working with
various human-machine systems
• A multi-dimensional rating procedure that
derives an overall workload score based on a
weighted average of ratings on six subscales
http://humansystems.arc.nasa.gov/groups/tlx/
20/11/2017
11
Consider Your User
• Wearable User
– Probably Mobile
– One/no hand interaction
– Short application use
– Need to be able to
multitask
– Use in outdoor or indoor
environment
– Want to enhance
interaction with real
world
Billinghurst, M. Designing for Wearables, AWE Asia 2015.
How To Take A Note?
Billinghurst, M. Designing for Wearables, AWE Asia 2015.
Glass Pictures Example
• On Glass there are three ways to take a picture
– Voice commands – “Ok Glass, Take a Picture”
– Touch navigation through menu
– Winking with right eye
• Which you use depends on context
– Riding a bike outdoors – voice commands
– During a meeting – winking
Billinghurst, M. Designing for Wearables, AWE Asia 2015.
Important Note
• Provide many different ways of accessing
functionality
• Each person is different!
Design For Device
• Simple, relevant information
• Complement existing devices
Billinghurst, M. Designing for Wearables, AWE Asia 2015.
20/11/2017
12
Glass User Interface
Billinghurst, M. Designing for Wearables, AWE Asia 2015.
Test Indoors & Outdoors
Billinghurst, M. Designing for Wearables, AWE Asia 2015.
Design for Context
Billinghurst, M. Designing for Wearables, AWE Asia 2015.
Design for Ecosystem of Wearables
• User have multiple devices
– Phone, watch
– Fitness band, HMD
• Each device should be used
when it’s most relevant and
when it’s the easiest interaction
available
Billinghurst, M. Designing for Wearables, AWE Asia 2015.
Interface Guidelines
• Design for device
• Use multiple input options
• Do one thing at a time
• Consider user context
• Design for indoor and outdoor use
• Design for device ecosystem
Billinghurst, M. Designing for Wearables, AWE Asia 2015.
Social Acceptance
• People don’t want to look silly
– Only 12% of 4,600 adults would be
willing to wear AR glasses
– 20% of mobile AR browser users
experience social issues
• Acceptance more due to social
than technical issues
– Needs further studies
• Ethnographic, field tests, longitudinal
Billinghurst, M. Designing for Wearables, AWE Asia 2015.
20/11/2017
13
Social Implications
• Freak or Trendy?
Social Implications Questions
• Will the use of wearable computers become a
symbol of elitism or will they become
accepted as part of the daily routine?
• Is the integration of computer equipment into
the body more acceptable than a wearable
computer module?
Prototyping
Main Goal
• How can we quickly prototype wearable
computing applications with little or no
experience
• Understand the market and user needs first
Why Prototype?
• Quick visual design
• Capture key interactions
• Focus on user experience
• Communicate design ideas
• Learn by experience
Billinghurst, M. Designing for Wearables, AWE Asia 2015.
Prototype Design Process
20/11/2017
14
Typical Development Steps
Billinghurst, M. Designing for Wearables, AWE Asia 2015.
Sketched Interfaces
• Sketch + Powerpoint/Photoshop/Illustrator
Billinghurst, M. Designing for Wearables, AWE Asia 2015.
Paper Prototype
• Use sketched interface in template
Billinghurst, M. Designing for Wearables, AWE Asia 2015.
Smart Watch Templates
• https://dribbble.com/jaysuthar/buckets/260235-watch
Billinghurst, M. Designing for Wearables, AWE Asia 2015.
Wearables Today
Application Areas
• Warehouse picking
• Inspection
• Maintenance
• Repair
• Medical
• Security
• Military
20/11/2017
15
A Prototypical Wearable Device
• Hearing aid computer
• Permanently useful
• Augments user‘s perception
• Situation sensitive
– Adjusts amplification to the situation
• Virtually unnoticeable
Roadmap: Wearable Computing 2020, Wear it at work.
Consumer Applications
• Fossil has created the wrist PDA,
it uses the Palm OS, and has
almost all the functionality of a
standard Palm Pilot
• Accenture Technology Labs has
created a device that uses two
small microphones, and a camera
to assist in remembering a
persons name
Consumer Applications .
• MIT Media Lab has developed
handbags that alert you when you
leave
– Things behind, your wallet, or an
umbrella if you need one
• Oakley has developed the first digital
music eyewear
– The Oakley Thump, comes equipped
with a solid state hard drive, for skip
free listening
Intel Wearable Video
https://www.youtube.com/watch?v=iwSpn7H7vKg
Medical Applications
• Wrist worn medical monitoring devices
Medical Applications .
• The C-Leg
– Uses the C programming language to do
all of the calculations required to function,
hence “C”-leg
– Sensors from the foot and ankle get load
information, sensors from the knee get
the precise angle of the leg and swing
speed, this is all sent to a microprocessor
for processing
Ottobockus.com
20/11/2017
16
C-Leg Video
https://www.youtube.com/watch?v=F0bpXmzCw7A
Personal/Recreational Use
• Web surfing
• Email/Text/Video Messaging
• Note taking
• Audio/Video Entertainment
Hiris Video
https://www.youtube.com/watch?v=VF9Qkt89u30
Military Wearables
Early Years - The Soldier's Computer
• James Schoening, Matt Zieniewicz 1989, John
Flatt, Sal Barone, and Almon Gillette, 1990
• Schoening:
– small wearable computer, integrated with a wireless
link and HMD
• Matt Zieniewicz:
– wireless data transmission, image capture,
integrated Global Positioning System (GPS)
receivers, and menu-driven software
Zieniewicz, M.J., Johnson, D.C., Wong, D.C., Flatt , J.D. The Evolution of Army Wearable Computers, Research, Development, and Engineering
Center, US Army Communications Electronic Command
Army Material Command's - First Trade
Show
• Agilis bricktype 386-based
computer
• Software:
– Creating reports, displaying
battlefield situation maps
– Could enter and transmit simple
reports to other units
• HMD:
– 14-inch monochromatic display
• Interaction:
– Trackball for input
Zieniewicz, M.J., Johnson, D.C., Wong, D.C., Flatt , J.D. The Evolution of Army Wearable Computers, Research, Development, and Engineering
Center, US Army Communications Electronic Command
20/11/2017
17
The SIPE project
• Spring of 1990
– Led by Carol Fitzgerald
• New digitized battlefield concept:
– portable, wearable battery-powered computer
• Computer needed to include:
– Image capture
– Integrated radio
– Portable display unit
Zieniewicz, M.J., Johnson, D.C., Wong, D.C., Flatt , J.D. The Evolution of Army Wearable Computers, Research, Development, and Engineering
Center, US Army Communications Electronic Command
SIPE Requirements
• Challenges
– Integrate these components into a lightweight
package
– Bring computing devices to the individual soldier
• None of the functions were commercially
available
• Software:
– Developed in C
Zieniewicz, M.J., Johnson, D.C., Wong, D.C., Flatt , J.D. The Evolution of Army Wearable Computers, Research, Development, and Engineering
Center, US Army Communications Electronic Command
SIPE Functionality
• The new system aimed to digitize basic
battlefield operations to help soldiers
– Read maps, navigate, and maintain situation
awareness
– Receive, prepare, and send written field reports
– Capture and transmit color still images for
reconnaissance purposes
– Access battlefield operations reference material
Zieniewicz, M.J., Johnson, D.C., Wong, D.C., Flatt , J.D. The Evolution of Army Wearable Computers, Research, Development, and Engineering
Center, US Army Communications Electronic Command
SIPE System Architecture
• Computer processor with memory
• GPS receiver and a digital compass
• Data radio
• Video capture system
• A miniature color camera
• A video controller subsystem
• An HMD
• A power supply subsystem
• Wiring harnesses and packaging
Zieniewicz, M.J., Johnson, D.C., Wong, D.C., Flatt , J.D. The Evolution of Army Wearable Computers, Research, Development, and Engineering
Center, US Army Communications Electronic Command
Feedback From Soldiers
• Operate longer on a set of batteries
• Computer-radio-GPS:
– 18 pounds
• HMD into helmet
– nearly 8 pounds
• CRT display
– 15 pounds
• Drawback
– Delay in capturing and sending a still video image
Zieniewicz, M.J., Johnson, D.C., Wong, D.C., Flatt , J.D. The Evolution of Army Wearable Computers, Research, Development, and Engineering
Center, US Army Communications Electronic Command
Land Warrior Project
• Land Warrior requirements:
– Integrate small arms with high-tech equipment
– Provide communications and command and control
at the infantry soldier level
– Look at the individual infantry soldier as a complete
unit rather than as a segment of a larger force
• Cancelled in 2007, but restarted in 2008
https://en.wikipedia.org/wiki/Land_Warrior
20/11/2017
18
Major Subsystems and Components
• Computer subsystem
• Helmet subsystem
• Control and communications subsystem
• Weapons subsystem
• Navigation system
Zieniewicz, M.J., Johnson, D.C., Wong, D.C., Flatt , J.D. The Evolution of Army Wearable Computers, Research, Development, and Engineering
Center, US Army Communications Electronic Command
Helmet Subsystem
Zieniewicz, M.J., Johnson, D.C., Wong, D.C., Flatt , J.D. The Evolution of Army Wearable Computers, Research, Development, and Engineering
Center, US Army Communications Electronic Command
Computer Subsystem
Zieniewicz, M.J., Johnson, D.C., Wong, D.C., Flatt , J.D. The Evolution of Army Wearable Computers, Research, Development, and Engineering
Center, US Army Communications Electronic Command
Land Warrior Video
https://www.youtube.com/watch?v=dDrvZzfmuk4
21st-Century Soldier
• 21st-Century Soldier (Czech: Voják 21. století)
is a Czech Future Soldier military project
• The agreement of Czech Ministry of Defence
and VOP-026 Šternberk about the future
soldier program was signed in 2004
• A functional prototype was created at the end
of 2005
– Expected to be operation in 2012
https://en.wikipedia.org/wiki/21st-Century_Soldier
Timeline of Army’s Wearable Systems
Zieniewicz, M.J., Johnson, D.C., Wong, D.C., Flatt , J.D. The Evolution of Army Wearable Computers, Research, Development, and Engineering
Center, US Army Communications Electronic Command
20/11/2017
19
Military Suit & Suspended Armor
https://www.youtube.com/watch?v=Ix_KVBLrEdo
Conclusions
• Wearables mainly used by Universities
– Industrial applications are catching up
• Major obstacles
– Power, cooling, processing power, lightweight
components, displays, graphics
• Future:
– A single wearable will replace all separate devices
we carry and use on a daily basis
Questions