Foundations and Trends R• in Human-Computer
Interaction
Vol. 8, No. 2-3 (2014) 73–272
c• 2015 M. Billinghurst, A. Clark, and G. Lee
DOI: 10.1561/1100000049
A Survey of Augmented Reality
Mark Billinghurst, Adrian Clark, and Gun Lee
The Human Interface Technology Laboratory New Zealand
University of Canterbury
Christchurch, New Zealand
{mark.billinghurst, adrian.clark, gun.lee}@hitlabnz.org
Contents
1 Introduction 74
2 Deﬁnition and Taxonomy 77
3 History 85
4 AR Tracking Technology 103
4.1 Magnetic Tracking . . . . . . . . . . . . . . . . . . . . . . 104
4.2 Vision Based Tracking . . . . . . . . . . . . . . . . . . . . 105
4.3 Inertial Tracking . . . . . . . . . . . . . . . . . . . . . . . 122
4.4 GPS Tracking . . . . . . . . . . . . . . . . . . . . . . . . 122
4.5 Hybrid Tracking . . . . . . . . . . . . . . . . . . . . . . . 123
4.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
5 AR Display Technology 126
5.1 Combining Real and Virtual View Images . . . . . . . . . 126
5.2 Eye-to-World spectrum . . . . . . . . . . . . . . . . . . . 139
5.3 Other Sensory Displays . . . . . . . . . . . . . . . . . . . 144
5.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
6 AR Development Tools 147
6.1 Low Level software libraries and frameworks . . . . . . . . 148
ii
iii
6.2 Rapid Prototyping/Development Tools . . . . . . . . . . . 153
6.3 Plug-ins to Existing Developer Tools . . . . . . . . . . . . 156
6.4 Stand Alone AR Authoring Tools . . . . . . . . . . . . . . 158
6.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
7 AR Input and Interaction Technologies 165
7.1 AR Information Browsers . . . . . . . . . . . . . . . . . . 166
7.2 3D User Interfaces in AR . . . . . . . . . . . . . . . . . . 167
7.3 Tangible User Interface in AR . . . . . . . . . . . . . . . . 169
7.4 Natural User Interfaces in AR . . . . . . . . . . . . . . . . 172
7.5 Multimodal Interaction in AR . . . . . . . . . . . . . . . . 174
7.6 Other Interaction Methods . . . . . . . . . . . . . . . . . 176
7.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
8 Design Guidelines and Interface Patterns 179
8.1 Case Study: levelHead . . . . . . . . . . . . . . . . . . . . 183
8.2 Using Design Patterns . . . . . . . . . . . . . . . . . . . . 184
8.3 Designing for Special Needs . . . . . . . . . . . . . . . . . 187
8.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . 188
9 Evaluation of AR Systems 189
9.1 Types of User Studies . . . . . . . . . . . . . . . . . . . . 191
9.2 Evaluation Methods . . . . . . . . . . . . . . . . . . . . . 203
9.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . 206
10 AR Applications Today 207
10.1 Education . . . . . . . . . . . . . . . . . . . . . . . . . . 207
10.2 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . 211
10.3 Marketing . . . . . . . . . . . . . . . . . . . . . . . . . . 217
11 Research Directions 219
11.1 Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . 220
11.2 Interaction . . . . . . . . . . . . . . . . . . . . . . . . . . 223
11.3 Displays . . . . . . . . . . . . . . . . . . . . . . . . . . . 229
11.4 Social Acceptance . . . . . . . . . . . . . . . . . . . . . . 231
11.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . 235
iv
12 Conclusion 236
Acknowledgements 238
References 239
Abstract
This survey summarizes almost 50 years of research and development
in the ﬁeld of Augmented Reality (AR). From early research in the
1960’s until widespread availability by the 2010’s there has been steady
progress towards the goal of being able to seamlessly combine real and
virtual worlds. We provide an overview of the common deﬁnitions of
AR, and show how AR ﬁts into taxonomies of other related technologies.
A history of important milestones in Augmented Reality is followed
by sections on the key enabling technologies of tracking, display
and input devices. We also review design guidelines and provide some
examples of successful AR applications. Finally, we conclude with a
summary of directions for future work and a review of some of the
areas that are currently being researched.
M. Billinghurst, A. Clark, and G. Lee. A Survey of Augmented Reality. Foundations
and TrendsR•
in Human-Computer Interaction, vol. 8, no. 2-3, pp. 73–272, 2014.
DOI: 10.1561/1100000049.
1
Introduction
In 1977 many moviegoers were amazed as a small robot projected a
three-dimensional image of a woman in mid air. With the words "Help
me Obiwan-Kenobi, you’re my only hope", a recording of Princess Leia
delivered a message that would change Luke Skywalker’s life forever.
In this Star Wars1 scene, special e ects were used to create the magical
impression that three-dimensional virtual content was appearing as
part of the real world. The movie forecast a future where people could
interact with computers as easily as interacting with the real world
around them, with digital and physical objects existing in the same
space.
Thirty years later, in the 2008 US presidential campaign, a version
of technology was shown for real. During the CNN election coverage
reporter Wolf Blitzer turned to an empty studio and suddenly
a life sized three-dimensional virtual image of reporter Jessica Yellin
appeared beamed in live from Chicago2. Just like Princess Leia, she
appeared to be part of the real world, but this time it was real and
not through movie special e ects. Wolf was able to talk to her as easily
1
http://www.starwars.com
2
http://edition.cnn.com/2008/TECH/11/06/hologram.yellin/
74
75
as if there was there face to face, even though she was thousands of
miles away. It had taken only thirty years for the Star Wars fantasy to
become reality.
The CNN experience is an example of technology known as Augmented
Reality (AR), which aims to create the illusion that virtual
images are seamlessly blended with the real world. AR is one of the
most recent developments in human computer interaction technology.
Ever since the creation of the ﬁrst interactive computers there has been
a drive to create intuitive interfaces. Beginning in the 1960’s, computer
input has changed from punch cards, to teletype, then mouse and keyboard,
and beyond. One overarching goal is to make the computer
interface invisible and make interacting with the computer as natural
as interacting with real world objects, removing the separation between
the digital and physical. Augmented Reality is one of the ﬁrst technologies
that makes this possible.
Star Wars and CNN showed how the technology could enhance
communication and information presentation, but like many enabling
technologies, AR can be used in a wide variety of application domains.
Researchers have developed prototypes in medicine, entertainment, education
and engineering, among others. For example, doctors can use
AR to show medical data inside the patient body [Navab et al., 2007,
Kutter et al., 2008], game players can ﬁght virtual monsters in the
real world [Piekarski and Thomas, 2002a], architects can see unﬁnished
building [Thomas et al., 1999], and students can assemble virtual
molecules in the real world [Fjeld and Voegtli, 2002]. Figure 1.1 shows
a range of applications.
The potential of AR has just begun to be tapped and there is more
opportunity than ever before to create compelling AR experiences. The
software and hardware is becoming readily available as are tools that
allow even non-programmers to build AR applications. However there
are also important research goals that must be addressed before the
full potential of AR is realized.
The goal of this survey is to provide an ideal starting point for those
who want an overview of the technology and to undertake research and
development in the ﬁeld. This survey compliments the earlier surveys of
76 Introduction
(a) ARQuake outdoor AR game
[Piekarski and Thomas, 2002a]
(b) AR architecture by Re+Public
http://www.republiclab.com
(c) AR in medicine [Kutter et al., 2008]
Figure 1.1: Typical AR applications.
Azuma [1997], Azuma et al. [2001], Van Krevelen and Poelman [2010]
and Carmigniani et al. [2011] and the research survey of Zhou et al.
[2008]. In the next section we provide a more formal deﬁnition of AR
and related taxonomies, then a history of the AR development over
the last 50 years. The rest of this survey gives an overview of key AR
technologies such as Tracking, Display and Input Devices. We continue
with sections on Development Tools, Interaction Design methods and
Evaluation Techniques. Finally, we conclude with promising directions
for AR research and future work.
2
Deﬁnition and Taxonomy
In one of the most commonly accepted deﬁnitions, researcher Ron
Azuma says that Augmented Reality is technology that has three key
requirements [Azuma, 1997]:
1) It combines real and virtual content
2) It is interactive in real time
3) It is registered in 3D
The CNN virtual presenter satisﬁes these requirements. The virtual
image of Jessica Yellin appears in a live camera view of the studio, she
is interactive and responds to Wolf Blitzer in real time, and ﬁnally, her
image appears to be ﬁxed or registered in place in the real world.
These three characteristics also deﬁne the technical requirements of
an AR system, namely that it has to have a display that can combine
real and virtual images, a computer system that can generate interactive
graphics the responds to user input in real time, and a tracking
system that can ﬁnd the position of the users viewpoint and enable the
virtual image to appear ﬁxed in the real world. In the later sections of
this survey we explore each of these technology areas in more depth. It
77
78 Deﬁnition and Taxonomy
Figure 2.1: Rekimoto’s comparison of HCI styles (R = real world, C = computer).
[Rekimoto and Nagao, 1995]
should be noted that Azuma’s deﬁnition doesn’t place any limitations
on the type of technology used, nor is it speciﬁc to visual information,
and some AR systems provide an audio or haptic experience.
In the larger context, Augmented Reality is the latest e ort by
scientists and engineers to make computer interfaces invisible and enhance
user interaction with the real world. Rekimoto distinguishes between
traditional desktop computer interfaces and those that attempt
to make the computer interface invisible [Rekimoto and Nagao, 1995].
As Figure 2.1(a) shows, with a traditional desktop computer and desktop
WIMP (Windows, Icons, Menus, Pointer) [Costabile and Matera,
1999] based graphical user interface (GUI) there is a distinct separation
between what the on-screen digital domain and the real world.
One approach to overcome this is through the approach of Ubiquitous
Computing [Weiser, 1993] (Figure 2.1(c)) where computing and sensing
technology is seamlessly embedded in the real world.
An alternative approach is through Virtual Reality (VR) [Burdea
and Coi et, 2003] (Figure 2.1(b)) when the user wears a head mounted
display and their view of the real world is completely replaced by
79
Figure 2.2: Typical Virtual Reality system with an immersive head mounted display,
data glove, and tracking sensors. [Lee et al., 2002]
computer-generated graphics (Figure 2.2). In a VR system the user
is completely separated from the real world, isolated in their head
mounted display, and so the computer is again invisible to the user.
In contrast, AR interfaces are designed to enhance interactions in the
real world (Figure 2.1(d)).
As can be seen from Rekimoto’s diagram, Augmented Reality is
complimentary to immersive Virtual Reality (VR). Many of the component
technologies used are the same, such as head mounted displays,
tracking systems, and handheld input devices. However there are some
important di erences between AR and VR systems. The main goal
of a Virtual Reality system is to use technology to replace reality and
create an immersive environment. In contrast, the main goal of an Augmented
Reality system is to enhance reality with digital content in a
non-immersive way. This means that in a VR system the display device
should be fully immersive with a wide ﬁeld of view (FOV), and
the 3D graphics shown as realistic as possible. The ﬁeld of view is the
amount of the users visual space ﬁlled with computer graphics and so
the greater the FOV the higher the level of immersion. Since the user
can no long see the real world, viewpoint tracking in the VR system
does not have to be very accurate relative to the real world.
80 Deﬁnition and Taxonomy
Table 2.1: Virtual Reality and Augmented Reality technology requirements.
Virtual Reality Augmented Reality
Replacing Reality Augmenting Reality
Scene Generation requires realistic images minimal rendering okay
Display Device fully immersive, wide FOV non-immersive, small FOV
Tracking and Sensing low accuracy is okay high accuracy needed
In contrast, in an AR system the display can be non-immersive
with a smaller FOV and use minimal graphics. For example, wearable
displays often have a FOV of less than 30 degrees, but some AR navigation
application applications can work well on these displays by using
very simple map and arrow graphics. However, in an AR application,
the tracking must be as accurate as possible to create the illusion that
the virtual content is ﬁxed in the real world. In a see-through AR display
it it very easy for the human eye to perceive a mismatch between
real and virtual graphics of even a few millimeters. Table 2.1 shows the
complimentary di erences between AR and VR systems.
Another way of deﬁning Augmented Reality is in the context of
other technologies. Milgram and Kishino [1994] introduced the concept
of "Mixed Reality", which is the merging together of real and virtual
worlds, and a Mixed Reality continuum which is a taxonomy of the
various ways in which the "virtual" and "real" elements can be combined
together (see Figure 2.3). On the right end is the Virtual Environment
(VE), where the user’s view of the world is completely replaced by
computer generated virtual content. On the opposite left end is the
Real Environment (RE) where none of the user’s view is replace by
virtual content. Towards the VE end is Augmented Virtuality where
most of the user’s view is replaced by computer graphics, but there
is still a view of the real world available. Finally, Augmented Reality
is closer to the RE end, where virtual cues enhances the user’s view
of the real world. As more or less virtual content is added to the AR
scene the interface moves closer or further away from the VE or RE
endpoints. The main lesson from this taxonomy is that AR interfaces
don’t exist as a discrete point between Real and Virtual experiences,
but can appear anywhere along the Mixed Reality continuum.
81
Figure 2.3: Milgram’s Mixed Reality continuum. [Milgram and Kishino, 1994]
Figure 2.4: Mann’s Mediated Reality. [Mann, 1994]
Milgram’s Mixed Reality continuum is a one-dimensional array from
the Real Environment to the Virtual Environment. However this can
be extended along a second dimension. Mann [1994] deﬁnes the concept
of Mediated Reality as ﬁltering or modifying the view of the real
world, rather than just adding to it as is done with Augmented Reality.
For example, warping video of the real world to compensate for visual
disability, or removing unwanted advertising billboards. Mann’s concept
of Mediated Reality extends the earlier deﬁnitions of AR, VR and
MR as shown in Figure 2.4. A Mediality Continuum [Mann, 2002] can
be constructed to compliment Milgram’s Mixed Reality (or Virtuality
Continuum), see Figure 2.5. In this case the vertical axis represents
the amount of mediation or ﬁltering that is being performed in the
user view of the real or virtual environment. For example, a severely
mediated virtuality application would be a VR system in which the
82 Deﬁnition and Taxonomy
Figure 2.5: Mann’s Mediality/Virtuality Continuum. [Mann, 2002]
user’s view was ﬁltered in some way. Like Milgram, Mann extends the
concept of Augmented Reality and places it in the context of other
interface technologies.
The Metaverse roadmap1[Smart et al., 2007] presents another way
of classifying the AR experience. Neal Stephenson’s concept of the
Metaverse [Stephenson, 1992] is the convergence of a virtually enhanced
physical reality and a physically persistent virtual space. Building on
this concept the Metaverse roadmap is based on two key continua (i) the
spectrum of technologies and applications ranging from augmentation
to simulation; and (ii) the spectrum ranging from intimate (identityfocused)
to external (world-focused). These are deﬁned as follows:
• Augmentation: Technologies that add new capabilities to existing
real systems
• Simulation: Technologies that model reality
• Intimate: Technologies focused inwardly, on the identity and actions
of the individual External: Technologies are focused outwardly,
towards the world at large
The technologies of Augmented Reality, Virtual Worlds, Life Logging,
and Mirror Worlds can be arranged within these continua (see Figure
2.6). They represent another way of classifying AR, alongside other
1
http://metaverseroadmap.org
83
Figure 2.6: Augmented Reality in the Metaverse taxonomy.
http://www.metaverseroadmap.org/overview
technologies such as Lifelogging [Achilleos, 2003] and Mirror Worlds
[Gelernter, 1991]. Lifelogging interfaces are those used by a person
to capture their everyday experience, such as the SenseCam [Hodges
et al., 2006] camera that continuously records audio and video from the
wearer. In this way they combine intimacy and augmentation. Mirror
World technologies are those that are externally focused and try to
create a copy or simulation of the real world. For example, Google’s
Streetview2 provides a panorama view of a street or location at a particular
point in time.
More recently, a number of other AR classiﬁcation schemes have
been proposed. For example, Hugues et al. [2011] presents a taxonomy
of AR environments based on functional purpose. They divide
environments into two distinct groups, the ﬁrst concerned with augmented
perception of reality, and the second with creating an artiﬁcial
environment. Similarly, Braz and Pereira [2008] present TARCAST, a
2
https://www.google.com/maps/views/streetview
84 Deﬁnition and Taxonomy
taxonomy based on the idea that any AR system can be made of six subsystems
that can characterized using existing taxonomies. These subsystems
include; Image Acquisition, Virtual Model Generator, a Mixing
Realities Subsystem, Display, Real Manipulator, and a Tracking Subsystem.
Finally, Normand et al. [2012] provides an AR taxonomy based
on four axes; (1) tracking required, (2) augmentation type, (3) content
displayed by the AR application, and (4) non-visual rendering modalities.
These taxonomies are not as well established as the earlier work of
Azuma, Milgram and Mann, but provide alternative perspectives from
which to characterize AR experiences.
In summary, Ron Azuma provides a useful deﬁnition of the characteristics
of Augmented Reality, which can help specify the technology
needed to provide an AR experience. However, to fully understand the
potential of AR it is important to consider it in the broader context
of other taxonomies such as Milgram’s Mixed Reality continuum, the
Metaverse Taxonomy or Mann’s Augmented Mediation. In the next
section we review the history of Augmented Reality, and show that
researchers have been exploring AR for many years.
3
History
Although Augmented Reality has recently become popular, the technology
itself is not new. For thousands of year people have been using mirrors,
lenses and light sources to create virtual images in the real world.
For example, beginning in 17th Century, theatres and museums were
using large plates of glass to merge reﬂections of objects with the real
world in an illusion that became known a "Pepper’s Ghost" [Brooker,
2007]. However, the ﬁrst truly computer generated AR experience can
be traced back to computer interface pioneer, Ivan Sutherland.
Ivan Sutherland is well known for developing Sketchpad, the
world’s ﬁrst interactive graphics application at MIT in 1963 [Sutherland,
1964]. Shortly after that he moved to Harvard University, and
in 1968, with Bob Sproull, he created the ﬁrst prototype AR system
[Sutherland, 1968] (see Figure 3.1). This combined a CRT-based optical
see-through head mounted display, with a ceiling mounted mechanical
tracking system connected to a PDP-11 computer and custom graphics
hardware. Later the cumbersome mechanical tracker was replaced by
an ultrasonic system. Thus their system combined the necessary display,
tracking and computing components to provide an AR experience.
85
86 History
Figure 3.1: Sutherland’s AR system. [Sutherland, 1968]
Although primitive, the system was capable of creating threedimensional
graphics that appearing to be overlaid on the real world.
In this paper, Sutherland says: "The user has a 40 degree ﬁeld of view
of the synthetic information displayed on the miniature cathode ray
tubes. Half-silvered mirrors in the prisms through which the user looks
allow him to see both the images from the cathode ray tubes and objects
in the room simultaneously. Thus displayed material can be made
either to hang disembodied in space or to coincide with maps, desk
tops, walls, or the keys of a typewriter." [Sutherland, 1968].
Sutherlands vision was of an Ultimate Display [Sutherland, 1965]
in which "..the computer can control the existence of matter. A chair
displayed in such a room would be good enough to sit in. Handcu s
displayed in such a room would be conﬁning, and a bullet displayed
in such a room would be fatal. With appropriate programming such a
display could literally be the Wonderland into which Alice walked." His
Augmented Reality research was a ﬁrst step in this direction.
Sutherland’s work went on to play a signiﬁcant role in inﬂuencing
virtual reality researchers, but for the next couple of decades most AR
research continued in military and government research labs, rather
than academic or industrial settings.
While Sutherland was developing his prototype Ultimate Display,
Tom Furness at the Wright Pattern Air Force based was beginning research
on the Super-Cockpit program for the US Air Force [Furness III,
1969]. From the mid-sixties until the mid-eighties, Furness and others
were working on new ways to present complex ﬂight details to pilots so
87
that they wouldn’t get overloaded with information. The SuperCockpit
was envisioned as a ".. crew station which would exploit the natural
perceptual, cognitive and psychomotor capabilities of the operator"
[Furness III, 1986].
In the prototype developed the user wore a head mounted display
in the aircraft cockpit (see Figure 3.2(a)). During daylight conditions,
computer graphics appeared superimposed over part of the pilot’s real
world (Figure 3.2(b)). During night and in bad weather conditions the
display could show graphics appears as a "substitute" for the real world,
replacing the user’s view. These early experiments evolved into the
head mounted AR displays used in modern aircraft such as the helmet
targeting system in the Apache attack helicopter. The early HMD
developments of Sutherland and Furness were continued by various government
organizations and private companies, such as Honeywell and
Hughes to produce a wide variety of designs.
(a) User wearing the
head mounted display
(b) Artist’s drawing of the view seen
through the display
Figure 3.2: The US Air Force Super Cockpit system. [Furness III, 1986]
Inspired by Furness, in 1981 the National Aeronautics and Space
Agency (NASA) built their own HMD from LCDs taken out of cheap
Sony Watchman TV’s. This was called the Virtual Visual Environment
Display (VIVED) and was connected to a magnetic tracking system
and graphics computer to create a fully immersive virtual environment.
88 History
This became the VIEW (Virtual Interface Environment Workstation)
project and was used to explore possible user interface for astronauts
and immersive visualizaton [Fisher et al., 1987]. A key part of this
system was a glove interface developed by VPL Research, a company
founded by Jaron Lanier and Jean-Jacques Grimaud in 1985 to provide
Virtual Reality technology [Kalawsky, 1993]. Lanier coined the term
"Virtual Reality" and in the middle and late 1980s VPL became famous
for making the DataGlove glove and the EyePhone HMD. This meant
that by the late 80’s, technologies were becoming available to enable
people to conduct AR research in academic and industrial settings.
Furness left the Air Force in 1989 to found the inﬂuential Human
Interface Technology Laboratory at the University of Washington,
transferring key Virtual and Augmented Reality technology into
an academic setting. Other key academic research groups studying Virtual
and Augmented Reality around that same time included Frederick
Brooks’ group at the University of North Carolina, Steve Feiner’s group
at Columbia University, and Paul Milgram’s group at the University
of Toronto. The UNC group conducted early research in see-through
head mounted displays [Rolland et al., 1995a, Kancherla et al., 1996],
registration and tracking techniques [Azuma, 1993, Azuma and Bishop,
1994] and medical applications [Bajura et al., 1992, Fuchs et al., 1996]
(see Figure 3.3). Feiner published one of the ﬁrst academic papers on
AR, describing the KARMA, a knowledge-based AR system [Feiner
et al., 1993b]. His group also researched interaction methods [Feiner
et al., 1993a], and developed the ﬁrst mobile AR systems[Feiner et al.,
1997]. Milgram was conducting research on overlaying virtual cues on
stereo video [Drascic and Milgram, 1991, Drascic et al., 1993], using AR
to enhance tele-operation [Zhai and Milgram, 1992], and AR methods
to enahance human robot interaction [Milgram et al., 1993].
At the same time, researchers Dave Mizell and Tom Caudell at
Boeing were exploring the use of Augmented Reality in an industrial
setting. They were trying to solve the problem of how to enable workers
to more e ciently create wire harness bundles, and developed an
AR system that showed which real wires should be bundled together
through the use of AR cues. Tom Caudell published the ﬁrst academic
89
(a) AR tracking and registration
[Azuma and Bishop, 1994]
(b) Medical AR system
[Fuchs et al., 1996]
Figure 3.3: AR researches at the University of North Carolina.
paper with the term "Augmented Reality" in it [Caudell and Mizell,
1992] and is often credited with coining the phrase. This research led
to a number of other projects at Boeing using AR, and a number of
other companies began researching industrial applications.
By the mid 1990’s research was well underway on some of the key
enabling technologies for AR, such as tracking, display and interaction.
Using these technologies a number of interesting application areas were
being explored. For example, Rekimoto [Rekimoto, 1996b], Billinghurst
[Billinghurst et al., 1996, 1997] and Schmalstieg [Schmalstieg et al.,
1996, Szalavári et al., 1998] were exploring how AR could be used
to enhance face-to-face collaboration, developing systems that allowed
people in the same location to see and interact with shared AR content
(see Figure 3.4). A range of di erent medical applications of AR were
being explored, such as for visualization of laparoscopic surgery [Fuchs
et al., 1998], X-ray visualization in the patient’s body [Navab et al.,
1999], and for image guided surgery [Leventon, 1997]. Azuma et al.
[2001] provides a good summary of key applications being researched
before 2000.
One related area of research emerging at the same time was wearable
computing. After several decades of exploration by early inventors,
researchers at CMU [Smailagic and Siewiorek, 1994] and MIT [Starner,
90 History
(a) Transvision [Rekimoto 96]
(b) Studierstube [Schmalsteig 96]
Figure 3.4: Early collaborative AR systems.
1996], began developing computer systems that could be worn all day.
With a wearable computer, a head mounted display is an obvious choice
for information presentation, and so the wearable systems are a natural
platform for Augmented Reality. Starner et al. [1997] showed how
computer vision based tracking could be performed on a wearable computing
platform, and used as the basis for AR overlay on real world
markers. Feiner et al. [1997] combined wearable computers with GPS
tracking to produce a number of outdoor AR interfaces for showing
information in place in the real world. Thomas et al. [1998] also investigated
terrestrial navigation application of outdoor wearable AR
interfaces, and later demonstrated using them for viewing and creating
CAD models on-site [Thomas et al., 1999, Piekarski and Thomas, 2003],
91
(a) Feiner’s Touring Machine [Feiner et al., 1997]
(b) Thomas and Piekarski’s wearable AR systems in 1998, 2002, and 2006.
http://www.tinmith.net/backpack.htm
Figure 3.5: Early wearable AR systems.
and for game playing [Thomas et al., 2002, Piekarski and Thomas,
2002a]. These wearable systems were typically very bulky due to the
size of the custom computer hardware, batteries, tracking sensors and
display hardware although they eventually became more wearable (see
Figure 3.5).
As discussed in the next section, one of the most signiﬁcant challenges
of Augmented Reality is viewpoint tracking. Early AR systems
such as those of Azuma [Azuma, 1993, Azuma and Bishop, 1994] and
Rekimoto [Rekimoto and Nagao, 1995] used expensive magnetic tracking,
or complex computer vision tracking based on InfraRed LEDs
[Bajura et al., 1992]. In 1996 Jun Rekimoto developed a simple computer
vision tracking system based on a printed matrix code [Rekimoto,
1996a, 1998], that evolved into CyberCode [Rekimoto and Ayatsuka,
92 History
(a) CyberCode
[Rekimoto and Ayatsuka, 2000]
(b) ARToolKit
[Kato et al., 2000]
Figure 3.6: Computer vision-based AR marker tracking.
2000], printed tracking pattern software that provided both marker
identiﬁcation and real time camera pose estimation (see Figure 3.6).
Soon after, Kato and Billinghurst developed the ARToolKit tracking
library that also provided real time computer vision tracking of a square
marker [Kato and Billinghurst, 1999]. ARToolKit1 was released as open
source software in 2000 and became one of the most widely used AR
tracking libraries, making it signiﬁcantly easier for researchers and developers
to build their own AR applications. ARToolKit was important
because it solved two fundamental problems; (i) tracking the user’s
viewpoint, (ii) enabling interaction with real world objects.
In addition to wearable AR, researchers were also exploring handheld
AR systems. The ﬁrst handheld AR interfaces involved LCD
screens tethered to desktop computers that provided the tracking and
graphics generation. For example, Fitzmaurice’s Chameleon system allowed
a person to move an LCD display over a piece of paper and see
virtual content [Fitzmaurice, 1993], and Rekimoto’s NaviCam [Rekimoto
and Nagao, 1995] superimposed virtual tags on real objects using
handheld and headworn displays (see Figure 3.7(a)). By 2003 researchers
such as Geiger [Geiger et al., 2001] and Assad [Assad et al.,
2003] were exploring wireless data transfer of images from handheld
1
http://artoolkit.sourceforge.net
93
(a) NaviCam
[Rekimoto and Nagao, 1995]
(b) PDA based AR
[Wagner and Schmalstieg, 2003]
(c) Mobile phone AR [Mohring et al., 2004]
Figure 3.7: Handheld Augmented Reality.
PDAs to a remote PC for image processing and AR overlay. Handheld
devices eventually became powerful enough to run AR computer
vision tracking algorithms, and the ﬁrst self contained handheld AR
application was demonstrated by Wagner et al. in 2003 [Wagner and
Schmalstieg, 2003]. Mobile phones received the ﬁrst cameras in 1997
and the ﬁrst mobile phone based AR application was demonstrated by
Mohring and Bimber in 2004 [Mohring et al., 2004] (see Figure 3.7(c)).
This was followed soon after by the porting of ARToolKit over to mobile
phones [Henrysson and Ollila, 2004]. These developments meant
that for the ﬁrst time millions of people had technology in their pocket
that would allow them to have an Augmented Reality experience.
Some of this early research was spurred on by several signiﬁcant
large national research projects in AR. One of the most important was
the Mixed Reality Systems Laboratory in Japan, a research laboratory
94 History
Figure 3.8: ARVIKA AR product maintenance. [Weidenhausen et al., 2003]
that ran from 1997 to 2001 as a joint venture between Canon and
the Japanese Government. The lab received over $50 million USD for
research into Mixed Reality technologies for four years and had a sta of
around 30 researchers. The focus of the research was on technologies for
3-D imaging and display for Mixed Reality systems, content creation,
tools for real time seamless fusion of the physical space and cyberspace,
and other related topics. For example, the AR2Hockey AR interface
they developed [Ohshima et al., 1998] was one of ﬁrst collaborative
AR systems. Since 2001 Canon took over research for Mixed Reality
technology, and continues to conduct research in this area, targeting
manufacturing applications.
In Germany, signiﬁcant research on the application of AR technologies
for manufacturing was conducted through the ARVIKA consortium2.
This was a German government supported research initiative
with 23 partners from the automotive and airplane industry, manufacturing,
SMEs, universities and other research providers. The project
ran for four years from 1999 to 2003 with the focus on developing AR
technology that could improve manufacturing in the automobile and
aerospace industries. Figure 3.8 shows a prototype product maintenance
application developed in the project. Weidenhausen et al. [2003]
provides an excellent summary of the lessons learned and overview of
the key research outputs.
In 1998 the ﬁrst research conference dedicated to Augmented Reality
began, the IEEE/ACM International Workshop on Augmented Re-
2
http://www.arvika.de/www/index.htm
95
ality (IWAR). This was followed by the International Symposium on
Mixed Reality (ISMR) from 1999, and the International Symposium
on Augmented Reality (ISAR) in 2000. All of these meetings combined
into the International Symposium on Mixed and Augmented Reality
(ISMAR)3 from 2002, which remains the leading research conference
in AR, where the most prominent researchers present their latest work.
The development of PC based AR tracking also led to the creation
of the ﬁrst dedicated AR companies. The ﬁrst of these was Total Immersion4,
founded in 1998, a French company that initially focused on
providing AR for location based experiences and events and marketing
campaigns. ARToolworks5 was established in 2001 to commercialize
the ARToolKit software and became the ﬁrst AR company focusing on
proving AR tools for developers. Metaio6 was created in 2003 from an
initial industrial AR project with Volkswagen in the ARVIKA project,
and has since grown to provide a AR platform for both desktop and
mobile AR applications. All three of these pioneering companies are
still in business, and have since been joined by dozens of others in the
growing commercial space.
At this time most AR experiences either existed in the research
laboratory or in museums or theme parks. The technology required
specialized display and tracking hardware and high end computing to
provide a good user experience. For example, the ARCO project [Wojciechowski
et al., 2004] explored how AR technology could be used to
show virtual artifacts in museums. The Virtual Showcase project [Bimber
et al., 2001b] demonstrated how projection tables could be used to
create high-end AR enhanced shared experiences (see Figure 3.9), and
the HIT Lab NZ’s Black Magic AR kiosk was seen by over 250,000
people at the 2005 America’s Cup [Woods et al., 2004]. A number of
theme parks opened AR enhanced rides, such as the AR Adventure
Game opened in the Guandong Science and Technology Center, China
using wide area tracking technology described in [Huang et al., 2009]
and see-through HMDs described in [Weng et al., 2009].
3
http://ismar.vgtc.org
4
http://www.t-immersion.com
5
http://www.artoolworks.com
6
http://www.metaio.com
96 History
(a) ARCO
[Wojciechowski et al., 2004]
(b) Virtual Showcase
[Bimber et al., 2001b]
Figure 3.9: Museum and theme park AR.
One of the ﬁrst widely available consumer AR experiences was the
Sony PlayStation 3 game The Eye of Judgement7, released in October
2007. This card-based battle game used the PS-3 camera and a derivative
of Sony’s original CyberCode8 to show AR characters on physical
playing cards. The game sold over 300,000 copies, making it the
most widely used AR experience at the time. Other consumers experienced
AR technology through virtual enhancements on live broadcast
TV, with companies such as PVI/Sportsvision9 providing AR cues on
sports broadcasts. For example, the "yellow-line" that appeared in US
football games to show the ﬁrst down line, or the virtual ﬂags that
appeared behind swimmers in the Olympics to identify their country.
In this case, the broadcast cameras were ﬁxed and careful calibration
could be performed to create the illusion that the virtual content is
part of the real world. A similar approach was used to create virtual
billboards and advertising that could be included into live footage.
The year 2009 represented a turning point for Augmented Reality
as it moved from the research laboratory and specialist applications, to
widely available commercial technology. Figure 3.10 shows the Google
Trends10 graph of Google searches that contain the term "Augmented
7
http://www.playstation.com/en-us/games/the-eye-of-judgment-ps3
8
http://www.sonycsl.co.jp/en/research_gallery/cybercode.html
9
http://www.sportvision.com
10
https://www.google.com/trends
97
Figure 3.10: Relative Google search terms: "Virtual Reality" in blue and "Augmented
Reality" in red. https://www.google.com/trends
Reality" compared to "Virtual Reality". Since 2004 there was a steady
decrease in searches on Virtual Reality, while activity in Augmented
Reality remained low. Then interest in Augmented Reality quickly rose
in 2009, passing Virtual Reality searches by June and being twice as
popular by the end of the year. This was mainly due to three factors (1)
the emergence of Flash-based AR, (2) smart phone based AR, and (3)
use of AR for global marketing campaigns. These combined to mean
that from 2009 onwards billions of people have had the technology to
have an AR experience and many millions have had one. More recently,
in June of 2013 searches for VR passed AR, as the Oculus Rift and other
cheap HMDs revived interest in Virtual Reality.
Flash based AR arose at the end of 2008 when Adobe added camera
support to its popular Flash platform. A pair of Japanese developers
nick-named Sqoosha and Nyatla ported the ARToolKit library to Flash
creating FLARToolKit , and for the ﬁrst time people could have an AR
experience from within their web browsers. This lead to a large number
of web-based AR experiences. For example, the GE Smart Grid website
in early 2009 allowed people to see a virtual windfarm displayed over
a printed marker with windmill blades that spun faster as people blew
on them11. Developing for Flash is signiﬁcantly easier than writing
native C or C++ applications and so FLARToolKit enabled many more
developers to begin creating AR experiences.
A second important factor was the rise of smart phone. Although
early mobile phone based experiences were shown several years earlier
11
https://www.youtube.com/watch?v=NK59Beq0Sew
98 History
Figure 3.11: Wikitude AR browser showing AR tags over the real world.
these were run on phones that were di cult to develop for, had slow
processing and graphics power, and limited sensors. The launch of the
iPhone in 2007 provided a smart phone platform that was easy to develop
for, with a processor fast enough for real time computer vision
tracking and powerful 3D graphics. However it was the release of the
ﬁrst Android phone in October 2008 that provided a signiﬁcant boost
to mobile AR. The Android platform combined the camera and graphics
of the iPhone with GPS and inertial compass sensors, creating the
perfect platform for outdoor AR. Taking advantage of this platform,
Austrian company Mobilizy released the Wikitude12 AR browser for
Android devices in late 2008. Wikitude allowed users to see virtual
tags overlaid on live video of real world, providing information about
points of interest surrounding them (see Figure 3.11). Since that time
a number of other AR browsers have been released, such as Sekai Camera13,
Junaio14, and Layar15, and have been used by tens of millions of
people.
The third factor contributing to the rise in popularity and awareness
of AR was use of the technology in advertising. AR provides a
12
http://www.wikitude.com
13
http://www.sekaicamera.com
14
http://www.junaio.com
15
http://www.layar.com
99
(a) Zoo campaign (b) Living Sasquatch
Figure 3.12: AR in advertising.
compelling, almost ’magical’ experience that grabs user’s attention in
a way that makes it easy to draw attention to the marketing message.
The HIT Lab NZ created one of the world ﬁrst mobile AR campaigns
in 2007 where they worked with Saatchi and Saatchi to show virtual
animals appearing over a printed newspaper advertisement for a local
zoo (see Figure 3.12(a)). Since that time hundreds of mobile campaigns
have been created for companies such as Nike, Coke, Ford and others.
Similarly, many online advertising campaigns have been run. For example,
Bo swana produced a virtual Sasquatch that users could animate
and place in the real world16, and found that people would spend over
three times as long on the website compared to a traditional website
(see 3.12(b)). AR enhanced print media also became increasingly common.
The December 2009 edition of the Esquire magazine used AR17
to provide virtual content overlaid on twelve of its pages, including a
virtual video of Robert Downey-Jr on its cover. Following that there
have been many examples of printed advertisements or article with AR
content in magazines (e.g. Red Bull Bulletin, Colors, Wired), or even
entire books.
At the time of this article writing (2015) the AR industry is growing
strongly. There are a number of platforms available that make it pos-
16
https://vimeo.com/4233057
17
https://www.youtube.com/watch?v=LGwHQwgBzSI
100 History
Figure 3.13: AR market prediction on global AR users by Tomi Ahonen (2014).
sible for users to easily access the technology, ranging from the web,
to smart phones, and head worn displays such as Google Glass18. It
is easier than ever before to develop AR applications with free available
tracking libraries such as ARToolKit and Qualcomm’s Vuforia19,
and authoring tools like BuildAR20 that make is possible for even nonprogrammers
to create AR experiences. This ease of access has lead
to the use of AR technology in one form or another by hundreds of
companies, and a fast growing commercial market. Industry analysts
have predicted that the mobile AR market alone will grow to be a more
than $6 billion USD by 2017 with an annual growth rate of more than
30%21. Comparing growth of AR users to early cell phone adoption,
Tomi Ahonen has predicted that by 2020 there will be more than a
Billion AR users22 (see Figure 3.13).
18
http://glass.google.com
19
http://www.vuforia.com
20
http://www.buildar.co.nz
21
http://www.juniperresearch.com
22
http://www.tomiahonen.com
101
Figure 3.14: Gartner’s 2014 Hype Cycle for Emerging Technologies.
Every year the Gartner group provides a graph of its Hype Cycle23,
predicting where certain technologies are on the path to mass-market
adoption and how long it is before they are widely accepted. As Figure
3.14 shows, in July 2014 they were predicting that AR was about
to reach the lowest point on the Trough of Disillusionment, suggesting
a coming period of retrenchment in the commercial sector. However
they are also forecasting that it will only take 5 to 10 years to reach
the Plateau of Productivity where the true value of the technology is
realized.
In summary, as can be seen, the history of AR research and development
can be divided into four phases:
1) Pre-80’s: Experimentation: Early experimentation that help deﬁne
the concept of Augmented Reality and show the types of
technology require
2) 1980’s ~ mid-90’s: Basic Research: Research into enabling technologies
such as tracking, displays, and input devices
23
http://www.gartner.com/technology/research/hype-cycles
102 History
3) mid 1990’s ~ 2007: Tools/Applications: Using AR enabling technologies
to develop early applied and explore interaction techniques,
usability, and design theory
4) 2007 ~ present day: Commercial Applications: Widespread AR
available in a number of application areas such as gaming,
medicine, mobile and marketing
Over the last 50 years the technology has moved from being buried
in Government and Academic research labs to the verge of widespread
commercial acceptance. It is clear that the near future will bring exciting
development in this fast growing ﬁeld.
In the remainder of this survey we will ﬁrst review basic enabling
technologies such as tracking and displays, and then discuss interaction
techniques, design guidelines for AR experiences, and evaluation methods.
Finally we close with a review of important areas for research in
Augmented Reality and directions for future work.
4
AR Tracking Technology
As discussed in section 2, for a system to meet Azuma’s deﬁnition of
Augmented Reality system [Azuma, 1997], it must fulﬁll three main
requirements:
1) It combines real and virtual content
2) It is interactive in real time
3) It is registered in 3D
The third requirement, "Registered in 3D", relates to the ability
of a system to anchor virtual content in the real world such that it
appears to be a part of the physical environment. This section focuses
on technologies that aim to fulﬁll this requirement.
In order to register virtual content in the real world, the pose (position
and orientation) of the viewer with respect to some "anchor" in the
real world anchor must be determined. Depending on the application
and technologies used, the real world anchor may be a physical object
such as a magnetic tracker source or paper image marker, or may be
a deﬁned location in space determined using GPS or dead-reckoning
from inertial tracking.
103
104 AR Tracking Technology
Depending on which technology is used, the process of registering
the system in 3D may be comprised of one or two phases; (1) a registration
phase, which determines the pose of the viewer with respect
to the real world anchor, and (2) a tracking phase, which updates the
pose of the viewer relative to a previously known pose. In accordance
to common terminology, in this document we use the term "Tracking"
to refer to the combined application of both phases.
The following sections explore some of the common tracking techniques
used for registering virtual content in the real world for the
purposes of Augmented Reality.
4.1 Magnetic Tracking
Magnetic trackers are devices that use properties of magnetic ﬁelds in
order to calculate the pose of a receiver with respect to a transmitter,
which is used as the real world anchor. In a magnetic tracker the
transmitter produces alternating magnetic ﬁelds, which are detected
by one or more receiving units. By measuring the polarization and orientation
of the magnetic ﬁeld detected, the pose of each receiver can
be calculated at high speed.
When used for augmented reality, the magnetic tracker transmitter
acts as the origin of the virtual coordinate system, and by mounting a
receiver to the viewer, the position and orientation can be determined
[Caudell and Mizell, 1992].
Magnetic trackers have high update rates, are invariant to occlusion
and optical disturbances, and the receivers are small and lightweight.
However, the strength of magnetic ﬁelds fall of with the cube of distance
and resolution falls o with the fourth power of distance [Nixon
et al., 1998], and thus magnetic trackers have a limited working volume.
Magnetic trackers also are prone to measurement jitter, and are
sensitive to magnetic materials and electromagnetic ﬁelds in the environment
[Bhatnagar, 1993]. Figure 4.1 shows the resolution of a Polhemus
magnetic tracking device as a function of the distance between
the transmitter and receiver. As can be seen, at up to 50 inches separation
the resolution of the position and orientation measurement is
4.2. Vision Based Tracking 105
Figure 4.1: Resolution of position and orientation measurements for a magnetic
tracker as a function of range between the transmitter and receiver. (Image courtesy
of Polhemus http://www.polhemus.com)
very low, but this resolution steadily degrades at an increasing rate as
the distance between transmitter and receiver increases.
Magnetic tracking technology has been used in a range of AR
systems, with applications ranging from manufacturing [Caudell and
Mizell, 1992] to maintenance [Feiner et al., 1993a] to medicine [Bajura
et al., 1992] (Figure 4.2). Further discussion of Magnetic Tracking approaches
can be found in Rolland et al.’s survey [Rolland et al., 2001].
4.2 Vision Based Tracking
We deﬁne vision based tracking as registration and tracking approaches
that determine camera pose using data captured from optical sensors.
These optical sensors can be divided into three main categories: Infrared
Sensors, Visible Light Sensors, and 3D structure sensors.
Vision based tracking for augmented reality has become increasingly
popular in recent times due to the minimal hardware requirements,
improved computational power of consumer devices, and the
ubiquity of mobile devices such as smart phones and tablets which
feature both a camera and screen, making them ideal platforms for
augmented reality technologies.
In the following sub sections, we discuss various vision based tracking
techniques, categorized by the sensor hardware that is used.
106 AR Tracking Technology
Figure 4.2: 2D slices of ultrasound data is overlaid on a patient’s abdomen. Both
the HMD and ultrasound transducer are tracked using a Polhemus magnetic tracker.
[Bajura et al., 1992]
4.2.1 Infrared Tracking
Some of the earliest vision based tracking techniques employed the use
of targets which emitted or reﬂected light, which made detection easy
due to their high brightness compared to the surrounding environment
[Bajura and Neumann, 1995]. The targets, which emitted their own
light, were also robust to adverse illumination e ects such as poor ambient
lighting or harsh shadows. These targets could either be attached
to the object which was being tracked with the camera external to the
object, known as "outside-looking-in" [Ribo et al., 2001], or external
in the environment with the camera mounted to the target, known as
"inside-looking-out" [Gottschalk and Hughes, 1993].
With comparable sensors, the inside-looking-out conﬁguration offered
a higher accuracy of angular orientation, as well as greater
resolution than that of the outside-looking-in system (3). A number
of systems were developed using this inside-looking-out conﬁguration
[Azuma and Bishop, 1994, Welch et al., 1999, Ward et al., 1992, Wang
et al., 1990], typically featuring a head mounted display with external
facing camera, and infrared LEDs mounted on the ceiling (see Figure
4.3). The LEDs were strobed in a known pattern, allowing for
recognition of individual LEDs and calculation of the position and orientation
of the user’s head in the environment. The main disadvantages
4.2. Vision Based Tracking 107
Figure 4.3: Conceptual drawing of inside-looking-out system (Left). Roof mounted
infrared led arrays (Middle) and the Hi-ball camera tracker (Right). [Welch et al.,
1999]
of the inside-looking-out conﬁguration were the considerable setup required
to add all the LED sources to the environment and the additional
weight added to the wearable device from the camera.
A major disadvantage of using light emitting targets is the required
power and synchronization of the light sources. Many systems swapped
light emitting targets in favour of retro-reﬂective markers [Dorfmüller,
1999], which reﬂect back infrared light provided from an external light
source, typically located by the camera.
While infrared tracking systems are scalable and o er precision and
robustness to illumination e ects, they are complex, expensive, and
invasive due to the physical infrastructure required.
4.2.2 Visible Light Tracking
Visible light sensors are the most common optical sensor type, with
suitable cameras being found in devices ranging from laptops to smartphones
and tablet computers, and even wearable devices. For videosee-through
AR systems, these sensors are particularly useful as they
can be both used for the real world video background that is shown
to the users, as well as for registering the virtual content in the real
world.
Tracking techniques that use visible light sensors can be divided
into three categories: Fiducial tracking, Natural Feature tracking, and
Model Based tracking. We discuss these techniques in the following sub
sections.
108 AR Tracking Technology
Figure 4.4: Colour sticker ﬁducial landmarks (left), Multi-Ring ﬁducial landmarks
(right). [Cho et al., 1997, 1998]
Fiducial Tracking
We deﬁne ﬁducials as artiﬁcial landmarks that are added to the environment
to aid in registration and tracking. Depending on the application
and the technology used, the complexity of ﬁducials varies considerably.
Early systems typically used small coloured LEDs or pieces of paper
(see left of Figure 4.4), which could be added to the environment and
detected using colour matching [Neumann and Cho, 1996]. The pose of
the camera could be determined if enough ﬁducials of known position
were detected in the scene. This approach had an additional beneﬁt
that the workspace could be dynamically extended by introducing additional
ﬁducials to the scene at runtime and then estimating their
position based on the position of other known ﬁducials [Cho et al.,
1997, Neumann and Park, 1998]. Later works expanded on the concept
and complexity of ﬁducials, adding features like multi-rings to allow for
the detection of ﬁducials at greater distances [Cho et al., 1998] (right
of Figure 4.4).
In order to calculate the pose of a viewer in the real world, a minimum
of four points of known position need to be located in the environment
[Fischler and Bolles, 1981]. With these simpler ﬁducials, the
design of the space had to be carefully considered to ensure enough
points were visible at any point in time, which required additional effort
to physically place the ﬁducials into the environment and record
their real-world positions. A simpler alternative which fulﬁlled the four
point requirement was to use a single planar ﬁducial featuring a quadrilateral
shape, where the corners of the quadrilateral serve as the known
reference points [Comport et al., 2006]. Additional identiﬁcation infor-
4.2. Vision Based Tracking 109
mation can then be encoded inside the quadrilateral, allowing for multiple
unique ﬁducials to be used in the same application [Rekimoto and
Ayatsuka, 2000].
The planar quadrilateral ﬁducial proved to be an extremely popular
technique for AR tracking due to its simplicity to use and high accuracy
of detection. One of the most popular planar ﬁducial systems was the
ARToolkit, which spawned a number of successors, and whose ﬁducial
design still informs most planar ﬁducial registration techniques used
today.
ARToolKit The ARToolkit, developed by Kato and Billinghurst in
1999, is a widely known and used ﬁducial marker based registration
system [Kato and Billinghurst, 1999]. The ﬁducial markers are characterized
by a thick black outlined square with a pattern inside for
identiﬁcation and orientation recognition.
The ARToolkit system uses binary thresholding and partial line
ﬁtting to determine potential edges of the marker (see Figure 4.5).
All regions which are enclosed by four line segments are considered
potential marker candidates, and are used to calculate a six degree of
freedom (6DOF) matrix transformation for the marker with respect to
the viewing camera. With this transformation known, the marker can
be rectiﬁed so that it appears parallel to the camera’s sensor plane, and
the internal pattern is matched against a database of known patterns.
This pattern matching allows to identify whether it is a valid marker,
which marker it is (in the case of multiple marker systems), and also to
determine the orientation of the marker around the marker’s vertical
axis.
Similar to simpler ﬁducials, multiple ARToolkit markers can be
spaced through the environment to extend the tracking range. Because
the 6DOF transformation of each marker relative to the camera can
be determined, the transformation between individual markers can be
calculated dynamically at run time [Uematsu and Saito, 2005], allowing
for dynamic extension of the tracking environment.
The ARToolkit system has been applied to a range of di erent
research areas such as Teleconferencing [Billinghurst and Kato,
110 AR Tracking Technology
Figure 4.5: The ARToolkit registration process. [Billinghurst et al., 2000]
Figure 4.6: The ARToolkit falsely identifying real world features as tracking markers
.[Fiala, 2004]
2000], wide area navigation [Wagner, 2002], and hand position tracking
[Piekarski and Thomas, 2002b], and ported to run on mobile devices
[Henrysson et al., 2005].
Despite the ARToolkit’s popularity, there were a number of shortcomings.
The partial line ﬁtting algorithm is highly susceptible to occlusion,
such that even a minor occlusion of the edge of the marker
will cause registration failure. Although it provides ﬂexibility in the
appearance of the markers, the pattern matching on the internal image
is prone to cause false positive matches, causing the AR content to
appear on the wrong markers, or even on non-marker, square shaped
regions in the environment, as shown in Figure 4.6.
ARTag The ARTag system [Fiala, 2005a] was developed to address a
number of the limitations of ARToolkit. In particular, the system was
designed to improve occlusion invariance, reduce the occurrence of false
4.2. Vision Based Tracking 111
Figure 4.7: Occlusion susceptibility of ARToolkit (Middle) compared to that of
ARTag (Right). [Fiala, 2004]
positive matches caused by inaccurate pattern matching, and improve
performance in challenging lighting. Figure 4.6 shows an example of the
ARToolKit falsely identifying visual features in the real environment
as correct ARToolKit markers.
In ARTag, the partial line ﬁtting algorithm was replaced by a
quadrilateral heuristic search algorithm, which allowed for breaks in
the continuity of the lines which deﬁne the edges of the marker. Figure
4.7 shows how this makes the tracking robust to edge occlusion in
the tracking image. In addition to improving robustness to occlusion,
this algorithm also improved the detection of markers in the presence of
poor lighting, shadows and specular lighting reﬂections. The occurrence
of false positive matches was reduced by replacing the image-based template
matching with a binary barcode system inside the marker outline.
Although reducing the ﬂexibility for the appearance of the markers,
the six by six binary grid allowed a greater accuracy in determining
the marker ID, and a built in checksum provided redundancy and error
correction to add further invariance to the e ects of poor lighting
or occlusion of the marker. When using multiple markers, the ARTag
system deﬁnes the order which markers should be used to maximize
the di erence between the markers, further reducing the likelihood of
false positive matches. Fiala [Fiala, 2005b, 2004] provided a very good
comparison between ARTag and ARToolKit and how ARTag provides
signiﬁcant improvements.
ARToolkitPlus and Studierstube ES ARToolkitPlus [Wagner and
Schmalstieg, 2007] was a ground up rebuild of the original ARToolkit,
focusing on performance optimization and various image processing
112 AR Tracking Technology
Figure 4.8: Markers supported by Studierstube ES: Template Markers, BCH Markers,
DataMatrix Markers, Frame Markers, Split Markers, and Grid Markers.
[Schmalstieg and Wagner, 2009]
improvements to provide a ﬁducial marker tracking system that could
run on the mobile devices of the time. The ARToolkitPlus followed the
example set by ARTag of using binary barcodes to improve marker
identiﬁcation.
ARToolkitPlus was later re-implemented from scratch and integrated
into the existing PC-based Studierstube collaborative AR environment
[Schmalstieg et al., 2002]. The implementation used in
Studierstube ES (Embedded Systems) [Schmalstieg and Wagner, 2007],
the handheld device component of the Studierstube environment,
supported several di erent marker conﬁgurations, including template
markers similar to the original ARToolkit markers, ARTag themed binary
markers, as well as Frame markers, which encoded all the identifying
information for the marker in the frame, allowing for more visually
pleasing markers. Figure 4.8 shows the range of di erent tracking makers
supported by Studierstube.
Natural Feature Tracking
The planar ﬁducials used in the ARToolkit, ARTag and ARToolkitPlus
frameworks overcame di culties faced when using simpler colour
ﬁducials, and the frame and split ﬁducial markers in Studierstube ES
allowed for more visually pleasing experience. However, all these ﬁducial
types still require modiﬁcation of the real environment, which may
not be desirable or even possible, particularly when the real world anchor
is valuable, such as a painting in an art gallery.
4.2. Vision Based Tracking 113
As the computational power of devices used for AR applications
improved, it became possible to register the pose of the camera, in realtime,
using features which already exist in the natural environment.
Complicated image processing algorithms are used to detect features
in the captured images which are unique in their surroundings, such as
points, corners, and the intersections of lines. For each of these features
a unique "descriptor" is calculated which allows for identiﬁcation and
di erentiation of each feature. By matching features detected in the
scene with those detected in the object to be tracked, the pose can be
computed using similar algorithms as those used in the ﬁducial marker
techniques.
In the following subsections, some of the more common natural
feature detection and description algorithms are discussed, including
SIFT, SURF, BRIEF, ORB, BRISK and FREAK.
SIFT The Scale Invariant Feature Transform (SIFT) is a natural feature
detector and descriptor which was inspired by the function of
neurons in the inferior temporal cortex of primates [Lowe, 1999]. First
published in 1999, it remains as one of the most referenced natural
feature algorithms, and is still commonly used as a benchmark when
evaluating new techniques.
SIFT allows for scale invariant detection of features using the concept
of scale spaces (see Figure 4.9). At each scale, known in the paper
as an "octave", images are convolved using a Gaussian operation,
and the di erence between consecutively processed images is found,
known as the Di erence of Gaussian (DoG). The DoG emphasizes important
structures in the image, and functions as an approximation of
the Laplacian of Gaussian, a common technique used to ﬁnd edges and
corners in an image. With the DoG calculated, pixels which are the
maxima/minima amongst neighbouring pixels in the same scale space
are selected as good feature candidates. Each feature is recorded with
the scale that it was detected at, and one or more orientations based
on local image gradient directions [Lowe, 2004], providing scale and
rotation invariance respectively.
114 AR Tracking Technology
Figure 4.9: Calculating the Di erence of Gaussian (DoG) images across di erent
scales (Left). Calculating the maxima and minima of a pixel and it’s neighbouring
pixels in the same scale space (Right). [Lowe, 2004]
For each feature detected, a descriptor is calculated for identiﬁcation
and di erentiation. The SIFT descriptor was modelled on the
function of complex neurons in the visual cortex, which are hypothesized
to allow for matching and recognition of 3D objects from a range
of viewpoints. SIFT calculates the gradient magnitudes and orientations
for all points in a 16x16 window around the feature point, with
the magnitudes Gaussian weighted relative to distance from the feature
point so that points further away from the feature point have less
impact on the descriptor. The 16x16 points are split into 4x4 regions,
and their gradient magnitudes placed into an 8 bin histogram based on
their gradient orientation. The resulting 8 histogram values are used
to calculate one component of the descriptor, resulting in a complete
SIFT descriptor of 128 values (4x4 regions x 8 histogram values per
region), which is then normalized for illumination invariance.
Figure 4.10 illustrates how the process works. A 16x16 window,
divided into 4x4 regions, is selected (Left). For every pixel in the region,
the gradient magnitude and orientation is calculated, and these are
then put into an 8 bin histogram based on the orientation (Middle).
Finally, a 128 dimensional vector descriptor is generated, based on the
8 histogram values for each of the 4x4 regions.
Comparison of feature points is done by calculating the distance
between their descriptors, either as the sum of square di erences between
each element, or using a nearest neighbour approach such as
Best-Bin-First [Beis and Lowe, 1997].
4.2. Vision Based Tracking 115
Figure 4.10: Calculating the SIFT Descriptor.
The main strength of SIFT is its invariance to a number of common
visual recognition challenges, including changes in marker scale,
rotation, perspective, as well as scene illumination. Unfortunately this
robustness comes at a cost, and the usefulness of SIFT for Augmented
Reality is limited due to the high computational requirements for feature
detection, feature description, and feature matching. Additional research
aimed to resolve this, including Fast Approximated SIFT [Grabner
et al., 2006], which uses approximations of the DoG and orientation
histograms algorithms to increase feature detection and feature
description speeds, and PCA-SIFT [Ke and Sukthankar, 2004], which
uses principal component analysis to reduce the descriptor length from
128 to 36 to increase feature matching speeds, however the algorithm
was still too computationally intensive to be practical for real time
natural feature tracking.
Although SIFT was unsuitable for real time natural feature tracking,
it served as inspiration for SURF, a natural feature detector and
descriptor capable of real-time performance.
SURF SURF, or Speeded Up Robust Features [Bay et al., 2006], is
a natural feature detector and descriptor algorithm that was designed
to be faster to compute than the state of the art algorithms, while
maintaining a comparable level of accuracy. The authors chose to focus
on invariance to scale and rotation only, leaving the general robustness
of the descriptor to minimize the e ects of other image transforms and
distortions.
116 AR Tracking Technology
Figure 4.11: Gaussian second order partial derivatives in y-direction and xydirection
(Left) and the SURF box ﬁlter approximations (Right). [Bay et al., 2008]
To reduce the computational complexity required for feature detection
and description, SURF features a number of approximations of
previously used techniques. The feature detector is based on the Hessian
matrix, with the determinant of the Hessian used to select for both
location and scale. The Gaussian ﬁlters used by SIFT to provide scale
space invariance were replaced by box ﬁlters operating on integral images,
as shown in Figure 4.11. This removes the need to create scale
spaces using image pyramids; instead box ﬁlters of di erent sizes can
be run on the original integral image.
The SURF feature descriptor was based on similar properties to
that of SIFT, but with reduced complexity. For each feature point,
x and y direction Haar-wavelet responses are calculated for a circular
neighbourhood around the point. The radius of circular neighbourhood
is relative to the scale that the feature was detected at, and the results
are Gaussian weighted by distance from the interest point. Using a
sliding orientation window, the vertical and horizontal responses are
summed, and the longest resulting vector is used as the orientation of
the keypoint.
The orientation information is then used to create an oriented
square region around each interest point, from which the descriptor
is computed. Haar wavelet responses, oriented to the orientation calculated
previously, are computed over 4x4 square sub-regions, and Gaussian
weighted by distance from the interest point, as shown in Figure
4.12. The wavelet responses dx dy are summed over each region,
and combined with the sum of the absolute value of the responses |dx|
|dy|, resulting in a four dimensional vector v = ( dx, dy, |dx|, |dy|)
for each of the 4x4 sub-regions, resulting in a descriptor vector of length
64.
4.2. Vision Based Tracking 117
Figure 4.12: The SURF Feature descriptor operates on an oriented square region
around a feature point. Haar-wavelets are computed over 4x4 sub-regions,
and each sub-region’s descriptor ﬁeld is comprised of the resulting responses
dx, dy, |dx|, |dy|. [Bay et al., 2008]
In the authors evaluation, the number of features found by the
SURF detector was on average very similar to that of SIFT and other
feature detectors. The repeatability scores of SURF were comparable
or better, while being three to ten times faster than the competition
[Bay et al., 2008]. The SURF descriptor outperformed other tested
descriptors in recall, and was up to approximately 3 times faster than
SIFT [Bay et al., 2006].
BRIEF The speed and robustness of SURF allowed for real-time natural
feature based registration for augmented reality. However, the complexity
of SURF was still too great for real time performance on mobile
devices with limited computational power.
BRIEF [Calonder et al., 2010], the Binary Robust Independent Elementary
Features descriptor, attempted to reduce the computational
time by reducing the dimensionality of the descriptor and changing
from a rational number descriptor to a binary descriptor. These changes
signiﬁcantly sped up matching, which previously had to be done iteratively
or using techniques like approximate nearest neighbours [Indyk
118 AR Tracking Technology
and Motwani, 1998], but could now be performed using the Hamming
distance, which can be computed very quickly on modern devices. An
additional beneﬁt of the smaller descriptors was that they reduced the
amount of memory required by the algorithms, making it more suitable
for mobile devices where memory capacity is often less than on a
desktop PC.
The BRIEF descriptor was inspired by feature classiﬁers, which
transform the challenge of feature description and matching into one of
classiﬁcation. For the feature classiﬁer, every feature in the marker
underwent a number of pairwise pixel-intensity comparisons, which
were used to train Classiﬁcation trees [Lepetit and Fua, 2006] or Naive
Bayesian classiﬁers [Ozuysal et al., 2010] as an o ine process. These
classiﬁers were then able to quickly classify new features at runtime
with a minimal number of binary comparisons. BRIEF disposed of the
classiﬁer and trees, and instead used the results of the binary tests to
create a bit vector.
To reduce the impact noise has on the descriptor, the image patches
for each feature are pre-smoothed using a Gaussian kernel. The spatial
arrangement of the pairwise pixel comparisons was determined by
evaluating a set of arrangements for descriptor lengths of 128, 256 and
512 bits (known in the paper by their byte amounts as BRIEF-16,
BRIEF-32 and BRIEF-64). The arrangements for the 128 bit descriptor
are shown in Figure 4.13, where the ﬁrst four arrangements were
generated randomly, using uniform (1) and gaussian distributions (2,3)
and spatial quantization of a polar grid (4), and the ﬁfth arrangement
taking all possible values on a polar grid. For the 128 bit descriptor,
the second arrangement had the highest recognition rate.
When compared with the SURF feature descriptor, The BRIEF-64
outperformed SURF in all sequences where there was not a signiﬁcant
amount of rotation, as the BRIEF descriptor is not rotation invariant.
Shorter descriptor lengths such as BRIEF-32 performed well for easy
test cases; however BRIEF-16 was at the lower bound of recognition.
The main advantage was the increase in speed, with BRIEF descriptor
computation being on average 35 to 41 times faster than SURF, and
BRIEF feature matching being 4 to 13 times faster than SURF.
4.2. Vision Based Tracking 119
Figure 4.13: Tested spatial arrangements for the pairwise pixel comparisons for the
128 bit descriptor. The ﬁrst four were randomly generated used various distributions,
while the ﬁnal one uses all possible values on a polar grid. The second arrangement
was chosen as it had the greatest recognition rate. [Calonder et al., 2010]
The main disadvantage of BRIEF is its lack of rotation invariance.
Resolving this was the main aim of ORB [Rublee et al., 2011], or Oriented
FAST and Rotated BRIEF. ORB combined the FAST feature
detector [Rosten and Drummond, 2006], modiﬁed to determine orientation
of a feature using an intensity centroid [Rosin, 1999], with
a version of BRIEF that uses lookup tables of pre-computed rotated
spatial arrangements. Including all the additional improvements, an
evaluation showed ORB is, on average, an order of magnitude faster
than SURF and two orders of magnitude faster than SIFT.
The success of BRIEF lead to a number of other feature detector
and descriptors which aimed to drastically reduce the computational
time required for feature detection, description and matching. Most
notable were BRISK (Binary Robust Invariant Scalable Keypoints)
[Leutenegger et al., 2011], which used a feature detector inspired by
AGAST [Mair et al., 2010] and a feature descriptor inspired by DAISY
[Tola et al., 2010], and FREAK [Alahi et al., 2012] (Fast Retina Keypoint),
which was designed to mimic features of the human visual processing
system, and has been shown to outperform SIFT, SURF and
even BRISK.
Model Based Tracking
Although not as popular as planar ﬁducial or natural feature tracking,
there has been some interest in tracking real world objects using a
known 3D structure, such as a CAD model. In early works, the 3D
model of the object to be tracked was typically created by hand, and
120 AR Tracking Technology
the structure of the object was often approximated as a combination of
primitives, such as lines, circles, cylinders and spheres [Comport et al.,
2003]. Edge ﬁlters were used to extract structure information about the
scene, and these were matched to the primitive objects to determine
pose [Wuest et al., 2005].
Combining natural feature tracking with the edge based tracking
approaches provided additional robustness [Vacchetti et al., 2003], and
also allowed for large trackable spaces and even tracking in outdoor environments
[Comport et al., 2006]. Later works extended the primitive
models to include texture [Reitmayr and Drummond, 2006] and introduced
the concept of keyframes [Vacchetti et al., 2003], which allowed
for even greater robustness in complex and variable environments.
Most recently, there has been signiﬁcant interest in techniques that
are able to simultaneously create and update a map of the real environment
while localizing their position within it. The original motivation
behind SLAM (Simultaneous Localization and Map Building) was for
robot navigation in unknown environments [Dissanayake et al., 2001];
however the technique was adapted for use in Augmented Reality [Davison
et al., 2007], where it allowed for drift free tracking of unknown
environments. Further optimizations of the process led to PTAM, or
Parallel Tracking and Mapping [Klein and Murray, 2007], where the
tracking of the camera and mapping of the environment components
were separated (see Figure 4.14). PTAM was speciﬁcally designed for
AR, and improved both the accuracy of tracking as well as overall performance,
although due to its diminished ability to close large loops
like SLAM it is more suited for small environments.
4.2.3 3D Structure Tracking
In recent years, commercial sensors capable of detecting 3D structure
information from the environment have become very a ordable, beginning
with the introduction of the Microsoft Kinect [Zhang, 2012].
These sensors commonly utilise technologies such as structured light
[Scharstein and Szeliski, 2003] or time-of-ﬂight [Gokturk et al., 2004]
to obtain information about the three dimensional positions of points
in the scene.
4.2. Vision Based Tracking 121
Figure 4.14: PTAM tracking from real world feature points. [Klein and Murray,
2007]
Figure 4.15: KinectFusion. From left to right: RGB image, 3D mesh created from
single depth image, Reconstructed model of environment, AR particle e ects, Multitouch
interaction. [Izadi et al., 2011].
These new technologies have opened up new possibilities for tracking
and mapping the real environment using depth information [Henry
et al., 2010]. The most well known approach for Augmented Reality
currently is the KinectFusion system [Newcombe et al., 2011], developed
by Microsoft. KinectFusion uses data obtained from the Kinect’s
structured light depth sensor to create high quality three dimensional
models of real objects and environments, and these models are also
used for tracking the pose of the Kinect in the environment. Figure 4.15
shows some examples of KinectFusion based tracking and interaction
[Izadi et al., 2011].
122 AR Tracking Technology
In addition to tracking for AR, KinectFusion o ers 3D reconstruction
of real objects and the environment, allowing for simulated physical
interactions between real and virtual objects, as shown in Figure 4.15.
By combining tracking of the environment with tracking of the users
limbs, KinectFusion can also detect multi-touch user interactions with
the environment.
4.3 Inertial Tracking
Inertial tracking uses Inertial Measurement Unit (IMU) sensors such as
accelerometers, gyroscopes and magnetometers to determine the relative
orientation and velocity of a tracked object. Inertial tracking allows
the measurement of three rotational degrees of freedom (orientation)
relative to gravity, and the change in position of the tracker can be
determined using the inertial velocity and time period between tracker
updates.
Inertial sensors have "no range limitations, no line-of-sight requirements,
and no risk of interference from any magnetic, acoustic, optical
or RF interference sources. They can be sampled as fast as desired, and
provide relatively high-bandwidth motion measurement with negligible
latency." [Foxlin et al., 2004].
Unfortunately, inertial sensors are very susceptible to drift over time
for both position and orientation. This is especially problematic for position
measurements, as position must be derived from measurements of
velocity. The e ects of drift can be reduced with ﬁltering, although this
can reduce the update rate and responsiveness of the trackers [Foxlin,
1996]. Due to these issues with drift, applications requiring accurate
tracking should aim to combine inertial sensors should with other sensors
for correction, such as optical trackers or ultrasonic range measurement
devices.
4.4 GPS Tracking
GPS technology allows for positional tracking in outdoor environments
over the majority of the earth’s surface. The current average accuracy
of satellite based GPS is less than 3 metres, with the accuracy
4.5. Hybrid Tracking 123
scheduled to improve as new advancements and satellite technology
becomes available. In addition, GPS enhancement technologies such
as RTK (Real Time Kinematic), which uses the GPS signal’s carrier
wave for measurement, have the potential to improve accuracy to the
centimetre level.
The ﬁrst Augmented Reality system using GPS was Feiner et al.’s
Touring Machine [Feiner et al., 1997], which featured di erential GPS
positional tracking accurate to 1 metre and magnetometer/inclinometer
orientation tracking. The interface was designed in such a way that
the inaccuracy of the GPS data did not signiﬁcantly a ect usability,
however the authors found that loss of GPS signal due to trees and
buildings blocking line of sight to satellites caused signiﬁcant problems.
GPS tracking for AR has since been applied to a number of application
areas, including military [Julier et al., 2000], gaming [Piekarski
and Thomas, 2002a, Thomas et al., 2002], and visualisation of historical
data [Lee et al., 2012]. However, due to its low accuracy and only
supporting positional tracking, GPS tracking has typically only been
utilised in applications where accurate pose registration is unimportant,
or as an input into a hybrid tracking system.
4.5 Hybrid Tracking
Hybrid tracking systems fuse data from multiple sensors to add additional
degrees of freedom, enhance the accuracy of the individual
sensors, or overcome weaknesses of certain tracking methods.
Early applications utilising vision based tracking approaches often
included magnetic trackers [Auer and Pinz, 1999] or inertial trackers
[You and Neumann, 2001]. This allowed the system to take advantage of
the low jitter and drift of vision based tracking, while the other sensor’s
high update rates and robustness would ensure responsive graphical
updates and reduced invalid pose computation [Lang et al., 2002]. An
additional beneﬁt was that, as magnetic and inertial trackers do not
require line of sight, they can be used to extend the range of tracking.
Figure 4.16 shows the system diagram of You’s hybrid tracking system
[You et al., 1999] and the tracking results compared to using the inertial
124 AR Tracking Technology
Figure 4.16: Hybrid tracking system (left) and tracking error over time (right).
[You et al., 1999]
tracking by itself. As can be seen, the combined hybrid tracking has
far less error than just using inertial tracking.
Due to their low accuracy and only providing positional information,
GPS tracking systems are often combined with inertial sensors
and/or vision based tracking in order to obtain full 6DOF pose estimation
[Azuma et al., 1998]. Furthermore, to overcome issues with loss of
signal due to occlusions or being inside a building, vision based tracking
is often used as for backup tracking when GPS fails [Thomas et al.,
2002].
Inertial trackers are often paired with ultrasonic rangeﬁnders
[Foxlin et al., 1998] or optical tracking systems [Foxlin et al., 2004]
to reduce the amount of positional drift caused by the di erential measurement
approach. As inertial sensors use forces such gravity as a reference,
hybrid tracking also becomes important when the application
involves being on a moving platform, such as in a vehicle or simulator
Foxlin [2000].
Mobile devices such as smartphones and tablets present new opportunities
for hybrid tracking. Many mobile devices include cameras,
accelerometers and gyroscopes [Lang et al., 2002], GPS [Höllerer et al.,
1999], and wireless networking Evennou and Marx [2006], which can be
combined to provide highly accurate pose estimation for both indoor
and outdoor AR tracking Newman et al. [2004].
4.6. Summary 125
4.6 Summary
In summary, in this section we have shown that a wide variety of tracking
approaches have been used create the illusion that AR virtual cues
are ﬁxed in space. The most widely used approaches currently use computer
vision techniques and can provide marker based and markerless
tracking at real time frame rates. However there are limitations with
systems just relying on computer vision and it is becoming increasingly
obvious that truly robust and accurate AR tracking will require hybrid
tracking. In the next section we summarize display technology available
to achieve the ﬁrst of Azuma’s AR characteristics, combing real
and virtual content together. Combined with AR tracking, AR displays
create the illusion that virtual content is really part of the user’s real
world.
5
AR Display Technology
Returning to Azuma’s deﬁnition, in order to combine real and virtual
images so that both are seen at the same time, some sort of display
technology is needed. In this section we review various type of display
technologies used in AR systems and applications. First we look into
di erent approaches used for combining and compositing images of
virtual scenes with the physical scene. Next we review a range of AR
displays categorized based on where they are located between the user’s
eye and the physical scene. Although AR displays could be used for
visualizing 3D graphics as is with generic 3D display technology (e.g.
such as stereoscopic displays or volumetric displays), here we focus on
combining the images of virtual and physical scenes on displays that
are speciﬁcally designed for AR visualization.
5.1 Combining Real and Virtual View Images
Combining images of the real and virtual scenes for AR visualization
requires a number of procedures including camera calibration, registration,
tracking and composition. Camera calibration is a procedure
to match the camera parameters of the virtual camera to that of the
126
5.1. Combining Real and Virtual View Images 127
Figure 5.1: Camera calibration: matching internal and external parameters of the
virtual camera to the physical view.
physical camera (or an optical model of the user’s view deﬁned by eyedisplay
geometry), so that the computer rendered image of a virtual
scene is correctly aligned to the view of the real world (see Figure 5.1).
There are two types of camera parameters: internal and external parameters.
Internal parameters are those that determine how a 3D scene
is projected into a 2D image, while the external parameters deﬁne the
pose (position and orientation) of the camera in a known coordinate
frame.
Internal parameters of a physical camera can be calculated using
a process that involves taking a set of pictures of a calibration pattern
that has known geometric features and calculating the parameters
based on the correspondence between the 3D structure of this geometric
features and its projection on the 2D image [Tsai, 1987]. This is
usually performed o -line before the AR system is in operation, but
some AR systems do this on the ﬂy [Simon et al., 2000]. In case of not
using a physical camera to capture the real world scene, the internal
128 AR Display Technology
parameters are decided by a geometrical model of the relationship between
the user’s eye and the display’s image plane (e.g. as a pinhole
camera model) [McGarrity and Tuceryan, 1999, Fuhrmann et al., 1999,
Tuceryan and Navab, 2000].
External parameters are decided by tracking the pose (position and
orientation) of the physical camera. When the physical scene is static,
only the pose of the camera relative to the world’s reference frame
needs to be tracked, while in dynamic scenes the poses of each dynamic
object of interest should be also tracked so that the change in the real
world can be reﬂected in the virtual scene. As described in the previous
section, there are various technologies for tracking physical objects,
ranging from using 6 DOF sensors to computer vision techniques.
Tracking technologies provide the position and orientation of target
objects (either the camera or other physical objects of interest) relative
to a coordinate frame set by the tracking system. To make the virtual
scene correctly aligned with the real world environment, the coordinate
frame used for rendering the virtual scene should be matched with the
tracking coordinate frame in the physical environment in a process
called registration.
Once the virtual camera is calibrated, and the virtual space is registered
to (or correctly aligned with) the physical space, rendering the
virtual scene based on the tracking update generates images of the virtual
scene that corresponds to the users’ real world view. Composition
of the generated virtual view image (image of the virtual scene) into
the real world view can be achieved both digitally or physically (or
optically) depending on the display system conﬁguration used in the
AR system.
AR displays can be categorized into mainly four types of conﬁguration
depending on how they combine the virtual view image with the
real world view: (1) video based, (2) optical see-through, (3) projection
onto a physical surface, and (4) eye multiplexed.
5.1.1 Video based AR Displays
Video based AR displays use digital processes to combine virtual view
images with video of the real world view. This type of display ﬁrst
5.1. Combining Real and Virtual View Images 129
Figure 5.2: Structure of video based AR displays.
digitizes the real world scene using a video camera system, so that the
image of the real environment could be composited with the rendered
image of the virtual scene using digital image processing technique.
Figure 5.2 shows the structure of video based AR displays.
In many cases, the video camera is attached on the back of the
display, facing towards the real world scene, which the user is looking
at when watching the display. These types of displays create the digital
illusion of seeing the real world "through" the display, hence they are
called "video see-through" displays. In other cases, the camera can be
conﬁgured in di erent ways such as facing the user to create virtual
mirror type of experience, providing a top down view of a desk, or even
placed at a remote location. Figure 5.3 shows various conﬁgurations.
Video based AR displays are the most widely used systems due to
the available hardware. As digital camera become popular in various
types of computing devices, video based AR visualization can be easily
implemented on a PC or laptop using webcams and recently even on
smartphones and tablet computers.
Video based AR also has an advantage of being able to accurately
control the process of combining the real and virtual view images. Advanced
computer vision based tracking algorithms provide pixel accurate
registration of virtual objects onto live video images (see section
4). Beyond geometric registration of the real and virtual worlds,
130 AR Display Technology
(a) Video see-through (b) Virtual Mirror
(c) Augmented Desk (d) Remote AR
Figure 5.3: Di erent conﬁgurations of placing camera in Video based AR visual-
ization.
with video based AR displays, real and virtual image composition can
be controlled with more detail (e.g. correct depth occlusion or chromatic
consistency) using digital image processing.
One of the common problems in composition of virtual and real
world images is incorrect occlusion between real and virtual objects due
to virtual scene image being overlaid on top of the real world image.
Video based AR displays can easily solve this problem by introducing
depth information from the real world, and doing depth tests against
the virtual scene image which already has depth data. The most naive
approach is to get depth information of the real world image based
5.1. Combining Real and Virtual View Images 131
(a) Chroma-keying (user’s hand with a virtual structure) [Lee et al., 2010a]
(b) Using mask objects (a real box with a virtual robot) [Lee et al., 2008a]
(c) Using depth camera (a real book with a virtual toy car) [Clark and Piumsomboon,
2011]
Figure 5.4: Compositing AR images with correct depth occlusion.
on heuristics and use chroma-keying techniques when compositing the
two images. For instance, an AR system can identify the region in the
real world image with skin colour to prevent the user’s real hand being
occluded by virtual objects (see Figure 5.4(a)).
Advanced methods can estimate or measure the depth information
of the real world image. Breen et al. [1996] proposes two ways of obtaining
depth image of the real world scene: (1) using mask objects
and (2) depth cameras. Mask objects (a.k.a. ghost objects) are virtual
objects that represent corresponding physical objects, and by rendering
them only into the depth-bu er of the computer graphics pipeline the
system can obtain the depth information for the real world scene (see
132 AR Display Technology
Figure 5.5: Compositing AR image with correct chromatic consistency.
(left: a virtual glass and a virtual box with a dragon [Kán and Kaufmann, 2013],
right: a virtual excavator [Klein and Murray, 2010])
Figure 5.4(b)). Compared to using mask objects that needs manual
modelling of the real world scene, depth cameras are becoming widely
available (e.g. Microsoft Kinect) and they can capture the depth image
of the real world scene together with the RGB image (see Figure 5.4(c)).
Di erences in lighting [Agusanto et al., 2003, Kán and Kaufmann,
2013] and colour space [Klein and Murray, 2010] between the real and
virtual scenes can be also reduced relatively easily in video based AR
systems during image composition process, as shown in Figure 5.5.
Other beneﬁts of video based AR displays include having no temporal
gap between the real world image and the rendered virtual scene,
and being more ﬂexible with controlling the ﬁeld of view of the AR
scene using wide FOV lenses on the camera.
The most prominent problem of video based AR displays is having
an indirect view of the real world. Since the real world view is provided
by video image captured through a camera, it has limitations
in terms of resolution, distortion, delay, and eye displacement. These
problems can be critical in safety demanding applications where direct
view of the real world is necessary. There are e orts to reduce these
problems, such as using a camera with higher resolution and faster update
rate, undistorting the image while rendering, and using a specially
5.1. Combining Real and Virtual View Images 133
designed optical system to match the eye displacement [Takagi et al.,
2000]. However, there will be certain amount of limitations remaining
compared to other types of AR displays.
A ﬁnal limitation of video based AR displays is requiring more computational
power. While recent advance in computing hardware made
even mobile phones able to run AR applications in real time, compared
to other types of displays that use optical systems for image composition,
video based AR demand more computing power for combining
real and virtual view images.
5.1.2 Optical See-through AR Displays
Optical see-through AR displays use optical systems to combine virtual
view images with the real world view. The optical system in this
type of AR display usually includes beam splitters (e.g. half mirrors
or combined prisms) that combine the real world view seen through
the splitter with the reﬂection of image from a video display (see Figure
5.6). Head up displays (HUD) on airplane cockpits or modern cars
are typical examples of optical see-through AR displays using beam
splitters in this way. Virtual mirrors based on optical systems combine
the real and virtual view images in the other way around. In this case,
it is the real world scene that is reﬂected on the half mirror while the
user can see the video display through the mirror.
Transparent projection ﬁlms are another type of optical component
used for optical see-through AR displays. This type of projection ﬁlm
not only di uses the light to show the projected image, but is also semitransparent
so that the viewer can also see what is behind the screen
through it. See Figure 5.7 for examples of optical see-through displays.
Most optical see-through AR displays use optical combiners, separate
from the displays, for combining the real and virtual view images,
however recently there have been technical advances in making the
display itself transparent. With the development of transparent ﬂat
panel displays (e.g. LCD), their use in AR visualization is being actively
investigated. Using transparent ﬂat panel displays is expected to
contribute to simplifying and miniaturizing the structure and size of
optical see-through AR displays.
134 AR Display Technology
Figure 5.6: Structure of optical see-through displays.
Compared to video based AR displays, the most prominent advantage
of optical see-through AR displays is providing a direct view of
the real world. With a direct view of the real world, optical see-through
displays do not su er from limitations in resolution, lens distortion, eye
displacement, or time delay. This could be a very important feature for
safety demanding applications that require a direct view of the real object,
such as medical or military applications. Optical see-through AR
displays also needs simpler electronic components and demands less
processing power since the composition of real and virtual view image
is achieved physically.
The main problems in optical see-through AR displays are less accurate
registration between the real and virtual view images. In many
cases, optical see-through displays require manual calibration process
which ends up with relatively poor quality of registration compared to
automated computer vision based approach in video based AR displays.
Since the calibration parameters are dependent on the spatial relationship
between the user’s eyes and the display’s image plane, there is a
higher chance of parameters changing over time (e.g. wearable displays
sliding o from the original position) and eventually causing misalignment
between the real and virtual view images. Hence, accurate 3D eye
tracking (relative to the display) becomes more important to visualize
a correctly aligned AR scene with optical see-through displays.
5.1. Combining Real and Virtual View Images 135
Figure 5.7: Optical see-through displays using half-mirror [Hilliges et al.,
2012] (top), transparent projection ﬁlm (lower left, http://www.laser-magic.com/
transscreen.html), and transparent LCD (lower right).
Temporal delay between the real world and the virtual views is another
challenging problem in optical see-through displays. Even with a
spatially accurate tracking system, there always would be temporal delay
for tracking physical objects. As the virtual view is updated based
on the tracking results, unless the physical scene and the user’s viewpoint
is static, the virtual view will have temporal delay compared to
the direct view of the real world in optical see-through displays.
In many cases, visualizing correct depth occlusion between the real
and virtual view images is more challenging with optical see-through
displays. Due to the physical nature of optical combiners, users see
semi-transparently blended virtual and real view images with most of
the optical see-through displays, neither of the images fully occluding
the other. To address this problem Kiyokawa et al. [2001] developed
an electronically controlled image mask that masks the real world view
136 AR Display Technology
where the virtual objects are to be overlaid. By closing the pixels of
the LCD panel on the region where the virtual objects are visualized,
the real world view was optically occluded and the virtual view image
from the display became more vivid.
Lighting conditions in the real world environment can also a ect
perceived brightness of optical see-through displays. In many cases,
optical combiners have a ﬁxed physical attribute of amount of transparency
that can lead to unbalanced brightness between the real world
view and virtual view image depending on the lighting conditions in the
real world. In outdoor environments, the virtual view image will appear
relatively dim compared to the bright real world view. To overcome this
problem, some types of head mounted display provide a replaceable
cover with various level of transparency, while some researchers also
experimented using grayscale LCD shutters to control the brightness
of the real world view.
5.1.3 Projection based AR Displays
While other types of displays combine the real and virtual world view at
the display’s image plane, projection based AR displays overlay virtual
view images directly on the surface of the physical object of interest
(see Figure 5.8), such as real models [Raskar et al., 1999, 2001] or
walls [Raskar et al., 2003]. In combination with tracking the user’s
viewpoint and the physical object, projection based AR displays can
provide interactive augmentation of virtual image on physical surface
of objects [Raskar et al., 2001].
In many cases, projection based AR displays use a projector
mounted on a ceiling or a wall. While this could have an advantage
of not requiring the user to wear anything, this could limit the display
to be tied with certain locations where the projector can project images.
To make the projection based AR displays more mobile, there
have been e orts to use projectors that are small enough to be carried
by the user [Raskar et al., 2003]. Recent advance in electronics miniaturised
the size of the projectors so that they could be held in hand
[Schöning et al., 2009] or even worn on the head [Krum et al., 2012] or
chest [Mistry and Maes, 2009].
5.1. Combining Real and Virtual View Images 137
Figure 5.8: A physical model of Taj Mahal augmented with projected texture.
[Raskar et al., 2001]
As projection based AR displays display virtual images directly on
real object surface, it requires a physical surface where the image can
be projected onto. This could limit its application to augmenting only
objects near to the user, as it might not be appropriate to projecting
images on far away objects as in outdoor AR applications. Other limitations
of projection based AR displays include being more vulnerable to
lighting conditions, and su ering from shadows created by other physical
objects (such as user’s body). While stereoscopic projectors can be
used for presenting virtual objects that pop out from the projection
surface, providing correct occlusion between another physical object
(such as user’s hand) and the virtual object could be more challenging
compared to video or optical see-through displays.
5.1.4 Eye Multiplexed AR Displays
While the three methods above provide the ﬁnal combined AR image
to the user, another approach for visualization in AR applications is to
138 AR Display Technology
Figure 5.9: Structure of Eye Multiplexed AR Displays.
let users combine the views of the two worlds mentally in their minds.
We refer to such a way of using displays as "eye multiplexed" AR visualization.
As illustrated in Figure 5.9, with eye multiplexed AR displays,
the virtual scene is registered to the physical environment, hence the
rendered image shows the same view of the physical scene that the user
is looking at. However, the rendered image is not composited with the
real world view, leaving it up to the user to mentally combine the two
images in their minds. To ease the mental e ort of the user in this case,
the displays are needed to be placed near to the user’s eye and should
follow the user’s view, so that the virtual scene shown on the display
could appear as an inset into the real world view (see Figure 5.10).
As in the case of optical see-through and projection based AR displays,
eye multiplexed AR displays do not require digital composition
of the real world view and the virtual view image hence demands less
computing power compared to video based AR displays. Furthermore,
eye multiplexed AR displays are more forgiving of inaccurate registration
between the real and virtual scene. As the virtual view is shown
next to the real world view, pixel accurate registration is not necessary
but showing the virtual scene from the perspective of the real world
view is similar enough for the user to perceive it as the same view.
5.2. Eye-to-World spectrum 139
Figure 5.10: Eye multiplexed AR visualization of virtual buildings on Google Glass
as an inset in the top right corner. http://arforglass.org
On the other hand, as the matching between the real world view
and the virtual view is up to the users’ mental e orts, the visualization
is less intuitive compared to other types of AR displays that show the
real and virtual scene in one single view.
5.2 Eye-to-World spectrum
While AR displays use di erent approaches to combine the real and
virtual views, they can also be categorized based on where the display
is placed between the user’s eye and the real world scene. Bimber and
Raskar [2006] summarises this in a diagram shown in Figure 5.11.
5.2.1 Head-attached Displays
Head-attached displays present virtual images right in front of the
users’ eyes. No other physical objects come between the eyes and the
virtual image from the display, which guarantees the virtual image not
getting occluded by other physical objects. Head-attached displays vary
in size from a helmet or a goggle to the size of lightweight glasses (see
Figure 5.12). As the technology advances, the displays are becoming
140 AR Display Technology
Figure 5.11: AR displays on eye-to-world spectrum. [Bimber and Raskar, 2006]
lighter to wear while providing a wider and brighter image on the dis-
play.
Head Mounted Displays (HMD) are the most common type of display
that are used in AR research. HMDs mostly are in a form similar
to a goggle which users can wear on their head and the display is placed
in front of the user’s eyes. In some cases they are also referred to as
Near Eye Displays or Face Mounted Displays. While video and optical
see-through conﬁgurations are most commonly used when using HMDs
for AR display, head mounted projector displays are also actively researched
as the size of the projectors became small enough to wear on
user’s head.
Recently, with technical developments in wearable devices, smart
glass type devices, such as Google Glass or Recon Jet, are also actively
investigated for their use in AR applications. While many of the
smart glass devices are capable of working as both video or optical seethrough
AR displays, due to the narrow ﬁeld of view of the display, eye
5.2. Eye-to-World spectrum 141
Figure 5.12: Various types of head-attached displays. From left to right: video
and optical see-through HMD from Trivisio (www.trivisio.com), and Google Glass
(http://www.google.com/glass)
multiplexed AR visualization is also considered as an alternative way
of visualizing AR scene on smart glasses.
While most of the head-attached displays use lenses on an eyepiece
similar to telescopes or microscopes that does not directly contact with
the user’s eyeball, more direct methods are under development. For
example, in the Virtual Retinal Display [Tidwell et al., 1995] project
virtual images are beamed directly onto the retina of the user’s eye.
5.2.2 Handheld and Body-attached Displays
Head-attached displays provide a high mobility and visually immersive
experience, but in many cases they have certain limitations in terms
of wearability (feeling comfortable while wearing), safety issue with
indirect real world view, and social acceptance. Handheld or bodyattached
displays are considered to be mobile and personal, yet sharable
with others as needed. They are also more socially acceptable compared
to head-attached displays.
Early researchers [Rekimoto and Nagao, 1995, Mogilev et al., 2002]
used small LCD monitors tethered to a computer for experimenting
with handheld AR experiences. With advances in mobile device technology,
handheld computing devices became powerful enough to process
AR visualization, and various devices have been used as AR displays,
including UMPC (Ultra Mobile PC) [Reitmayr and Drummond,
2006], PDA (Personal Digital Assistant) [Wagner and Schmalstieg,
142 AR Display Technology
Figure 5.13: A smartphone with depth imaging camera from Google Tango project.
2003], Tablet computers [Klein and Drummond, 2004], and cell-phones
[Mohring et al., 2004].
Today, smartphones and tablet computers are widely adopted and
have powerful graphics processors, cameras, and various sensors that
can run AR applications. Most recently, even depth imaging sensors are
experimented on smartphones and tablets (e.g. Google Project Tango1)
for advanced tracking and visualization (see Figure 5.13).
While video-based AR display conﬁgurations have been widely investigated
on handheld devices, handheld optical see-through AR displays
are still experimental design concepts, as the electronic components
are still under development. Some electronics companies have
been producing transparent LCD and OLED panels, and it is expected
to be able to use these for building handheld optical see-through AR
displays in the near future.
Micro projectors are also another type of devices used as mobile
AR displays. While most of the use cases involve holding the projector
as a handheld device similar to a ﬂashlight [Beardsley et al., 2005,
Schöning et al., 2009], there is also various research on wearing the
device on di erent parts of the body such as on the chest [Mistry and
Maes, 2009], shoulder [Harrison et al., 2011], or on a wrist [Blasko et al.,
2005].
1
https://www.google.com/atap/projecttango/
5.2. Eye-to-World spectrum 143
5.2.3 Spatial Displays
Compared to head mounted and handheld displays, spatial displays
are limited in mobility and are usually installed at a ﬁxed location. As
spatial displays tend to provide a larger image in many cases, they are
more applicable as public displays which multiple users share.
Typical examples of spatial displays are those using a beam splitter
(e.g. half mirror) to create optical see-through conﬁguration for AR
visualization. A half mirror workbench [Poston and Serra, 1994, Bimber
et al., 2001a, Gunn et al., 2003, Hilliges et al., 2012] are one of the
examples of an AR display using this conﬁguration, which allows for
close interaction with virtual objects and hence are widely used in
combination with haptic interfaces (see Figure 5.14). Other work using
a similar conﬁguration of optical see-through AR displays include a
virtual showcase which can overlay virtual images on a physical object
inside a pyramid or cone shaped half mirror case [Bimber et al., 2001b],
and using multiple autostereosopic displays above and below a slanted
half mirror [Kim et al., 2012].
Figure 5.14: AR Haptic workbench. [Gunn et al., 2003]
Recently, with advances in transparent active display components,
such as transparent LCDs and OLEDs, it is expect to see more wide
adoption and application of AR displays used in show cases or even
show windows (see Figure 5.7). Another common optical element used
for creating optical see-through spatial AR displays is transparent projection
ﬁlm as described in §5.1.2. These types of screens are also known
144 AR Display Technology
as ’holographic projection ﬁlm’ that are transparent yet di uses the
projected image onto the surface of the screen so that the user can
see both the projected virtual image and the physical scene behind the
screen.
Instead of using a transparent projection screen, projecting the virtual
image directly onto the surface of physical objects is another common
approach used for implementing spatial AR display. As described
in §5.1.3, projection based AR displays overlay virtual images directly
on the surface of the physical object of interest (see Figure 5.8), which
can be physical models [Raskar et al., 2001, 1999] or walls [Raskar
et al., 2003].
Spatial displays can also use video-based AR visualization. A desktop
monitor paired with a video camera pointing on the desk in front
of the monitor is one of the widely used setup, a.k.a. Desktop AR or
Video-Monitor AR, for demonstrating AR applications. Another example
of using video-based AR visualization on a spatial display would
be virtual mirror like setup where a large screen (either a monitor or a
projection screen) with a camera is pointing towards the user standing
in front, showing the real world around the user and overlaying virtual
objects into the scene.
Using other types of 3D display technology such as volumetric 3D
displays [Favalora, 2005] and autostereoscopic displays [Dodgson, 2005]
for AR visualization is expected to be an interesting topic to investigate
in the future, together with using a combination of di erent AR display
technologies together [Lee et al., 2006] to provide richer AR experience.
5.3 Other Sensory Displays
While most of the research in AR focuses on visual augmentation of
the real world, there is also work on AR displays for the other senses,
such as audio or haptic. These types of non-visual displays are mostly
adopted from the ﬁeld of Virtual Reality (VR) where the main focus
is to replace all human sensory input with computer generated virtual
signals. A lot of this work in VR can be applicable in AR research
as well, while there are also certain speciﬁc topics that arise in AR
5.3. Other Sensory Displays 145
applications. In this section we brieﬂy introduce the non-visual sensory
displays that are used in AR research and some of the work that focuses
on their use in AR applications.
Compared to visual augmentation, computer generated sounds are
mixed with real world sound by nature and the main challenge of providing
augmented sound to the user is spatializing the sound in the
3D space. This has been widely investigated using various technologies
including stereo headphones with HRTF (Head-Related Transfer
Function) [Zotkin et al., 2003] or multi-channel speaker systems. While
traditional multi-channel 3D surround speaker systems use dedicated
speakers for each direction of the sound that has to be placed in space
around the user, digital sound projectors [Takumai, 2011] use an array
of speakers to generate 3D sound utilizing interference and reverberation
of sound in a room.
Traditional spatialized 3D sound technology can be easily applied
in AR systems, and there are attempts to investigate AR speciﬁc issues
in audio displays. For instance, Lindeman et al. [Lindeman et al., 2007]
proposed a concept of "hear-through" and "mic-through" Augmented
Reality analogous to optical and video see-through AR visualization,
and demonstrated its feasibility through a user study with bone conduction
headset.
Haptic displays have been also well investigated in the VR research
ﬁeld. Among various haptic devices, robotic arm based mechanical devices
(as shown in Figure 5.14) and vibro-tactile feedback available on
various modern mobile devices (e.g. smartphones) are the most widely
investigated in haptics research, and are also applied in various AR
research. Extending Milgram’s Reality-Virtuality continuum [Milgram
and Kishino, 1994], Jeon and Choi [2009] proposes a taxonomy for haptic
AR based on a composite visuo-haptic reality-virtuality continuum
and summarize various approaches for AR systems using haptic interfaces
to let the user feel a real environment augmented with synthetic
haptic stimuli.
Augmenting olfactory and gustatory senses is rarely explored compared
to other sense, yet considered as promising research area to complement
the other sensory experience. Narumi et al. [2011] investigated
146 AR Display Technology
Figure 5.15: Meta cookie - augmenting olfactory and gustatory experience.
[Narumi et al., 2011]
augmenting olfactory and gustatory sense through visual and olfactory
augmentation of cookie eating experience. The system built by the research
team, named Meta Cookie (see Figure 5.15), uses a plain cookie
with AR marker printed on it (with edible colour) to visually augment
di erent toppings to the cookie and as the user eats the cookie, a specially
designed scent distribution device provides olfactory stimuli to
reproduce the di erent ﬂavours of the cookie.
5.4 Summary
In this section we have provided an overview of a variety of di erent
types of displays that can be used to combine real and virtual images.
Although most early AR systems used head mounted displays, the most
common display type currently used is a handheld device such as a
mobile phone or tablet. However this may change with the emergence
of lower cost head worn wearable computers and displays, and also
with new research in di erent display systems. In the next section we
review tools for creating AR systems using the tracking and display
technology discussed.
6
AR Development Tools
In the previous sections we have reviewed important tracking and display
technologies that can be used to create Augmented Reality experiences.
However these are hardware devices and require AR application
software to provide that experience. In this section we discuss software
libraries and tools for developing AR applications. As Table 6.1 shows,
there are numerous tools for implementing AR systems that can be arranged
in a hierarchy of decreasing programming skill required to use
them; from low level software libraries that require consider skill on the
part of the developer, to simple authoring tools for novice users that
assume no programming ability. In this section we will review each of
these types of tools in term, covering both research and commercially
available products.
AR development tools also have di erent target application platforms.
Some tools are for developing desktop AR applications while
others target mobile AR application. Some support computer vision
based tracking for developing AR application, mainly for indoor use,
while others use sensors on a mobile device for developing outdoor AR
application. So each of the types of tools will also be discussed in terms
of the platforms that they are designed for.
147
148 AR Development Tools
Table 6.1: Hierarchy of AR development tools from most complex to least complex.
Type of Tool Skill Required Example
Low-level software li-
brary/framework
Strong programming/coding
ability
ARToolkit, osgART,
Studierstube, MXR-
ToolKit
Rapid prototyping
tools
Some programming ability,
but design/prototyping skills
FLARManager, Processing,
OpenFrame-
works
Plug-in for existing
developer tool
Skill with the developer tool
that the plug-in works with
DART, AR-Media
plug-ins, Vuforia
and Metaio Unity
plug-ins
Stand-alone AR authoring
tools
No programming ability, but
can learn stand-alone tool
BuildAR, Metaio
Creator, Layar
Creator, Wikitude
Studio
In the rest of this section we provide an overview of tools available
for developing AR applications. We ﬁrst look into low-level software
development tools for those who need full control of the technology with
programming skills. Then we present plug-ins that work with existing
applications, followed by stand-alone AR authoring tools for users with
a non-programming background. We next present rapid prototyping
tools that can be used to quickly develop AR concepts, and ﬁnally
research into the next generation of AR authoring tools. The tools
presented will be both novel tools published in many research papers,
and those available in public domain that readers can access.
6.1 Low Level software libraries and frameworks
Low level software libraries and frameworks for AR typically provide
access to the core tracking and display functionality need to build an
AR application. For example, in the previous section we introduced
ARToolkit1 one of the most widely adopted software library for developing
AR applications. ARToolKit provides marker based tracking in
the open source version, while the commercial version from ARTool-
1
http://artoolkit.sourceforge.net
6.1. Low Level software libraries and frameworks 149
Works2 also supports tracking natural features on an arbitrary image.
In this way it provides a solution for the key problem of being able to
know where the user view is currently. ARToolkit is written in the C
language and it supports various platforms including Windows, Linux,
and Mac OS X desktop operating systems. However, the main function
of ARToolKit is to provide tracking and so there is only low-level
OpenGL based rendering, and no built-in support for interaction techniques
or complex graphics. With ARToolKit alone it is di cult and
time consuming to develop a complete AR application.
This limitation is can be overcome by using the osgART3 library,
which provides a link between ARToolkit tracking and the OpenSceneGraph4
library for scene graph based real-time rendering. Scene graphs
represent computer graphics scenes as a node based graph, and provide
one way to manage the functionality of complex graphics applications,
such as being able to sort objects according to their visibility, remove
objects that are out of the camera ﬁeld of view, and provide state
management. They enable the developer to worry about interactions
between virtual objects rather than low-level object rendering. OpenSceneGraph
is open source and is one of the most widely used scene
graph libraries. It is written in C++ and uses OpenGL for it’s graphics
rendering, and has support for a large number of plug-ins that provide
support for additional functionality such as model loading, shadow rendering,
and physics.
By combing ARToolKit and OpenSceneGraph together, osgART
makes it very easy for developers to create interactive AR applications
with advanced rendering techniques. The osgART library includes
nodes for video capture, video viewing and for camera tracking. Figure
6.1 shows a simple osgART scene graph, showing nodes for video
input and tracking, combined with nodes for graphics rendering. The
end result is a simple AR scene with a 3D model of a cow overlaid on
an ARToolKit tracking marker. The osgART library is designed using a
modular approach meaning that it is easy to replace one module for an-
2
http://www.artoolworks.com
3
http://www.osgart.org
4
http://www.openscenegraph.org
150 AR Development Tools
Figure 6.1: Simple osgART scene graph for AR scene representation.
http://www.osgart.org
other in the application. So, if developers don’t want to use ARToolKit
as the base tracking library, they can replace it with another tracking
solution, such as the BazAR5 image tracking library, or PTAM6.
Oda and Feiner’s Goblin XNA platform7 uses a similar approach,
combining the artTag tracking library with the Microsoft XNA game
platform8 to create an infrastructure for developing AR games, on top
of an existing non-AR game development environment. GoblinXNA
includes modules for AR tracking, 3D user interface elements, video
capture, physics based interaction, and displaying output into AR head
mounted displays. Using Goblin XNA it was relatively easy to modify
an existing XNA car racing game, and make an AR version of it [Oda
et al., 2008].
Another AR framework developed for building complete systems
was Studierstube9 [Schmalstieg et al., 2002]. This was based on a het-
5
http://cvlab.epfl.ch/software/bazar
6
http://www.robots.ox.ac.uk/~gk/PTAM
7
http://graphics.cs.columbia.edu/project/goblinxna
8
http://xbox.create.msdn.com/en-US
9
http://www.cg.tuwien.ac.at/research/vr/studierstube
6.1. Low Level software libraries and frameworks 151
erogeneous distributed architecture that allowed the user easily incorporate
multiple di erent display types (AR HMDs, projected displays,
desktops, etc) with a variety of input devices and reconﬁgurable tracking
hardware. Studierstube was designed from the ground up to support
collaborative applications using a distributed shared scene graph
and so many of the early application were for shared AR experiences.
Another of its advantages was that it was very easy to rapidly incorporate
di erent interaction techniques such as Personal Interaction Panels
[Szalavári and Gervautz, 1997], 3d viewports, or mobile interfaces.
While Studierstube, osgART and GoblinXNA are freely available as
academic research projects, there are also commercial solutions available.
For example, the Metaio SDK10, provides a comprehensive set of
AR tracking technologies, including marker tracking, image tracking,
3D object tracking, face tracking, and even an external Infrared tracker
sold by Metaio. In terms of visualization, it provides a simple VRML
renderer and an option to use OpenSceneGraph together.
For mobile applications, the Studierstube platform originally developed
for desktop applications was ported over to the Windows CE
and a PDA platform [Wagner et al., 2005], and then Android, with the
Studierstube Light library. However there were many components of
Studierstube that did not have equivalent libraries on the PDA platform
and so had to be written from scratch. This included a mobile
rendering library (KLIMT), tracking (OpenTracker) and libraries for
hardware abstraction (PocketKnife), as well as existing libraries for mobile
SceneGraphs (Coin) and communication (ACE). This framework
was then used to build many mobile AR applications, including The
Invisible Train [Wagner et al., 2004], the ﬁrst self-contained handheld
collaborative AR experience. Figure 6.2 shows the typical components
used to build a mobile AR application using Studierstube Light and
the Invisible Train application.
ARToolkit for Mobile11 is a version of the ARToolkit library available
for mobile platforms including iOS and Android. It is also integrated
with the OpenSceneGraph library for rich real-time computer
10
http://www.metaio.com/products/sdk/
11
https://www.artoolworks.com/products/mobile/artoolkit-for-mobile
152 AR Development Tools
Figure 6.2: Studierstube mobile AR framework (left) and the Invisible Train app
developed with it (right). [Wagner et al., 2005, 2004]
graphics rendering. Since it is a porting of C language based desktop
version of ARToolkit, ARToolkit for Mobile needs a native development
kit to use it on Android platform. Alternatively, the AndAR library12
ﬁlls this gap by providing Java based wrapping classes of ARToolkit
library, making the development environment friendlier to Android and
Java developers.
For Android, DroidAR13 also provides vision-based marker tracking,
which is based on the OpenCV computer vision library and uses
square markers similar to that of ARToolkit. One of the most popular
low level libraries for mobile AR applications is the Vuforia library from
Qualcomm that specializes in tracking natural features of arbitrary images,
and also supports tracking 3D objects with multiple planar image.
Vuforia supports both iOS and Android platforms.
Finally, the Metaio SDK supports both ﬁducial markers and natural
feature based image tracking, and recently added a feature for tracking
an arbitrary 3D structure in the real world scene. This allows users
to track camera position from 3D models that provides more robust
tracking (see Figure 6.3). It also o ers its own 3D rendering engine
that allows developers to work in high-level in terms of 3D visualization.
Mobile SDK is also available both for free and under commercial license.
The low level libraries mentioned so far use computer vision techniques
for AR tracking. However there are also libraries that use sen-
12
http://code.google.com/p/andar
13
https://github.com/bitstars/droidar
6.2. Rapid Prototyping/Development Tools 153
Image courtesy of Metaio.
Figure 6.3: Metaio model based tracking. http://www.metaio.com
sors for outdoor location based AR. In addition to computer vision
based tracking, DroidAR also supports developing location-based mobile
AR applications using sensors (e.g., GPS, intertial) on the mobile
device. The Metaio Mobile SDK also provides a solution for developing
location-based mobile AR applications. In addition to sensor based
outdoor location tracking, it provides a partial solution for indoor environment
by coding location information in a square ﬁducial marker.
While these frameworks focus on providing low-level functionalities,
the HIT Lab NZ OutdoorAR14 library [Lee and Billinghurst,
2013] provides high-level abstraction of geo-located scenes. Designed
in a component based structure, the library allows developers to focus
on high-level design of AR applications such as domain content and
logic and user interface design, while it also provides direct access to
low-level functionalities.
In addition to the low level libraries mentioned there are many other
SDKs that can be used for low-level AR application development. The
Technical University of Graz has a list of all available libraries15.
6.2 Rapid Prototyping/Development Tools
Using a low level software library and framework like Studierstube or
OsgART provides the maximum ﬂexibility in development of the AR
14
http://www.hitlabnz.org/mobileAR
15
http://www.icg.tugraz.at/Members/gerhard/augmented-reality-sdks
154 AR Development Tools
application, but can also be time consuming and requires considerable
programming skill. In some cases it can be helpful to rapidly prototype
an idea to show to end users or clients before undertaking a major development
exercise. In this section we brieﬂy overview a range of di erent
tools that could be used for rapid prototyping of AR experiences.
Adobe Flash16 is one of the most popular authoring tools for creating
multimedia content for the web and desktop platform, and can
also be used to prototype a variety of desktop and mobile experiences.
Flash developers can easily create AR experience by using one of the
AR plug-ins available, such as FLARToolKit17 or FLARManager18.
FLARToolKit is a version of ARToolKit ported over to Flash that allows
people to have AR experiences from within their web browser.
Just by writing a few lines of code users can activate their camera,
look for AR markers in the camera view, and load and overlay virtual
content onto the tracked image. FLARToolKit makes it possible for
anyone with a Flash-enabled web browser and camera connected to
their computer to have an AR experience, and also makes it easier to
develop those experiences. These attributes make it a perfect platform
for prototyping AR applications.
The development of FLARToolKit has lead to an explosion of webbased
AR applications, especially for marketing. For example, when
GE were seeking a way to explain about Smart Grid power technology
they created a website with a simple AR marketing experience that
would allow people to see a variety of di erent clean energy power
generation operations superimposed over a printed marker on paper.
Users would hold up the paper and see a virtual windfarm appear, and
when they blew, the blades of the windmills would turn around faster.
Users found this very engaging and uploaded videos to YouTube
of themselves playing with the virtual objects, which attracted
millions of views. Since this early campaign there have been hundreds
of websites that have provided other similar AR marketing experiences.
16
http://www.adobe.com/flash
17
http://www.libspark.org/wiki/saqoosha/FLARToolKit/en
18
http://words.transmote.com/wp/flarmanager
6.2. Rapid Prototyping/Development Tools 155
FLARManager is the equivalent of osgART for Flash. Just like
ARToolKit or ARTag, FLARToolKit just provides AR tracking services
with little support for 3d graphics or interaction. FLARManager
provides an easy way to add more interactivity to Flash based AR
applications and also combine a variety of di erent AR tracking and
graphics libraries together. Table 6.2 shows the di erent Flash based
AR tracking and 3D graphics libraries supported by FLARManager.
Using the FLARManager library it is very easy for users to build Flash
based AR applications with support for complex 3D model and animation
loading, setting di erent camera parameters, loading of tracking
images and many other features.
Table 6.2: FLARManager support for di erent 3d graphics and AR tracking li-
braries.
AR Tracking Library 3D Graphics Library
FLARToolkit Alternativa3D
ﬂare*tracker Away3D
ﬂare*NFT Away3D Lite
Papervision3D
Sandy3D
There are also similar Flash based commercial solutions available
from companies such as Metaio and Total Immersion. As an alternative
to Flash, there are also AR plug-ins for other authoring tools, such
as SLARToolKit19 for Microsoft Silverlight. Recently researchers have
also begun exploring how to using HTML5 to create web-based AR
experiences that don’t require any third-party browser plug-in. For
example, Oberhofer et al. [2012] has shown how Javascript, WebGL and
HTML5 can be used to implement AR natural feature tracking viewable
on desktop or mobile web browsers. Similarly Ahn et al. [2013] have
developed a complete mobile AR framework using just normal HTML
and other web component technologies.
In addition to web-based AR prototyping tools, there are a number
of desktop and mobile tools for rapidly creating AR applications. One
of the most popular tools to use is OpenFrameworks20, an opensource
19
http://slartoolkit.codeplex.com
20
http://openframeworks.cc
156 AR Development Tools
C++ middleware library that makes It very easy to build interactive
experiences. It provides a framework for combining a wide range of
di erent interactive technologies, and is designed to work as a general
purpose glue that wraps together several commonly used libraries21.
For example, an AR application with gesture-based input could be
developed by combining a tracking library such as ARToolKit, with
hand tracking libraries, 3D scene graph libraries and perhaps audio
playback libraries. Previously this would have required using very different
programming APIs for each library and sometime incompatible
event mechanisms. Openframeworks provides a common API for all
these di erent libraries and centrally manages application messaging
and events.
For mobile applications, one of the most useful prototyping tools
are AR browser applications such as Junaio, Layar or Wikitude. These
are free applications that run on the mobile device, but connect back to
servers for providing content and specifying the user interface elements
on the mobile. Using these tools developers rapidly prototype application
ideas by providing their own content information. For example,
Junaio server architecture allows content developers to provide content
on their own server which could be accessed by the mobile AR browser
through the Junaio server. Junaio provides support for either image
based tracking or location based tracking using GPS and compass sensor
input. So with a small amount of code on the server developers can
rapidly test out their mobile AR application ideas.
6.3 Plug-ins to Existing Developer Tools
The software libraries mentioned in the previous section require considerable
programming ability to use them for creating AR applications.
However there are other tools that are plug-ins to existing software
packages that add AR functionality to existing interactive 2D and 3D
content authoring tools. AR plug-ins for non-AR authoring tools are
useful when one already knows how to use the authoring tool that
the plug-in supports. These types of tools simply add AR visualization
21
http://ofxaddons.com
6.3. Plug-ins to Existing Developer Tools 157
Figure 6.4: DART plug-in interface inside Macromedia Director. [MacIntyre et al.,
2004]
and tracking function to existing authoring tool, and rely on content
authoring functions provided by the main authoring tool.
One of the good examples of plug-in based AR authoring tool is
the Designers AR Toolkit (DART)22 [MacIntyre et al., 2004]. This was
designed as a plug-in to the popular Macromedia Director software for
multimedia development. DART was created speciﬁcally for designers
to enable them to rapidly develop AR applications with a tool that they
were already familiar with. Using a the Director graphical interface (see
Figure 6.4) the user was able to mix drag and drop visual input with
lingo scripting to create interactive AR experiences. Using the plug-in
architecture of Director, DART provided AR speciﬁc extensions such
as support for ARToolKit marker based tracking, physics, 3D camera
viewing, and live video display. The authors report that designers found
DART very easy to use and were able to successfully create a wide range
of di erent AR applications, such as AR tours of historic sites, and AR
theatrical experiences.
There are also various AR plug-ins available for 3D content authoring
tools, as well. For example, AR-Media23 provides a range of
22
http://ael.gatech.edu/dart/
23
http://www.inglobetechnologies.com/en/products.php
158 AR Development Tools
plug-ins for the Trimble SketchUp 3D modelling software and other
tools such as 3ds Max, Maya and Cinema 4D. Using AR-Media plugins,
content creators can model in SketchUp, then immediately see their
model in an AR scene. There is also a free AR-Media player that can be
distributed to allow other people to see the AR experience. However,
while 3D modeling and animation software can be useful for creating
3D scenes, they have limitations in providing interactive features.
To create highly interactive 3D applications, there are also various
interactive 3D graphic authoring tools such as Unity24, Vizard25,
and Quest3D26. While most of these tools have their own plug-ins for
AR visualization, Unity has widest range of solutions available for AR
visualization. For example, ARToolkit for Unity from ARToolworks27,
provides both marker and image based tracking and AR visualization.
Similarly, the Vuforia tracking library has an AR plug-in that works
with Unity3D to create mobile AR applications for the iOS and Android
platform. In this case developers can use the normal Unity3D
visual programming and scripting interface to create rich interactive
experiences that can then be
While AR plug-ins provide a easy and quick method for developing
AR applications, they have certain limitations. User interface and
interaction design could be limited to those provided by the authoring
tool, and the software and hardware conﬁguration also depends on
what is provided by the tool. In addition, the content produced with
an authoring tool usually becomes dependent on the proprietary player
software. In order to create an AR application with highly customized
interface and interaction, it is unavoidable to use low-level software
development tools such as software libraries.
6.4 Stand Alone AR Authoring Tools
A third type of AR authoring tool is a stand alone application that
allows users to create their own AR experiences will no-programming
24
http://unity3d.com
25
http://www.worldviz.com
26
http://quest3d.com
27
https://www.artoolworks.com/products/integrated/artoolkit-for-unity
6.4. Stand Alone AR Authoring Tools 159
Figure 6.5: AR furniture assembly application created with the AMIRE framework.
[Zauner et al., 2003]
experience. Unlike the plug-ins mentioned in the previous section, these
tools don’t require any additional software to work. A wide range of
stand alone authoring tools provides di erent set of functions. Scene
construction, animation, and adding interactive behaviours are the
common function sets that many AR authoring tools provide. Some
tools only provide scene construction functions, while others have full
capability of building complex interactive behaviours. Most of the AR
authoring tools provide graphical user interfaces so that a user who has
basic personal computer skills could easily learn and use them, and they
are useful for non-programmers who want to build AR application.
Early examples of AR authoring tools include AMIRE [Dörner
et al., 2003] and CATOMIR [Zauner and Haller, 2004]. AMIRE was
designed as an object oriented application framework that could be
used to build an AR application out of various available components.
AMIRE had a number of features including (1) a generic conﬁguration
mechanism of components by so-called properties, (2) slot based
communication between components, (3) components for 2D and 3D
interaction, and (4) support for prototyping new components. Zauner
[Zauner et al., 2003] shows how AMIRE was used to create author
an AR application for step by step furniture assembly (Figure 6.5).
Träskbäack and Haller [2004] used AMIRE to develop a tablet based
Mixed Reality training application for an oil reﬁnery.
However, AMIRE was a component framework that required some
programming to use. Zauner and Haller [Zauner and Haller, 2004,
Haller et al., 2005] extended this by creating CATOMIR (Component
160 AR Development Tools
Figure 6.6: Catomir visual interface showing visual links between components.
[Haller et al., 2005]
oriented Authoring Tool for Mixed Reality) a visual programming interface
based on top of AMIRE. Using CATOMIR a user could dragand-drop
components from a pool and connect them with visual links
to deﬁne logical behaviours in the AR application. The visual interface
was shown overtop of the live camera view and AR scene, allowing
the user to switch rapid between authoring and viewing the AR
experiences. CATOMIR supported a three-step authoring approach,
(1) ﬁnding the right components for the application, (2) tweaking the
components parameters, and (3) connecting components to deﬁne the
logical behaviour of the AR application (see Figure 6.6). In their paper
they describe how CATOMIR was used to develop a furniture assembly
application. However one of the disadvantages of this approach is
that AR applications can only have the functionality supported by the
components, and it is di cult to add new components.
BuildAR28 is a standalone AR authoring tool for quickly building
AR scenes. It provides computer vision based tracking of both square
markers and arbitrary images, and users can add 3D models, images,
text, video and sound to the AR scene (see Figure 6.7). While it can
play animations built into the 3D model, BuildAR lacks tools for creating
animations, and also does not support adding interactive behaviours
to the AR scene. The content built by the user is saved into a proprietary
ﬁle format, and can be viewed using the BuildAR viewer software
that is freely available for download. More recently Choi et al. [2010]
developed a similar desktop authoring tool that added interactive behaviours
to overcome the limitations of BuildAR.
28
http://www.buildar.co.nz
6.4. Stand Alone AR Authoring Tools 161
Figure 6.7: The BuildAR authoring tool. http://www.buildar.co.nz
In addition to basic AR scene construction functions mentioned
in BuildAR, D’Fusion Studio29 from Total Immersion provides a GUI
based scene authoring environment using Lua scripting language for
describing scenario and behavior of the AR scene. The content is exported
into proprietary ﬁle format, and it can be shown through its
player software.
Some of the AR authoring tools for desktop applications are also
capable of publishing content onto mobile platforms. For instance,
D’Fusion Studio provides a solution for viewing the content on a mobile
platform using its own viewer software. With these AR plug-ins
and standalone AR authoring tools, users can use the same authoring
environment to create the content and export it to run on di erent
platforms including mobile environment.
On the other hand, there is also AR authoring tools that speciﬁcally
targets for developing mobile AR applications. Metaio Creator30 is a
simple tool to create AR scenes targeted for the Junaio mobile AR
browser. Users can easily include their own tracking images and add
content (e.g., pictures, 3D models, video and audio clips) onto it. It
also provides simple behaviours that triggers actions (e.g., showing or
playing the content, simple animation) based on simple events (e.g.,
user tapping on the content).
29
http://www.t-immersion.com/products/dfusion-suite/dfusion-studio
30
http://www.metaio.com/products/creator
162 AR Development Tools
Figure 6.8: Layar Creator, a web-based AR authoring tool. http://www.layar.
com/creator
Figure 6.9: Wikitude Studio, a web-based AR authoring tool. http://studio.
wikitude.com
Recently, web-based authoring tools that can deploy AR content directly
onto mobile AR browsers became popular. The Layar Creator31
is a web based authoring tool that specializes for creating AR links on
printed materials (see 6.8). At the Layar Creator website, users upload
images of the printed pages to track, and then they can add virtual
buttons that have links to various services available on the mobile device,
such as opening a webpage or a youtube video, calling to a speciﬁc
number, or sharing the information on social network service. The created
content is published as a layer in the Layar mobile AR browser.
31
http://www.layar.com/creator
6.5. Summary 163
Figure 6.10: Hoppala web-based geo-tagging service. http://www.
hoppala-agency.com
Another web-based authoring tool Wikitude Studio32 allows users
to create rich mobile AR content and deploy either onto the Wikitude
AR browser app or even create a custom mobile app (see Figure 6.9). It
supports various types of media including 3D models and animations
which can be registered to the real world with computer vision-based
tracking technology.
There are also web-based authoring tools focusing on location based
mobile AR applications. For example, Hoppala Augmentaion33 is a
web-based geo-tagging tool for mobile AR browsers (see Figure 6.10).
Instead of asking developers for a list of geo-coordinates of custom
points of interest, it provides an interactive web-based map interface
for adding custom geo-tags. The geo-tags created by the user can be
browsed on a number of AR browsers including Wikitude, Layar, and
Junaio.
6.5 Summary
In this section we reviewed various types of AR development and authoring
tools. Low level software libraries and frameworks provide high-
32
http://studio.wikitude.com
33
http://www.hoppala-agency.com
164 AR Development Tools
est level of ﬂexibility while they require programming skills. On the
other hand, standalone authoring tools enable end users to create their
own AR content easily with minimal computer skills, although the content
created are mostly simple. While commercial authoring tools and
AR plug-ins for game engines are widely adopted, the main interface
of these authoring tools still remain to be based on 2D GUI which has
certain limitations for manipulating 3D content. As the technology advances,
it is expected that the authoring and development tools will
adopt 3D and AR interfaces more actively as shown in early experiments
[Lee et al., 2004b, 2005, Ha et al., 2010, 2012]. In the next section
we review interaction technologies for AR systems.
7
AR Input and Interaction Technologies
In this section we review common input methods and interaction technologies
for Augmented Reality. AR systems can incorporate various
types of input methods, ranging from traditional 2D user interfaces
(UI), such as keyboard, mouse and touch screen input, to 3D and multimodal
interfaces such as handheld wands, speech, gesture, etc. Di erent
input methods have been used for di erent types of AR application
depending on the user interaction tasks required in the application, and
so the type of interface used has evolved over time. There have been a
number of AR interface types developed since the 1960’s, including:
1) Information Browsers: Interfaces for showing AR information on
the real world
2) 3D User Interfaces: Using 3D interaction techniques to manipulate
content in space
3) Tangible User Interfaces: Using real objects to interact with AR
virtual content
4) Natural User Interfaces: Using natural body input such as free
hand gestures
165
166 AR Input and Interaction Technologies
Figure 7.1: NaviCam AR browser. [Rekimoto and Nagao, 1995]
5) Multimodal Interfaces: Using combined speech and gesture input
We will review each of these interface types in turn and describe
some interaction technologies used to implement them.
7.1 AR Information Browsers
AR information browsers are one of the representative types of AR
applications where AR displays are considered as a window into an
information space, and the main task of the user is to manipulate this
window to browse the information. One of the earliest AR systems,
NaviCam [Rekimoto and Nagao, 1995] is a typical example of AR information
browser (see Figure 7.1), as well as many of the AR browser
apps on smartphones such as Wikitude, Junaio, and Layar. This type
of AR application requires the most basic interaction tasks: viewing
the visualized AR scene and browsing the information provided.
Compared to VR systems where a navigational interface is usually
needed to manoeuvre in the 3D virtual space, AR systems visualize
virtual objects registered to the real world space. So users do not need
additional interface object for navigation, but they can simply use their
7.2. 3D User Interfaces in AR 167
natural skills to move through the physical environment. For instance,
when using head mounted AR displays users simply have to turn their
head to change their view, while with handheld AR displays holding
and moving the device and pointing at the direction to view is su cient.
Other types of interactivity common to AR information browsers
includes choosing di erent information (e.g. channels) to view, ﬁltering
information shown, navigating into details of the information provided,
and changing visualization style, etc. Most of these interactions can be
accomplished using traditional 2D graphical user interfaces (GUI) and
screen input. Users can use traditional input devices, such as keyboard,
mouse, joystick, touch pad or touch screen, to interact with the GUI
provided with the AR content.
AR information browsers are one of the important classes of AR
interface that is widely used in AR navigation with wearable computers,
AR simulation and training. The interaction method provided is simple
and easy to learn as the users can use their knowledge of traditional
mobile user interfaces. On the other hand, these types of applications
have limitations in providing direct interaction with virtual objects.
7.2 3D User Interfaces in AR: Interacting with Virtual Objects
through Controllers
One of the straightforward approaches for supporting direct interaction
with virtual objects is to adopt traditional 3D user interfaces techniques.
Various 3D UI technologies and interaction methods have been
investigated to provide direct interaction with 3D objects in virtual
environments [Bowman et al., 2004], and many of these techniques can
be applied to AR as well.
Bowman [Bowman et al., 2004] summarizes various types of 3D interactions
into three categories: (1) navigation, (2) selection, and (3)
manipulation. While the 3D navigation techniques might not be directly
applicable to AR applications, as navigation in AR environments
is naturally achieved by the users’ moving their body, selection and
manipulation interaction techniques can be easily adopted in AR applications
[Schmalstieg et al., 2000]. For example in Kiyokawa’s VLego
application users have their hands tracked in 3d space and can reach
168 AR Input and Interaction Technologies
Figure 7.2: Using 3D user interface in a collaborative AR system. [Kiyokawa et al.,
2000]
out and directly pick up virtual blocks in the collaborative AR interface,
in much the same way that they could interact in an immersive
VR application (see Figure 7.2) [Kiyokawa et al., 1999].
In many cases, 3D UIs use input devices that can be used for 6
degree of freedom (DOF) manipulation (translation and rotation in
3D) of virtual objects. There are various types of devices invented and
used in VR and 3D UI ﬁeld, including 3D mouse or wand type pointing
devices, 6 DOF joysticks, spaceballs, etc. Among them, 3D motion
tracking sensors are one of the most widely used technologies. These
allow tracking of various physical objects including the user’s body
motion, and let users to point at or manipulate virtual objects. VLego
used 3D motion tracking magnetic sensors for manipulating virtual
objects registered in the physical environment.
Haptic devices (e.g. the Phantom1) are another type of traditional
3D UI that researchers investigated using in AR environments [Vallino
and Brown, 1999, Adcock et al., 2003] (see Figure 7.3). Haptic interfaces
not only work as a 3D pointing device but also provide force and tactile
1
http://geomagic.com/en/products-landing-pages/haptic
7.3. Tangible User Interface in AR 169
Figure 7.3: Using haptic interface in AR. [Adcock et al., 2003]
feedback which complements the visual experience by creating illusion
of physical existence of a virtual object.
Traditional 3D UI can provide good interactivity in AR applications
for entertainment, design, and training. With 3D UI, users can interact
with 3D virtual objects everywhere in space in a natural and familiar
way. While using traditional 3D UI for AR interactions beneﬁt from
previous work in VR and 3DUI ﬁelds, there are certain limitations
with this approach as well. One of the most prominent problems is
that the methods used for interacting with virtual objects are di erent
from interacting with physical objects. With most of the traditional
3D UI, users will have to hold a control device and use it to point and
manipulate virtual objects, while with physical objects, they mainly
use their hands for direct manipulation.
7.3 Tangible User Interface in AR
Using physical objects as a medium for interacting with computers
[Ishii and Ullmer, 1997] has been one of the main themes in modern
human-computer interface technology research. The concept is referred
to as Tangible User Interfaces (TUI). Tangible UI use physical objects
for representing virtual entities and information, and to bridge between
the physical and digital worlds.
170 AR Input and Interaction Technologies
Figure 7.4: Tangible AR interface with a physical book with virtual object overlaid
on it.
While Tangible UI provide natural and intuitive interaction with
digital information through manipulating physical objects, they can
have limitations with display capabilities, either showing very limited
information with di erent status of physical objects. Even when using
a visual display in combination with Tangible UI, it still has problem of
separation between the physical interaction space and the display where
the virtual information is shown. For example, in a TUI digital content
may be projected on a physical surface or shown on a screen while the
user manipulates physical controllers some distance away creating a
gap between the interaction space and presentation space.
To reduce the gap between the interaction methods for virtual and
physical objects, Kato et al. [2000] proposed the concept of Tangible
AR (TAR). TAR uses Tangible UI as input interaction metaphor while
using AR for visualizing virtual information overlaid on the physical
object used for interaction. Figure 7.4 shows a typical example of Tangible
AR interface that uses a physical book as a Tangible UI while
visualizing virtual objects overlaid on its pages. As can be seen in this
case the interaction space and display space are seamlessly merged to-
gether.
Billinghurst et al. [2005] deﬁnes the characteristics of Tangible AR
interfaces as 1) each virtual object being registered to a physical object,
7.3. Tangible User Interface in AR 171
and 2) the user interacting with virtual objects by manipulating physical
objects. To use physical objects as input devices for interaction
requires accurate tracking of the objects, and for this purpose many
Tangible AR applications use computer vision based tracking software,
such as ARToolkit2. With accurate tracking, the computer system not
only can recognize and identify di erent physical objects, but also can
estimate the 3D motion of the objects so that the pose of the object or
the motion gesture can be used for various interactions.
The basic goal of designing a Tangible AR interface is to map physical
objects (input) with virtual objects (output) using an appropriate
interaction metaphor. Lee et al. [2007] summarizes common interaction
metaphors and methods used in various Tangible AR applications, and
provides guidelines for designing and implementing interaction methods
for Tangible AR applications. Providing a static mapping between
the two is a very basic yet useful way for Tangible AR applications
where the main purpose is to view 3D objects. However, in order to
provide more interactivity, many Tangible AR applications also dynamically
map them using a metaphor of tools for performing certain
tasks.
Similar to how physical tools are designed, Tangible AR interfaces
use two di erent design approaches, or a combination or both methods.
One approach is space multiplexed interfaces where each physical tool is
dedicated for one function, and the other is time multiplexed interfaces
where a tool is used for many functions and purposes depending on the
status and context. Space multiplexed interfaces are considered to be
more intuitive to learn as each function is mapped to a single tool, while
time multiplexed interfaces could have di erent function depending on
the status and context hence requiring further steps of understanding
and learning the functions. Figure 7.5 shows examples of Tangible AR
applications that use time multiplexed and space multiplexed interfaces
for interaction.
While Tangible AR interfaces provide an intuitive, natural and
seamless way to interact with both physical and virtual objects in AR
applications there is also a drawback of requiring physical objects for
2
http://www.hitl.washington.edu/artoolkit
172 AR Input and Interaction Technologies
Figure 7.5: Examples of time multiplexed (left: VOMAR [Kato et al., 2000]) and
space multiplexed (right: Tiles [Poupyrev et al., 2002]) Tangible AR interfaces.
interaction which might be inappropriate for mobile or wearable AR
applications.
7.4 Natural User Interfaces in AR: Body Motion and Gesture
With traditional 3D user interfaces, body motion can be tracked and
recognized using various types of motion tracking sensors that user’s
wear. Various motion sensors of di erent size and shape can also be
applied in AR applications, ranging from a glove type device [Lee et al.,
2010b] used for tracking hand gestures to full body motion tracking
systems. An overview of various types of tracking technologies can be
found in section 4.
With advance in computer vision technology, AR systems became
capable of recognizing user’s body motion and gesture in real time
without requiring the user to wear any sensors. For example, Lee and
Hollerer [2007] developed the HandyAR system capable of bare hand
interaction using a standard web camera, although the supported gestures
were limited (see Figure 7.6). Another work [Lee et al., 2008b]
used a stereo camera to detect natural hand interaction. Most of this
research focused on recognizing the human body through detecting
features or skin colour Lee and Hollerer [2007], McDonald and Roth
[2003] from video image that had limitations of accuracy of recognized
hand postures yet requiring high speed processing power for running
computer vision algorithms. Some research [Lee et al., 2004a] tried to
7.4. Natural User Interfaces in AR 173
Figure 7.6: HandyAR free hand interaction with AR content. [Lee and Hollerer,
2007]
overcome this through reversing the focus from human body to objects
of interest and used occlusions by the user’s body as an input for
interaction.
With recent technical advances and commercialization of depth
cameras (e.g. Microsoft Kinect3), more accurate motion and gesturebased
interaction became widely available for VR and AR applications
(see Figure 7.7). Use of depth camera in AR applications enabled
tracking dexterous hand motions for physical interaction with
virtual objects using bare hands. Microsoft HoloDesk [Hilliges et al.,
2012] demonstrated use of the Kinect camera to recognize the user’s
hands and other physical objects and allow them to interact with virtual
objects shown on an optical see-through AR workbench. ARET
[Corbett-Davies et al., 2013] shows a similar technique applied in AR
based exposure treatment application. With growing interest in using
hand gestures for interaction in AR applications, Piumsomboon et al.
[2013] categorized a user-deﬁned gesture set that can be applied to
di erent tasks in AR applications.
Integrating hand motion and gesture based interaction with mobile
or wearable AR systems is one of the topics actively investigated. Mistry
and Maes [2009] demonstrated a wearable camera and projector
system that recognizes user’s hand gestures for interaction by detecting
ﬁngertips with color markers (see Figure 7.8). Using the camera
on Google Glass, On the Go Platforms4 developed hand gesture recognition
software though supported gestures are limited to simple ones.
Along the e orts for improvement, mobile version of depth sensing
3
http://www.microsoft.com/en-us/kinectforwindows
4
http://www.otgplatforms.com
174 AR Input and Interaction Technologies
Figure 7.7: Hand gesture based physical interaction with virtual objects (left:
HoloDesk [Hilliges et al., 2012], right: ARET [Corbett-Davies et al., 2013]).
Figure 7.8: Tracking ﬁngers positions in the Sixth Sense projection based AR
interface. [Mistry and Maes, 2009]
cameras are also under development, such as SoftKinetic5, that would
beneﬁt from implementing more accurate hand gesture interaction in
mobile and wearable AR systems. There are also other approaches to
use biometric sensors for gesture interaction in mobile and wearable
environment, such as the Myo6 gesture armband controller.
7.5 Multimodal Interaction in AR
To provide richer interactivity in AR applications, there have been efforts
to combine di erent modalities of input. Among di erent combination
of input modalities, speech and gesture recognition combined is
5
http://www.softkinetic.com
6
https://www.thalmic.com/myo
7.5. Multimodal Interaction in AR 175
one of the most widely and actively researched combinations. Previous
research has shown that multimodal interfaces (MMI) that combine
speech and gesture input can be an intuitive way to interact with 2D
interfaces and 3D graphics on the desktop [Bolt, 1980, Hauptmann,
1989, Cohen et al., 1997]. This is because the input modalities are
complimentary, with speech being good for quantitative input, while
gesture is ideal for qualitative input [Cohen et al., 1989].
A number of researchers have used speech and gesture input in Virtual
Reality interfaces. For example, the Naval Research Laboratory’s
Dragon 3D VR system [Cohen et al., 1999] used a multimodal system
to create digital content in a 3D topographical scene, allowing users to
create and position objects by speaking while gesturing in 3D space.
Ciger et al. [2003] presented a multimodal user interface that combined
a magic wand with spell casting. The user could navigate in the virtual
environment, grab and manipulate objects using a combination of
speech and pointing with a physical wand. Similarly, LaViola Jr. [1999]
developed an immersive VR interface that used whole-hand input captured
with a data glove and speech input for interior design. The user
could create virtual objects with speech and then perform object manipulation
using hand gestures. These examples are just a few of many
and show that MMI are also a natural way to interact in immersive
virtual environments.
However, despite this earlier research there has relatively little research
on the use of multimodal interaction for AR. One of the ﬁrst
systems was the SenseShapes work of Olwal et al. [2003] which used
volumetric regions of interest that can be attached to the user, allowing
the user to point at virtual objects in the real world and manipulate
them with speech commands. Kaiser et al. [2003] extended Olwal’s
SenseShapes work by focusing on mutual disambiguation between input
channels (speech and gesture) to improve interpretation robustness.
Heidemann et al. [2004] presented an AR system designed for online acquisition
of visual knowledge and retrieval of memorized objects. More
recently, Irawati et al. [2006] added speech input to the earlier VOMAR
[Kato et al., 2000] interface allowing people to create AR scenes using
a combination of speech and tangible paddle gestures. In a user study
176 AR Input and Interaction Technologies
Figure 7.9: Natural hand gesture based multimodal AR system. [Lee et al., 2013b]
with her system, she found that people were able to complete a scene
creation task over 35% faster with the interface using combined speech
and paddle gesture input, than using gesture input on its own. The
multimodal interface was also more accurate and was preferred by the
users.
These multimodal systems all required the users to either wear
gloves or hold a special input tool in their hands. More recently systems
have been developed that support natural bare hand input. For example,
Kolsch et al. [2006] create a multimodal interface for outdoor AR
that used a wearable computer to perform hand tracking and recognize
command gestures, enabling users to interact in a very intuitive manner
with AR cues superimposed the real world. Piumsomboon et al.
[2014] used a Wizard of Oz technique to classify the types of gesture
that users would like to use in an AR multimodal interface, helping to
inform the design of such systems. Finally, Lee et al. [2013b] developed
a multimodal system that used a stereo camera to tracking users hand
gestures and allow them to issue speech and gesture commands to manipulate
virtual content in a desktop AR interface (see Figure 7.9. A
user study with the system found that the MMI was 25% faster that
using gesture only interaction and that users felt that the MMI was
the most e cient, fastest, and accurate interface.
7.6 Other Interaction Methods
While speech recognition is the most actively investigate as audio input,
there are also attempts to detect other types of sound for interaction
7.7. Summary 177
Figure 7.10: Whistle recognition in GeoBoids mobile AR game. [Lindeman et al.,
2012]
in AR applications. For instance, Lindeman et al. [2012] used whistle
recognition in a mobile AR game where users have to whistle for
the right pitch and length to scare virtual creatures in the game (see
Figure 7.10).
Another type of input modality gaining interest of AR researchers is
Brain Computer Interfaces (BCI). While BCI is at the very early stage
of investigation, studies show BCI and AR can be a good combination
where AR can be e cient and powerful tool for enhancing and studying
BCI technology [Kansaku et al., 2010, Lenhardt and Ritter, 2010,
Scherer et al., 2010]. While traditional electroencephalogram (EEG)
sensing equipments required cumbersome setup of wearing the sensors,
recent development of the technology made a headset style sensor (e.g.
NeuroSky7) that can be easily integrated with AR applications (See
Figure 7.11).
7.7 Summary
In this section we have reviewed some of the common input techniques
and interaction methods used in AR systems. As we have seen they have
evolved from simple information browsing which support very little
interaction with the virtual content, to complex interfaces that can
7
http://neurosky.com
178 AR Input and Interaction Technologies
Figure 7.11: A mobile BCI headset from NeuroSky. http://neurosky.com
recognise and respond to natural gesture input and speech. However,
even though there are now a wide range of di erent input methods
available for AR interfaces, they still need to be carefully designed. In
the next section we review research into design guideline and interface
patterns for e ective AR experiences.
8
Design Guidelines and Interface Patterns
As reviewed in the previous section there have been a wide range of
di erent, but there needs to be more research on how to design AR experiences.
In this section we review the design guidelines and interface
patterns that have been developed.
When a new interface medium, like Augmented Reality, is developed
it typically progresses through the following stages:
1) Prototype Demonstration
2) Adoption of Interaction techniques from other interface
metaphors
3) Development of new interface metaphors appropriate to the
medium
4) Development of formal theoretical models for modelling user in-
teractions
For example the well-known desktop metaphor of Windows, Icons,
Menus and Pointers (WIMP) has been through all of these stages.
Formal theoretical models have been developed that can predict exactly
179
180 Design Guidelines and Interface Patterns
Figure 8.1: The three elements to be designed in an AR interface.
how long it will take a mouse with certain characteristics to select an
icon of a given size. Models such as Fitts law [Fitts, 1954] can be used
to calculate expected pointing time in user interfaces. Virtual Reality
interfaces are at the third stage with a number of interface techniques
such as the Go-Go Interaction method [Poupyrev et al., 1996] that is
designed speciﬁcally for object selection and manipulation in immersive
Virtual Worlds.
As can be seen from the previous section, in many ways AR interfaces
have barely moved beyond the ﬁrst two stages. There are many
di erent AR interaction methods and input devices used, but these are
largely versions of existing 2D and 3D techniques from other desktop,
mobile or VR interfaces. For example, most handheld AR applications
use familiar touch screen input and gesture applications from other mobile
phone experiences. There is a need to develop interface metaphors
and interaction techniques speciﬁc to AR [Billinghurst et al., 2005].
Developing an appropriate interface metaphor can be achieved by
having a deeper understanding of AR interfaces. Unlike most other interfaces,
Augmented Reality experiences have a close relationship between
real and virtual objects. MacIntyre points out that AR design is
driven by the need to deﬁne and fuse the relationship between entities
in the physical world and virtual world [MacIntyre, 2002]. The basic
goal is the map user input with physical objects onto computer generated
virtual output using an appropriate interaction metaphor. Thus
there are three components that must be designed in an AR application
(see Figure 8.1): (1) the real physical objects, (2) the virtual elements
to be displayed, and (3) the interaction metaphor that links the real
and virtual elements together.
181
Design of the physical and virtual elements can be guided by the
concept of a ordances developed by Gibson [1979] and ﬁrst applied to
interface design by Norman [1988]. Norman deﬁnes a ordance as " ..
the perceived and actual properties of the thing, primarily those fundamental
properties that determine just how the thing could possibly
be used" [Norman, 1988]. For example a chair a ords sitting on, and a
hammer a ords holding by the handle. Even though they are not real
objects, computer graphics contact can also have visual a ordances
showing how they should be manipulated. So a virtual box could have
a handle on the top showing that it could be picked up.
In the context of an AR interface, the designer should make sure
that the shape of the physical input objects and the virtual content
shown on the both provide good a ordances showing how they should
be used. A good example of this is the Augmented Chemistry application
developed by Fjeld and Voegtli [2002]. In this application they
wanted to give the user the experience of being able to assemble their
own chemical molecules, using an AR display and a number of physical
input devices. The application was developed using ARToolKit for
tracking, but the researchers very carefully designed the shapes of the
physical props that the tracking symbols were attached to, shown in
Figure 8.2(a). There was a book that had one atomic element per page,
and users could brown through the virtual atoms simply by turning the
page. When the user found an element they wanted the could point a
shovel shaped selection tool at it, ’scoop’ it up and place is into the
assembly area (see Figure 8.2(b)). Once the virtual molecule starts to
come into shape the user can use a cube shaped rotation device in their
other hand that causes the virtual model to rotate. As can be seen the
physical shape of the input devices provide a strong clue as to how they
are supposed to be used.
In addition to the design of the physical and virtual elements of the
AR application, the interaction metaphor connecting the two must also
be developed. In the case of the Augmented Chemistry the metaphor
was that the user was physically scooping atoms to create molecules.
As described in §7.3 this metaphor is an example of a Tangible AR
approach that combines Tangible User Interface (TUI) methods for in-
182 Design Guidelines and Interface Patterns
(a) Input elements (b) Using the scoop to place an atom
Figure 8.2: The Augmented Chemistry interface. [Fjeld and Voegtli, 2002]
put and AR techniques for display. This means that the interaction
metaphor can be created following design principles learned from tangible
user interfaces. The basic principles of TUI include:
• The use of physical controllers for manipulating virtual content
• Support for spatial 3D interaction techniques
• Support for both time-multiplexed and space-multiplexed inter-
action
• Support for multi-handed interaction
• Matching the interface object a ordances to the task require-
ments
• Supporting parallel activity where multiple objects are being ma-
nipulated
• Collaboration between multiple participants
Papers from Antle and Wise [2013] and Waldner et al. [2006] among
others provide additional design guidelines for TUI applications. Following
these guidelines will ensure that the AR application will be very
intuitive to use because of seamless interaction between the physical
and virtual elements.
8.1. Case Study: levelHead 183
8.1 Case Study: levelHead
The award winning AR art application levelHead1 provides a good
example of how to follow this design approach. It is a compelling application
that involves real cubes with AR tracking symbols on their
sides. The cubes are places on a table in front of a camera and a projected
image on a screen from the camera view shows that the cubes
contain virtual rooms (see Figure 8.3). A human ﬁgure appears in one
of the rooms, and when the cube is titled the ﬁgure will walk to the
opposite corner of the room (see Figure 8.4). Rotating the cube to show
a di erent face to the camera will cause a di erent room to be loaded.
The aim of the installation is to tilt, turn and rotate the cubes until
the ﬁgure can escape from the connected rooms.
Figure 8.3: The levelHead installation. http://julianoliver.com/levelhead
As described above, creating an AR application involves designing
three components; (1) the real physical objects, (2) the virtual elements
to be displayed, and (3) the interaction metaphor. Thus, Level Head
has the following:
1) Real Object: physical cube that the user can easily manipulate in
their hands
1
http://julianoliver.com/levelhead
184 Design Guidelines and Interface Patterns
Figure 8.4: Cube manipulation in levelHead, moving the virtual ﬁgure.
http://julianoliver.com/levelhead
2) Virtual Content: Virtual rooms inside the cubes, human ﬁgure
3) Metaphor: The user is physically tilting the virtual human’s
world,
The combination of easily understood metaphor, intuitive tangible
user interface and e ective AR content, makes this a very well designed
AR experience that is extremely enjoyable to use.
8.2 Using Design Patterns
Another approach that can be used to help with AR interface design
is to use Design Patterns. Introducing the concept of Design Patterns,
Alexander et al. say that "Each pattern describes a problem which occurs
over and over again in our environment, and then describes the
core of the solution to that problem in such a way that you can use
this solution a million times over, without ever doing it the same way
twice." [Alexander et al., 1977] Design Patterns are commonly used in
computer science and interface design. For example, Borchers has written
the book "A Pattern Approach to Interaction Design" [Borchers,
2001] that provides an introduction to the use of Design Patterns in
user interface design, and Yahoo provides an online Design Patterns
library for web developers2.
2
https://developer.yahoo.com/ypatterns
8.2. Using Design Patterns 185
Reicher et al. [2003] explore how Augmented Reality development
can be improved by following a Pattern Language approach. This is
built on earlier work by MacWilliams et al. [2004]. Their approach is
mostly focused on the underlying software engineering and showing
how complex AR systems can be build by combining modules based on
Design Patterns. In this case each Pattern is described by name, goal,
motivation, a description, usability, consequences, and known project
usage. The DWARF framework [Bauer et al., 2001] is an example of
a component based AR framework based on this Design Pattern ap-
proach.
In contrast to Reicher’s approach, Xu et al. [2011] describe Design
Patterns that can be used for the user experience design of Handheld
AR interfaces. Their focus is on pre-patterns that bridge the gap between
Interaction Design and Game Design. In their paper they list
a number of Design Patterns that can be used, described in terms of
a title, deﬁnition, description, examples, and description of using the
pre-patterns. Table 8.1 shows a list of some of the Design Patterns developed,
expressed as a Title, Interaction Metaphor and the Embodied
Skills that are used. The Embodied Skills are those skills that users
have by virtue of having a body. For example. Body Awareness and
Skills from the Device Metaphors row, refers to the user’s ability to be
aware of their body and how it is able to move real objects.
As an example of how a Design Pattern can be used we consider
the case of Seamful Design. As Table 8.1 shows, the Seamful Design
pattern is that designers need to integrate technology seams into the
Handheld AR game design. The term "technology seam" refers to the
limitations of the technology used in the application. For example, most
handheld AR systems use computer vision based tracking, so one of
the technology seams are the conditions under which the tracking fails
(such as bad lighting or fast camera movement etc).
The handheld AR game Paparazzi3 shows how the Design Pattern
can be followed and the technology seam integrated into the application.
In this game the user plays the role of a Paparazzi photographer
and gets points for taking pictures of a character. The character ap-
3
http://pixel-punch.com/project.php?project=Paparazzi
186 Design Guidelines and Interface Patterns
Table 8.1: Design Patterns for Handheld AR games. [Xu et al., 2011]
Title Meaning Embodied Skills
Device
Metaphors
Using metaphor to suggest available
player actions.
Body A&Sú
Naive physics
Control
Mapping
Intuitive mapping between physical and
digital objects.
Body A&S
Naive physics
Seamful
Design
Making sense of and integrating the technological
seams through game design.
Body A&S
World
Consistency
Whether the laws and rules in the physical
world hold in digital world.
Naive physics
Environmental A&S
Landmarks Reinforcing the connection between
digital-physical space through landmarks.
Environmental A&S
Personal
Presence
The way that a player is represented in
the game decides how much they feel like
living in the digital game world.
Environmental A&S
Naive physics
Living
Creatures
Game characters that are responsive to
physical, social events that mimic behaviours
of living beings.
Social A&S
Body A&S
Body
constraints
Movement of one’s body position, constrains
another player’s action.
Body A&S
Social A&S
ú A&S: short for ’Awareness and Skills’
pears as a virtual model superimposed on a real piece of paper. Once
the user takes a few pictures of the character it will get angry and
appear to jump on the camera screen (see Figure 8.5). The user will
then need to shake their phone as hard as possible to dislodge the character
and continue the photo taking. Thus in this game there are two
distinct types of tracking with a technology seam between them; (1)
computer vision based tracking when the user is pointing the phone at
the AR tracking image, (2) inertial compass tracking when the virtual
character is attached to the phone and the user needs to shake it loose.
Quick shaking of the phone will cause the computer vision tracking
to fail, but in this case the game has designed for that as part of the
game story, transitioning to the compass based tracking. Due to this
design the technology seam becomes transparent and the game play is
not disrupted.
Xu et al. [2011] have many other examples of how handheld AR
games use the Design Patterns listed in Table 8.1 to improve their
game design and so make the game more enjoyable and intuitive.
8.3. Designing for Special Needs 187
Figure 8.5: Two di erent game states in Paparazzi tracking with computer vision
(left) and inertial sensor (right). http://pixel-punch.com
8.3 Designing for Special Needs
So far we have described several techniques that can be used to design
e ective AR experiences. However there are some categories of users
that may need extra attention when designing for them. In particular
Radu and MacIntyre [2012] point out that designing AR systems for
children aged 6 to 9 years old should take into account their developmental
stages. Previous research has shown that Augmented Reality
can be used to create powerful educational experiences [Kerawalla et al.,
2006, Billinghurst, 2002]. At the same time, research outside the AR
ﬁeld has shown that when designing technology for children it is very
important to take into account the children’s developmental abilities
[Bruckman and Bandlow, 2003, Wyeth and Purchase, 2003]. However,
there has been little research presenting guidelines for developing AR
experiences for children.
In Radu’s paper they provide a list of elements that must considered
from a developmental perspective, in four broad areas: (1) Motor
Abilities, (2) Spatial Abilities, (3) Attention Abilities, and (4) Logic
and Memory Abilities. For example, many AR interfaces support tangible
interaction with two-handed input, but studies have shown that
188 Design Guidelines and Interface Patterns
bi-manual input abilities doesn’t reach adult levels until around 9 years
old [Hourcade, 2008, Fagard, 1990], and before age children may have
trouble using both hands for input.
Radu makes the comment in their paper that young children tend to
hold handheld AR input devices with both hands and it is not until they
are older than 7 that they can hold the device with one hand, freeing
up the other hand to interact with the AR content. They suggest that
AR games could be created for training motor coordination skills in
children, particularly for two-handed input. In their paper they also
explore other aspects of motor ability such as hand-eye coordination,
ﬁne motor skills, and gross motor skills.
8.4 Summary
In summary, in this section we have provided an overview of design
principles for AR experiences. In general designers should begin by
considering the three core elements of physical objects, virtual content
and the interaction metaphor than connects them. They can also use
well-known Design Patterns to assist with the interaction techniques,
and in some cases they may need to consider special needs of people
using the system. In the next section we review methods than can be
used to evaluate the AR experiences once they have been developed.
9
Evaluation of AR Systems
In the previous sections we have reviewed AR technology and tools
for building AR applications. However, a key activity in developing
AR experiences is evaluating the usability of the AR application. Gabbard
and Swan [2008] argue that user-based experiments are critical for
driving design activities, usability, and discovery early in an emerging
technology’s development, such as in the case of Augmented Reality.
They point out that lessons learned from user studies provide value to
the ﬁeld as a whole in terms of insight into the user interface design
space. In this section we describe factors that should be taken into
account when developing AR usability studies, and review the major
types of user studies that are typically conducted with AR systems.
Although Augmented Reality has been researched for nearly ﬁfty
years, it is only recently that signiﬁcant numbers of AR user studies
have begun to appear in the research literature. In 2005, Swan and
Gabbard [2005] presented a survey of all of the user-based experimentation
in AR they could ﬁnd in the research literature. They reviewed
papers from the IWAR, ISAR and ISMAR conferences, the International
Symposium on Wearable Computers, the IEEE Virtual Reality
conference and the journal Presence: Teleoperators and Virtual Envi-
189
190 Evaluation of AR Systems
Table 9.1: The results of Swan and Gabbard’s survey of AR publications. [Swan
and Gabbard, 2005]
ronments, from 1992 until 2004. From these four sources they identiﬁed
266 AR-related papers, of which only 38 addressed some aspect of Human
Computer Interaction, and only 21 described a formal user study.
This meant that less than 8% of all the AR related papers published
in these conferences and journal had any user-based experimentation.
Table 9.1 shows the summary of this breakdown.
More recently Dünser et al. [2008] conducted a broader survey of
AR research published between 1993 and 2007, looking at 28 outlets
such as the ACM Digital Library, IEEE Xplore Journals, SpringerLink,
and others. They were able to identify a total of 557 AR related
publications, and of these a total of 161 publications that included a
user evaluation of one type or another. Figure 9.1 shows the number of
AR papers collected and the papers in this collection that had a user
evaluation. As can be seen the proportion of AR papers with a user
evaluation is increasing over time. They reported that overall their survey
showed that an estimated 10% of the AR papers published in ACM
and IEEE included some user evaluation, agreeing closely with the ﬁgures
that Swan and Gabbard [2005] found from their smaller survey.
These results show that there is still room for signiﬁcant improvement
in the number of formal user evaluations conducted in AR research.
One reason for the lack of user evaluations in AR could be a lack of
education on how to evaluate AR experiences, how to properly design
experiments, choose the appropriate methods, apply empirical methods,
and analyse the results. There also seems to be a lack of understanding
of the need of doing studies or sometimes the incorrect
motivation for doing them. If user evaluations are conducted out of
incorrect motivation or if empirical methods are not properly applied,
9.1. Types of User Studies 191
Figure 9.1: Total number of AR papers published and papers containing a user
evaluation. [Dünser et al., 2008]
the reported results and ﬁndings are of limited value or can even be
misleading.
9.1 Types of User Studies
Swan and Gabbard [2005] report that the ﬁrst user-based experimentation
in AR was published in 1995, and since then usability studies
have been reported in three related areas:
• Perception: How do users perceive virtual information overlaid on
the real world? What perceptual cues can be used to distinguish
between real and virtual content?
• Performance/Interaction: How do users interact with virtual information
overlaid on the real world? How can real world objects
be used to interact with augmented content?
• Collaboration: How can MR / AR interfaces be used to enhance
face-to-face and remote collaboration?
Dünser et al. [2008] provided a breakdown of the papers they reviewed
into these three categories and found that the majority of the
evaluations conducted were Performance Studies with 75 examples, followed
by Perception and Cognition based user studies with 35 papers,
192 Evaluation of AR Systems
and ﬁnally Collaboration user studies with 10 examples. This shows
that user evaluation of collaborative AR systems is an area that needs
to be studied more.
In the remainder of this section we describe typical user evaluations
conducted in the areas of Perceptual Studies, Performance/Interface
Studies and Collaborative Studies. This should provide readers with a
good representative set of AR user studies that they base their own
user evaluations on. For a more exhaustive survey readers should refer
back to the work of Swan and Gabbard [2005] and Dünser et al. [2008].
9.1.1 Perceptual Studies
In an Augmented Reality interface the blending of Reality and Virtuality
is a perceptual task in which the interface designer tries to convince
the human perceptual system that virtual information is as real as the
surrounding physical environment. It is di cult, if not impossible to
control all the possible perceptual cues, so perceptual biases can occur
that a ect task performance. Drascic and Milgram [1996] provide an
overview of eighteen perceptual issues that relate to Augmented Reality,
including miss-matches in clarity and luminance between real and
virtual imagery, accommodation and vergence conﬂict, and occluding
virtual imagery by real objects. As they point out, in a task in which
di erent depth cues conﬂict, results may be wildly inaccurate or inconsistent.
Thus experiment/interface designers should be aware of the
cues they are introducing that may a ect user perception.
Many researchers have conducted studies to quantify some of the
perceptual issues associated with AR interfaces. These studies often
relate to the use of optical or video see-through head mounted displays
to merge real and virtual images (see [Rolland and Fuchs, 2000] for a
description of head-mounted display issues).
Since the 1960’s studies have been conducted on size and distance
judgments of virtual imagery presented in AR displays. Rolland and
Gibson summarize these results in [Rolland et al., 1995b] as well as
reporting new results for the perceived size and depth of virtual objects
when presented in a see-through display. Their experimental design
is typical of perception studies. A bench-mounted optical see-through
9.1. Types of User Studies 193
display was built and a careful calibration technique used to adjust
the display elements for each individual subject. The subjects were
then shown a pair of objects (a cube and a cylinder) side by side and
asked to judge which was closet. Three object conditions were used;
both objects were real, one object is real and the other is virtual, both
objects are virtual. Objects were shown at a variety of depths and they
report ﬁnding that that the virtual objects are perceived systematically
farther away than real objects.
A similar design is used by Ellis et al. in their distance judgment
studies [Ellis and Menges, 1997]. Their work explored how accurately
subjects could place a physical cursor under a virtual object in 3 di erent
conditions; monocular display, binocular display and stereoscopic
display. In each case a subject looked through a bench-mounted seethrough
display and moved a real pointer until it appeared until a virtual
tetrahedron displayed at di erent depths. The main experimental
measure was the actual distance between the real and virtual objects.
Depth judgment in a stereoscopic display was found to be almost perfect,
while the monocular display produced large overestimates in depth
position. Ellis and Menges report on a suite of related experiments in
[Ellis and Menges, 2001].
9.1.2 Interaction
Section 7 described a range of di erent interaction methods and interface
metaphors for Augmented Reality. However, before new interaction
metaphors can be , experiments must be conducted to understand
how e ective the techniques are. In general, AR interaction studies can
use established techniques from immersive VR experiments although a
particularly important area of study is the e ect of physical objects on
virtual object manipulation. For example, Lindeman ﬁnds that physical
constraints provided by a real object can signiﬁcantly improve performance
in an immersive virtual manipulation task [Lindeman et al.,
1999]. Similarly Ho man ﬁnds adding real objects that can be touched
to immersive Virtual Environments enhances the feeling of Presence
in those environments [Ho mann, 1998]. While in Poupyrev’s virtual
tablet work, the presence of a real tablet and pen enable users to easily
194 Evaluation of AR Systems
enter virtual handwritten commands and annotations [Poupyrev et al.,
1998].
Mason et al. [2001] provide a good example of an interaction experiment
designed for an AR interface. Their work explored the role
of visual and haptic feedback in reaching and grasping for objects in
a table-top AR environment. They were particularly interested to ﬁnd
out if Fitt’s law held in an AR setting. Fitt’s law is a basic interaction
law that relates movement time to index of di culty [Fitts and Peterson,
1964]. In the experiment subjects reached for and grasped a cube
in the presence or absence of visual feedback of seeing their own limb.
In half the conditions the cube was purely virtual, while in the other
half the virtual cube was superimposed over a real cube. Finally, four
di erent sizes of cubes were used.
An optical tracking system was used to measure hand motion and
a set of kinematic measures used, including Movement Time, Peak
Velocity of the Wrist, Time to Peak Velocity of the Wrist, and Percent
Time from Peak Velocity of the Wrist. Using these measures Mason et
al. found that Fitt’s Law was followed when a real cube was present,
but not when the target was entirely virtual. This result implies that
some form of haptic feedback is essential for e ective task performance
in augmented and virtual environments. This work also illustrates that
kinematic variables can be a powerful tool for interaction experiments.
Similar AR interaction and object positioning experiments have been
reported by Wang and MacKenzie [2000] and Drascic and Milgram
[1991].
An important di erence between AR and VR interfaces is that in
an AR interface physical object manipulations can be mapped oneto-one
to virtual object operations, and so follow a space-multiplexed
input design [Fitzmaurice and Buxton, 1997]. In general input devices
can be classiﬁed as either space- or time-multiplexed. With a spacemultiplexed
interface each function has a single physical device occupying
its own space. Conversely, in a time-multiplexed design a single
device controls di erent functions as di erent points in time. The mouse
in a WIMP interface is a good example of a time-multiplexed device.
Space-multiplexed devices are faster to use than time-multiplexed de-
9.1. Types of User Studies 195
vices because users do not have to make the extra step of mapping the
physical device input to one of several logical functions [Fitzmaurice
and Buxton, 1997]. In most manual tasks space-multiplexed devices
are used to interact with the surrounding physical environment. In
contrast, the limited number of tracking devices in an immersive VR
system makes it di cult to use a space-multiplexed interface.
The use of a space-multiplexed interface makes it possible to explore
interaction metaphors that are di cult in immersive Virtual Environments.
One promising area for new metaphors is Tangible Augmented
Reality. Tangible AR interfaces are AR interfaces based on Tangible
User Interface design principles. The Shared Space [Billinghurst et al.,
2000, Kato et al., 2000] interface (see Figure 9.2) is an early example
of a Tangible AR interface. In this case the goal of the interface was
to create a compelling AR experience that could be used by complete
novices. In this interface several people stand around a table wearing
HMDs. On the table are cards and when these are turned over, in their
HMDs the users see di erent 3D virtual objects appearing on top of
them. The users are free to pick up the cards and look at the models
from any viewpoint. The goal of the game was to collaboratively match
objects that logically belonged together. When cards containing correct
matches are placed side by side an animation is triggered. Tangible User
Interface design principles are followed in the use of physically based
interaction and a form-factor that matches the task requirements.
The SharedSpace interface was shown at the Siggraph 1999 conference
where there was little time for formal evaluation. However an
informal user study was conducted by observing user interactions, asking
people to ﬁll out a short post-experience survey, and conducting
a limited number of interviews. From these observations it was found
that users did not need to learn any complicated computer interface
or command set and they found it natural to pick up and manipulate
the physical cards to view the virtual objects from every angle. Players
would often spontaneously collaborate with strangers who had the
matching card they needed. They would pass cards between each other,
and collaboratively view objects and completed animations. By combining
a tangible object with virtual image we found that even young
196 Evaluation of AR Systems
Figure 9.2: The Shared Space interface. [Billinghurst et al., 2000]
children could play and enjoy the game. When users were asked to
comment on what they liked most about the exhibit, interactivity, how
fun it was, and ease of user were the most common responses. Users
felt that they could very easily play with the other people and interact
with the virtual objects. Perhaps more interestingly, when asked what
could be improved, people thought that reducing the tracking latency,
improving image quality and improving HMD quality were most important.
This feedback shows the usefulness of informal experimental
observation, particularly for new exploratory interfaces.
9.1.3 Collaboration
A particularly promising area for AR user studies is in the development
and evaluation of collaborative AR interfaces. The value of immersive
VR interfaces for supporting remote collaboration has been shown by
the DIVE [Carlsson and Hagsand, 1993] and GreenSpace [Mandeville
et al., 1996] projects among others. However, most current multiuser
VR systems are fully immersive, separating the user from the real
world and their traditional tools. While this may be appropriate for
some applications, there are many situations where a user requires collaboration
on a real world task. Other researchers have explored the
use of augmented reality to support face-to-face collaboration and re-
9.1. Types of User Studies 197
mote collaboration. Projects such as Studierstube [Schmalstieg et al.,
1996, Szalavári et al., 1998], Transvision [Rekimoto, 1996b], and AR2
Hockey [Ohshima et al., 1998] allow users to see each other as well
as 3D virtual objects in the space between them. Users can interact
with the real world at the same time as the virtual images, supporting
spatial cues and facilitating very natural collaboration. Although
these projects have successfully demonstrated collaborative AR interfaces
there have been few formal user studies conducted. In contrast
there have been many decades of studies into various aspects of audio
and video conferencing. We can draw on the lessons from these
experiments when evaluating collaborative AR interfaces.
In the telecommunications literature, there have been many experiments
conducted comparing face-to-face, audio and video, and audio
only communication. Sellen [1995] provides a good summary. While
people generally do not prefer audio only, they are often able to perform
tasks as e ectively as in the video conditions. Both the audio and
video, and audio only cases typically produce poorer communication
than face-to-face collaboration, so Sellen reports that the main e ect
on collaborative performance is due to whether the collaboration was
technologically mediated or not, not on the type of technology mediation
used. Naturally this varies somewhat according to task. While
face-to-face interaction is no better than speech only communication
for cognitive problem solving tasks [Williams, 1977], visual cues can be
important in tasks requiring negotiation [Chapanis, 1975].
Although the outcome may be the same, the process of communication
can be a ected by the presence or absence of visual cues [O’Malley
et al., 1996] because video can transmits social cues and a ective information,
although not as e ectively as face-to-face interaction [Heath
and Lu , 1991]. However, the usefulness of video for transmitting nonverbal
cues may be overestimated and video may be better used to show
the communication availability of others or views of shared workspaces
[Whittaker and O’Conaill, 1997]. So even when users attempt nonverbal
communication in a video conferencing environment, their gestures
must be wildly exaggerated to be recognized as the equivalent
face-to-face gestures [Heath and Lu , 1991].
198 Evaluation of AR Systems
These results imply that in collaborative AR experiments process
measures and subjective measures may be more important than quantitative
outcome measures. Process measures are typically gathered by
transcribing the speech and gesture interaction between the subjects
and performing a conversational analysis. Measures that are often collected
include the number of words spoken, average number of works
per phrase, number and type of gestures, number of interruptions, number
of questions and the total speaking time. Although time consuming,
this type of ﬁne-grained analysis often reveals di erences in communication
patterns between experimental conditions.
One of the di culties with collecting process measures is that of
deciding which metrics to use in developing a data coding technique.
Transcribing audio and video tapes is a very time-consuming process
and can be unfruitful if the wrong metrics are used. Nyerges et al. [1998]
provide a good introduction to the art of coding groupware interactions
and give guidance on good metrics. Measures that have been found to
be signiﬁcantly di erent include:
• Frequency of conversational turns [Daly-Jones et al., 1998,
O’Conaill and Whittaker, 1997]
• Conversational Handovers [O’Conaill and Whittaker, 1997]
• Incidence/duration of overlapping speech [Daly-Jones et al., 1998,
Sellen, 1995]
• Use of pronouns [McCarthy and Monk, 1994]
• Number of interruptions [Boyle et al., 1994, O’Conaill and Whittaker,
1997]
• Turn Completions [Tang and Isaacs, 1993]
• Dialogue length [Boyle et al., 1994, O’Conaill and Whittaker,
1997, O’Malley et al., 1996, Anderson et al., 1996]
9.1. Types of User Studies 199
• Dialogue structure [Boyle et al., 1994, O’Malley et al., 1996, Anderson
et al., 1996]
• Backchannels [O’Conaill and Whittaker, 1997]
Gesture and non-verbal behaviors can also be analyzed for characteristic
features. Generally these behaviors are ﬁrst classiﬁed according
to type and the occurrences of each type and then counted. Bekker
et al. [1995] describe an observational study they performed on groups
of subjects engaged in a face-to-face design task. From video of the
subject groups four categories of gesture, kinetic, spatial, pointing and
other, were identiﬁed. They were then able to calculate the average
numbers of gestures per minute for each of the di erent stages in the
design task. These four categories were based on the more complex coding
categories used by McNeill [1992] and Ekman and Friesen [1981].
In contrast to this work, there have been very few user studies
with collaborative AR environments and almost none that examined
communication process measures. Kiyokawa et al. [2000] conducted an
experiment to compare gaze and gesture awareness when the same task
was performed in an AR interface and an immersive virtual environment.
In his SeamlessDesign interface, users were seated across a table
from one another and use a collaborative AR design application (see
Figure 7.2). A simple shared pointing task was used to compare gaze
and gesture awareness and the inﬂuence of a virtual body and gaze
directed viewing lines. The experimental measures were the time to
perform the task and a subjective survey on the ease of use. Subjects
performed signiﬁcantly faster in the AR interface than in the immersive
condition and felt that this was the easiest condition to work together.
There have been several studies performed using wearable computers
and displays for supporting remote collaboration. In this case the
remote users can typically manipulate a virtual pointer in the users
wearable display or share their view of the real world. An early example
was the SharedView system of Kuzuoka [1992]. This was a video
see-through head mounted display with a camera attached. A machine
operator would wear the HMD enabling a remote expert to see what
he was seeing and make gestures in the display to show him how to
200 Evaluation of AR Systems
operate the machinery. In a simple evaluation study, SharedView was
found to have better performance than collaboration with a remote
ﬁxed camera, but worse than the face-to-face collaboration.
Kraut et al. [1996] provides another example of collaboration using
a wearable interface. They were interested in how the presence or
absence of a remote expert might help a subject repair a bicycle and
what di erences in communication patterns may result with and without
shared video. Subjects wore a head mounted display that allowed
them to see video of the remote expert, or images of a repair manual.
Subjects could complete the repairs in half the time with a remote expert
and produced signiﬁcantly higher-quality work. When video was
used they found that the experts were more proactive with help and
that subjects did not need to be as explicit in describing their tasks. In
a follow-up experiment, Fussell et al. [2000] add a condition where the
expert is in the same room as the subject. The same metrics are used
(performance time and quality and conversational analysis of speech),
and they ﬁnd that the task is completed signiﬁcantly faster in the faceto-face
condition. This time they ﬁnd that speech patterns were significantly
di erent between face-to-face and mediated conditions; experts
in the face-to-face condition used signiﬁcantly more deictic references,
used shorter phrases, and were more e cient in their utterances.
Several collaborative experiments and the measures used are summarized
in Table 9.2.
The MR conferencing experiment [Billinghurst and Kato, 2000],
provides an example of how to conduct a collaborative AR experiment
with conversational analysis. The MR conferencing interface supports
conferencing between a desktop user and a person wearing a lightweight
head mounted display. The person in the HMD sees their remote collaborator
as a live video texture superimposed over a real world object
(a name card) (see Figure 9.3). This conﬁguration has a number of
possible advantages over normal video conferencing, so the goal of this
experiment was to compare MR conferencing to normal video and audio
conferencing.
Each pair of subjects talked with each other for 10 minutes in each
of audio only, video and MR conferencing conditions. Each of these
9.1. Types of User Studies 201
Table 9.2: Collaborative AR experiments.
Interface Task Conditions Measures
Used
Outcome
SeamlessDesign
(Kiyokawa 00)
Pointing,
Co-located
collabora-
tion
AR vs. VR Performance
time, Subjective
ease
of use survey
AR faster. AR
rated as easier to
use.
AR2 Hockey
(Ohshima 98)
Collaborative
game, Face-
to-face
AR vs. VR Subjective
survey,
Game scores
Mixed
SharedView
(Kuzuoka 92)
Machine
operation,
Remote
viewing,
Remote
pointing
AR vs. FtF
vs. Fixed
Camera
Performance
time, Speech
classiﬁcation
FtF faster than
SharedView
which is faster
than ﬁxed
Camera. Fixed
camera and
Shared View
have similar
speech patterns.
NetMan
(Bauer 99)
Remote
pointing,
Remote col-
laboration
AR pointing
vs. no
pointing
Gesture
count,
Speech clas-
siﬁcation,
Subjective
survey
Gestures used
more than
speech. Deictic
speech most
common type of
speech act.
Bike Repair I
(Kraut 96)
Bike repair,
Remote col-
laboration
Single user
vs. remote
expert,
Video vs.
no-video
Performance
time, Per-
formance
quality,
Coding of
speech acts
Subjects faster
and produce
better quality
work with a
remote expert.
With video
experts were
more proactive
and subjects did
not need to be
as explicit in
describing the
problem.
Bike Repair II
(Fussell 00)
Bike repair,
Remote col-
laboration
Co-located
expert vs.
Audiovideo
vs.
Audio only
Performance
time, Per-
formance
quality,
Conver-
sational
analysis
Performance
and quality best
in FtF condition.
Signiﬁcant
di erences in
conversational
coding.
202 Evaluation of AR Systems
Figure 9.3: The MR conferencing collaborative AR interface. [Billinghurst and
Kato, 2000]
sessions were video taped and after each condition subjects ﬁlled in a
survey about how present they felt the remote person was and how
easily they could communicate with them. After the experiment was
over the video taped were transcribed and a simple speech analysis
performed, including counting the number of works/minute each of the
users uttered, the number of interruptions and the back-channels spoken.
This analysis revealed that not only did the user feel that the
remote collaborators were more present in the MR conferencing condition,
but that they used less words and interruptions per minute than
in the two other conditions. These results imply that MR conferencing
is indeed more similar to face to face conversation than Audio or Video
conferencing.
An alternative to running a full collaborative experiment is to simulate
the experience. This may be particularly for early pilot studies for
multi-user experiments where it may be di cult to gather the number
of subjects. The WearCom project [Billinghurst et al., 1998] is an example
of a pilot study that uses simulation to evaluate the interface. The
WearCom interface is a wearable communication space that uses spatial
audio and visual cues to help disambiguate between multiple speakers.
To evaluate the interface a simulated conferencing space was created
where 1,3 or 5 recorded voices could be played back and spatialized in
real time. The voices were played at the same time and said almost
the same thing, expect for a key phrase in the middle. This simulates
the most di cult case for understanding in a multi-party conferencing
9.2. Evaluation Methods 203
experience. The goal of the subject was to listen for a speciﬁc speaker
and key phrase and record the phrase. This was repeated for 1,3 and
5 speakers with both spatial and non-spatial audio, so each user generated
a score out of 6 possible correct phrases. In addition, subjects
were asked to rank each of the conditions on how understandable they
were.
When the results were analyzed from this experiment, users scored
signiﬁcantly better results on the spatial audio conditions than with
the non-spatial audio. They also subjectively felt that the spatial audio
conditions were far more understandable. Although these results were
found using a simulated conferencing space, they were so signiﬁcant
that it is expected that the same results would occur in a conferencing
space with real collaborators.
9.2 Evaluation Methods
In their survey Dünser et al. [2008] also categorizes AR papers containing
a user evaluation according to the type of user study method and
approaches. They identiﬁed the following ﬁve main types of evaluation
techniques used:
1) Objective measurements: measures such as task completion times
and accuracy / error rates, scores, positions, movements, number
of actions, etc.
2) Subjective measurements: measures such as user questionnaires,
subjective user ratings, or judgements.
3) Qualitative analysis: measures such as formal user observations,
formal interviews, or classiﬁcation or coding of user behaviour
(e.g. speech or gesture coding).
4) Usability evaluation techniques: evaluation techniques that are often
used in interface usability evaluations such as heuristic evaluation,
expert based evaluation, task analysis, think aloud methods,
or Wizard of OZ methods.
204 Evaluation of AR Systems
Figure 9.4: Classiﬁcation of AR publications by evaluation method. [Dünser et al.,
2008]
5) Informal evaluations: evaluations such as informal user observations
or informal collection of user feedback.
Figure 9.4 shows a breakdown of the AR papers reviewed by Dünser
et al. [2008] according to the main type of usability method used in
the paper. As can be seen the majority of the papers reviewed used
objective and subjective measures and very few of them used usability
evaluation techniques.
In their work they classify AR papers according to the main type
of user evaluation performed. However is it common for researchers
to combine several evaluation methods together to explore di erent
aspects of the user experience. Gabbard et al. [1999] provides a design
methodology for combining together several di erent evaluation
techniques in Virtual Environment system design. Their model iteratively
moves through the stages of (1) User Task Analysis, (2) Expert
Guidelines-Based Evaluation, (3) Formative User-Centered Evaluation,
and (4) Summative Comparative Evaluation (see Figure 9.5). Outputs
at each of these stages help progress the system design to a usable
prototype.
Hix et al. [2004] demonstrate how this same methodology can be
applied to the design of AR systems. In this case they use the example
of designing the Battleﬁeld Augmented Reality System (BARS), an
outdoor AR interface for information presentation and navigation for
9.2. Evaluation Methods 205
Figure 9.5: Virtual Environment system design methodology. [Gabbard et al., 1999]
soldiers. In this case the design of the system was begun with a user
needs and domain analysis to understand the requirements of soldiers in
the ﬁeld. This was followed by an expert evaluation, and user-centered
formative and summative evaluations. In all there were six evaluation
cycles conducted over a two month period with nearly 100 mockup prototypes
created. Figure 9.6 shows one of these prototypes and the AR
view provided by the system to assist the user in outdoor navigation.
Hix et al. [2004] reported that applying Gabbard’s methodology and
progressing from expert evaluation to user-based summative evaluation
was an e cient and cost-e ective strategy for assessing and improving
a user interface design. In particular the expert evaluations of BARS
before conducting any user evaluations enabled the identiﬁcation of any
206 Evaluation of AR Systems
Figure 9.6: The BARS outdoor AR system. [Hix et al., 2004]
obvious usability problems or missing functionality early in the BARS
development life cycle. This allowed improvements to be made to the
user interface prior to performing user-based statistical and formative
evaluations.
9.3 Summary
In this section we have provided an overview of techniques for AR
evaluation, and a review of typical methods that have been used in
the past. We began by summarizing the meta-reviews conducted by
Swan and Gabbard [2005] and Dünser et al. [2008], showing that in the
AR papers that they collected only a small percentage (around 8-10%)
contained any formal evaluation. This demonstrated the need for more
user studies and evaluations in AR research and development going
forward. The report identiﬁed three main areas for types of user studies;
(1) Perception, (2) Performance/Interaction and (3) Collaboration. We
then reviewed example AR systems and user evaluations in each of
these areas. Finally we discussed an AR system design methodology
that combines di erent types of evaluation techniques, and show how
this was applied in the design of the BARS system. In the next section
we will review areas for ongoing research in Augmented Reality.
10
AR Applications Today
In section 3 we presented a short history of Augmented Reality and
discussed how the rise of mobile devices and advances in web technologies
means that AR experiences are available to more people than ever
before. In this section we give some examples of typical modern day
AR applications and identify some of the features of successful products.
AR technology can be used in many di erent domains, but we
focus on the areas of Marketing, Medicine, Education, Entertainment,
and Architecture, discussing applications in each of these areas in turn.
10.1 Education
When new technologies are developed, attempts are often made to try
and use them in an educational setting. Augmented Reality is no exception
and for over ten years AR technology has been tested in a
number of di erent educational applications. These trials have shown
that in some situations AR can help students learn more e ectively and
have increased knowledge retention relative to traditional 2D desktop
interfaces.
207
208 AR Applications Today
Figure 10.1: Children playing with an AR enhanced book together. [Dünser and
Hornecker, 2007]
This is particularly the case for reading and book centric learning,
where AR can be used to overlay interactive 3D digital content on
the real book pages. In §11.2 we describe the MagicBook application
[Billinghurst et al., 2001] and how AR can be used to transition from
reading a real book to exploring an immersive virtual reality space.
Since the development of the ﬁrst MagicBook prototype a wide range
of other AR enhanced books have been developed by academics and
companies, some of which have been commercially available since 2008.
The use of AR enhanced books in an educational setting has been
found to improve story recall and increase reading comprehension. One
study paired children together and observed them while the interacted
with an AR book using physical paddles to manipulate the virtual content
(see Figure 10.1) [Dünser and Hornecker, 2007]. They found that
children were able to easily interact with the books and AR content.
The use of physical objects to interact with the AR objects and characters
was very natural. However some children were confused by the
mirrored video view shown on the screen, and also had a tendency
to interact with the virtual content in the same way as they would
with real objects, which didn’t always work. Nevertheless, this research
shows that AR books could be introduced into an educational setting
relatively easily.
A second study explored how AR enhanced books could help enhance
story recall [Dünser, 2008]. In this case two groups of children
10.1. Education 209
read either traditional books, or books with animated AR scenes as
well as the printed text. The children were divided into those that had
a high reading ability and those with low reading ability (as determined
by their teacher). From the text-based story conditions, those in the
high-ability group could retell signiﬁcantly more key story points than
the low-ability group. However, the two groups showed no signiﬁcant
di erence in recalling key story points from the interactive AR condition.
The researcher concludes that AR books could beneﬁt students
who are less able to comprehend text-based learning materials, and so
could be very helpful in learning environments.
Another example of the power of Augmented Reality for education
is the colAR mix application1 developed by the company, Puteko. This
is an AR colouring experience available on iOS and Android mobile
devices. Using the application, users can point their mobile device to
a colouring book page and see an animated 3D virtual scene come to
life. However unlike other AR applications, with colAR mix users can
colour the pages and see the virtual objects textured with their own
colours (see Figure 10.2). Any colour that they add to the colouring
page will be textured mapped onto the 3D model and seen in the AR
scene. Some of the models also have simple interactivity. For example,
when the user views the animated dragon model, they can touch a
virtual button on the screen to cause it to breath ﬁre. Similarly, in the
scene with a dancing girl they can turn on the virtual radio to see her
dancing to the music.
The application works by using the Vuforia library to track the
black lines on the book page and calculate the phone or tablet camera
position from markerless image tracking. Once the camera position has
been determined then the pixels from the live video that are inside
the di erent parts of the coloured page can be found. The colours of
these pixels are then used to create a texture map that is applied to
the animated AR 3D model. The end result is that the AR model
appears coloured with the same colours that the user has applied to
the colouring book page. A more detailed technical explanation of how
the system works can be found in [Clark and Dünser, 2012].
1
http://colarapp.com
210 AR Applications Today
(a) Uncoloured page (b) Coloured AR model
Figure 10.2: ColAR Mix AR colouring book application. http://colarapp.com
From a technical perspective the colAR mix application is very
simple, however the ability for users to see their own colouring appear
in the AR scene creates a very compelling experience. Since it’s release
in the middle of 2013 the application has become very popular with
more than a million downloads across the Android Play store and iOS
App store.
Teachers have been using the application in a number of interesting
ways in the classroom. For example, one teacher has been using the
technology to inspire creative writing2. He showed the children the
uncoloured page and then asked them to write a short story about it.
Then he had them colour the page and used colAR mix to bring it to
life. Once students have had a chance to use the AR application and
view the content, he asked them to write a story describing what they
could see. He found that after trying the AR application, the quality of
the writing improved, and children used more precise language in their
descriptions. For some of the more reluctant writers, it was especially
useful, giving them a lot more ideas to work with. Figure 10.3 shows an
2
http://mrparkinsonict.blogspot.com.au/2014/04/
can-augmented-reality-improve-writing.html
10.2. Architecture 211
(a) Coloured page (b) Before trying AR (c) After trying AR
Figure 10.3: Using ColAR mix to teach creative writing. http://mrparkinsonict.
blogspot.com.au/2014/04/can-augmented-reality-improve-writing.html
example of a page coloured in by one of the children in his class, and
the writing sample before and after using colAR mix. As can be seen,
the student writes over twice as much after using the AR application.
As can be seen, these examples show the potential educational beneﬁt
of AR technology. Even simple AR scenes can be used to motivate
children in a classroom. Results show that AR’s high level of interactivity
enhances learning, particularly for students who learn through
kinesthetic, visual, and other non-text-based methods. After trying the
technology one teacher said that there is "no question that AR will
prove to be a highly e ective medium both for entertainment and education."
[Billinghurst and Dünser, 2012].
10.2 Architecture
Augmented Reality is an ideal technology for showing virtual information
superimposed over the real world, and as such it can be used to
solve one of the most important problems in architecture; how can you
see a building that hasn’t been built yet? Traditionally architects used
a range of di erent tools to show their clients what their buildings will
be like, including 2D plans, physical models, 3D renderings, interactive
walkthroughs and animated ﬂy throughs. However there are a num-
212 AR Applications Today
ber of shortcomings with these methods. For example, 2D plans can
show in great detail the layout of a building, but sometimes the client
has trouble imagining what the building would look like from the 2D
drawing. Physical models, 3D renderings and interactive ﬂy-throughs
help the client understand what the building looks like, but they are a
reduced scale version of the ﬁnal space. These may not convey a clear
understanding of what the building will look like full-size on the ﬁnal
location.
Mobile AR can be used to view full-sized 3D virtual models of
future buildings on the local where they will be built. This allows those
interested in the building to clearly understand what it will look like.
The application could also be used as a planning tool, placing several
versions of the same building on site and allowing the client to provide
feedback about the di erent design options.
There are many examples of AR applications that can provide this
functionality, but in this section we describe two in more detail. These
applications were developed by the HIT lab NZ in response to a terrible
natural disaster. In September 2010 and again in February 2011,
the city of Christchurch, New Zealand was hit by several very strong
earthquakes, magnitude 7.1 and 6.9. The earthquakes caused large scale
damage with the result that over 70% of the buildings in the inner city
needed to be demolished in the rebuild process. The city council had a
problem how to communicate to the residents what the city would look
like once the rebuild was ﬁnished. There was also considerable interest
in communicating to visitors what the city used to look like before the
earthquakes destroyed most of the historical buildings.
In response to this, the HIT Lab NZ developed two mobile AR
applications CCDU AR and CityViewAR [Lee et al., 2012], both of
which were made available on the Android Play store and iOS app
store. CCDU AR was an application designed to show architectural
information superimposed over the printed pages of a book the city
council had produced explaining the city rebuild and showing concept
images of what the future landmark buildings would look like. The
application was developed for the iOS and Android mobile devices and
used the Vuforia library for image based tracking of the real book pages.
10.2. Architecture 213
The Unity game engine was used to show the 3D virtual building models
and add simple interactivity to the application.
Figure 10.4 shows the application being used. The user can take
their tablet or smart phone and when they run the application and
point the camera at the printed map of the city they will see 3D virtual
labels appearing showing them the key building redevelopments in the
city (Figure 10.4(b)). The labels appear ﬁxed in space and turn to face
the user as they move around the map, so that they are always readable.
The user can then tap one of the labels to see a larger 3D model of the
building appearing ﬁxed to the printed map (Figure 10.4(c)). Some of
the buildings also had simple interactivity added to them. For example
if the user moved their device closer to the virtual stadium model they
would hear the sound of the crowd growing louder, or they can tap on
the police station model to see virtual police cars exit the building and
hear siren sounds.
(a) Using the application
(b) Label view (c) Building view
Figure 10.4: CCDU AR mobile AR application.
214 AR Applications Today
The application was used by the Christchurch City Council at a
number of public events, and was also made freely available so that
any interested people could try it out. Overall the feedback was very
positive and users felt that they could more clearly understand the key
redevelopments that were going to take place, and also how the ﬁnal
buildings could look.
The CCDU AR application enabled people to better understand
what key buildings in the rebuilt city would look like, but it had the
limitation that users weren’t able to see the buildings on-site. The
CityViewAR application addressed this limitation and allowed users
to see full-sized virtual buildings on the real site. This was achieved
using the HIT Lab NZ’s Outdoor AR framework [Lee and Billinghurst,
2013], a software library developed speciﬁcally for creating outdoor AR
applications for mobile devices. Unlike the CCDU AR application, it
was designed to using GPS and compass sensor input and so enabled
outdoor tracking.
The Outdoor AR framework consists of two layers of components
(see Figure 10.5). The lower layer provides abstraction of functional
modules that are essential for building outdoor AR applications. These
components include basic data structures, tracking sensors, network
communication, 3D graphics and sound rendering, and tools for managing
UI elements, ﬁle parsing and data loading. The framework includes
a custom 3D graphics engine based on a scenegraph data structure
designed and optimized for mobile rendering of geo-located 3D
scenes. The Scene and Tracking data manager components play the role
of models, and the Map, AR, and List View components are higherlevel
abstractions of the AR scene visualization function. These model
and view components are ready-to-use components and developers can
simply choose which components they want to use. Though developers
are free to modify these components, the controller components
are where most customization typically happens, mixing and matching
which model and view components that the application is going to use.
Using the Outdoor AR framework, the CityViewAR application was
developed to help people explore destroyed historical sites and buildings
after the major earthquakes in Christchurch. Using the applica-
10.2. Architecture 215
Figure 10.5: Outdoor AR Library architecture. [Lee and Billinghurst, 2013]
tions people can walk on site and see information about the buildings
that used to be around them. The geo-located content is provided in
a number of formats including 2D map views, AR visualization of 3D
models of buildings on-site, immersive panorama photographs, and list
views of all the content available. Figure 10.6(a) shows the map view
with points of interest icons shown on it. Users can select icons they
are interested in and switch to a panorama view (Figure 10.6(b)) or
AR view (Figure 10.6(c)). The panorama view allows users to view immersive
360 photospheres take at their location several weeks after the
earthquake, showing the destruction at the time. The users can rotate
their mobile device around to see di erent portions of the panorama,
using the compass in the device to set the viewing orientation. The AR
view shows virtual images superimposed over a live video background,
transforming the empty lots of Christchurch into the buildings that
existed before the earthquake.
A user study was conducted to see if the AR component enhanced
the user experience (AR condition), compared to using the application
without AR viewing (non AR condition) [Lee et al., 2012]. When the
AR feature was available users used it over half of the time that they
were using the application, far more than the map or panorama views,
and they judged their overall experience to be better than in the nonAR
condition. Participants were also asked which features they liked
216 AR Applications Today
(a) Map view (b) Panorama view
(c) AR view
Figure 10.6: Using the CityViewAR application to see a historical building on-site.
[Lee et al., 2012]
the most, and in the AR condition 42% of the users picked the AR
view, while in the non AR condition 62% of people picked the panorama
view. People with the AR condition also moved around the street more,
trying to explore the di erent elements of the 3D buildings they were
able to view. Overall users enjoyed having a rich interface that used
several di erent ways to show the building information, but they felt
that the AR element deﬁnitely enhanced the user experience.
These applications show that even with current technology, AR experiences
can be developed that help solve the key problem of client
communication for architects. Wearable devices will even accelerate the
use of AR experiences in architectural applications in outdoor environment
(see Figure 5.10 showing CityViewAR running on Google Glass).
CityViewAR and CCDU AR are just two of dozens of mobile AR appli-
10.3. Marketing 217
cations available that enable users to view architectural content ways
that are not possible with more traditional tools.
10.3 Marketing
One of the greatest opportunities for AR is in the use of the technology
for product marketing. Marketing material is typically designed to
capture the attention of a person and provide them with motivation
to learn more about the product. AR technology can be used to create
experiences that are more memorable than other more traditional
forms of marketing.
AR can be also be used to develop experiences that are impossible
with other technology. For example, in section 3, we already mentioned
the Living Sasquatch marketing campaign developed created by Bo swana
to market Jack Links beef jerky. This used a simple web interface
(see Figure 10.7(a)) to allow users to create their own animated AR
scene with a Sasquatch monster. This was developed through using
a FLARToolKit based Flash application installed in a web page that
didn’t require any complex software installation by the end user. The
application allowed people to arrange sasquatch animations simply by
dragging and dropping icons on the screen. Once the animation was
completed they could view the output in the AR window in the centre
of the screen.
The public response was very strong with hundreds of people creating
images and short videos of their interaction with the sasquatch and
posting them online. People discovered that the size of the sasquatch
depended on the size of the AR tracking marker, and so were able to
create a number of humorous situations3 (see Figure 10.7(b)). In this
way the technology enables users to have a new experience, but also
for the end users to provide a word of mouth campaign for the brand
being promoted.
A second way that AR can be used in marketing is to enhance
traditional printed media advertising. Many companies have developed
AR experiences that allow people to point their phones at magazine
3
See for example, https://www.youtube.com/watch?v=eA1X0vfJGs0
218 AR Applications Today
(a) Web interface (b) User created image
Figure 10.7: Living Sasquatch interface and user experience.
advertisements or posters and see virtual 3D content or 2D movies
appear. For example, in 2012 company Explore Engage developed an
AR marketing experience for the Transformers 3 movie that allowed
people to point their phone at the movie poster and see 3D content from
the movie pop out4. This turned the static poster into an interactive
experience that showed a wide range of di erent clips and content from
the movie. It also allowed users to unlock an immersive ﬁrst-person
AR game, playing as Optimus Prime defending Earth against the evil
forces of the Decepticons. The marketing campaign was a big success
with over 400,000 app downloads in the ﬁrst few weeks of the movie
opening.
From these examples we can see that Augmented Reality is a technology
that has proven itself in several di erent application areas. However
the technology is just being applied in commercial settings and it
is obvious that even more real world applications will emerge in the
coming years. In the next section we discuss research areas that are being
explored that will increase the number of application areas where
the technology could be used.
4
http://www.tf3ar.com
11
Research Directions
Augmented Reality have been researched for nearly 50 years and is
beginning to become more mainstream through the commercialization
e orts of many companies. Despite this there are still a number of
research areas that need to be explored further before the technology
reaches its full potential. In Azuma et al. [2001] review paper they identify
the major obstacles limiting the wider use of AR as falling into three
themes: (1) technological limitations, (2) user interface limitations, and
(3) social acceptance issues. Martínez et al. [2014] used the Technology
Acceptance Model [Davis et al., 1989] and Roger’s innovation di usion
theory [Rogers, 2010] to identify the following bottlenecks to its
widespread adoption; (1) No AR standards, (2) Limited computational
power in AR devices, (3) Tracking Inaccuracy, (4) Social acceptance,
(5) Information overload.
In 2008, Zhou et al. [2008] reviewed ten years of ISMAR papers
and identiﬁed the main research trends over that time as being in the
areas of Tracking, User Interaction and Display Technology. So it is
evident that research has been conducted in most of the areas identiﬁed
by Azuma and Martinez, although not in Social Acceptance. In this
section we review ongoing research and future opportunities in the
219
220 Research Directions
four areas of (1) Tracking, (2) Interaction, (3) Displays, and (4) Social
Acceptance. In addition to these topics, Zhou et al. [2008] identiﬁed
other AR research as being conducted on topics such as rendering and
visualization methods, evaluation techniques, authoring and content
creation tools, applications, and other areas. However space limitations
prevent a deeper review of these areas.
11.1 Tracking
Tracking is one of the most popular areas for AR research because
it is one of the most important of the AR enabling technologies. As
we have seen from section 4, AR tracking systems have evolved from
simple ﬁducial marker based systems, to natural feature tracking, and
hybrid sensor based methods. However there is still signiﬁcant work
that needs to be done before the vision of "anywhere augmentation"
[Höllerer et al., 2007] is achieved, where users will be able to have a
compelling AR experience in any environment.
In section 4 we showed how that there are a number of reliable systems
for images based tracking from printed images, or outdoor tracking
using GPS and other sensors. However there are still a number of
important research problems that need to be solved, including (1) Wide
area tracking, (2) Reliable indoor tracking, and (3) Indoor/Outdoor
ubiquitous tracking, among others.
For reliable outdoor Augmented Reality, tracking methods are
needed that can provide accurate location over a wide area. GPS hardware
can be used for position sensing, but consumer grade GPS in
mobile devices typically provides an average accuracy of 5 – 10 meters
depending on satellite signal strength. One alternative method is
to use computer vision techniques to estimate the camera pose relative
to known visual features, often in conjunction with other GPS
and inertial sensors [Reitmayr and Drummond, 2006]. However this
approach can be di cult to scale to wide area tracking because of the
di culty of creating a large scale feature set [Castle et al., 2008] or
capturing thousands of pictures for image matching [Arth et al., 2009].
In contrast Ventura and Hollerer [2012] have developed a system that
11.1. Tracking 221
Figure 11.1: Wide area AR tracking using panorama imagery (top) processed into
a point cloud dataset (middle) and used for AR localisation (bottom). [Ventura and
Hollerer, 2012].
combines panoramas captured from several positions to create a point
cloud model which can then be used for feature based localization using
a remote server, and real time tracking on a mobile device. In this
way they were able to track over of 1000 m2 with a positional accuracy
of less than 25cm error, and rotational accuracy of under 0.5 degrees.
Figure 11.1 shows the data captured by the system and the ﬁnal tracking
results. This approach looks promising for wide area tracking in an
unprepared outdoor environment, although more work will need to be
done to verify this.
In favourable conditions, GPS technology can be used to provide
accurate outdoor tracking however lack of satellite coverage means
that GPS systems fail indoors and other methods must be used. A
range of di erent methods have been explored for providing robust
222 Research Directions
indoor tracking. For example, Dissanayake et al. [2001] showed how
ultra-sound could be used for position localisation inside buildings, and
demonstrated several applications that this could enable, such as navigation,
asset localisation, and creating virtual interactive buttons in
space. However this system required the installation of a set of ceiling
mounted ultra-sonic receivers and so become more expensive the
larger the space tracked over, the tracking accuracy is also limited
and not precise enough for exact overlay of virtual content on real objects.
Other methods for precise indoor tracking use a camera combined
with a 2D laser tracker for range ﬁnding [Scheer and Müller, 2012],
computer vision tracking from markers pasted over the walls [Wagner,
2002, Schmalstieg and Wagner, 2007], and hybrid systems with computer
vision, inertial sensors and ultra-wide-band tracking [Newman
et al., 2006], among others.
One particularly promising direction for research is the use of handheld
depth sensors for indoor tracking. The Google Tango project1 is a
handheld device that uses customized depth sensors with motion tracking
cameras and inertial sensors to update the position and orientation
of the device in real time, while at the same time creating a map of the
users environment. This has the advantage that it is completely self
contained and doesn’t require any image markers or electronic sensors
to be placed in the tracked environment. Figure 5.13 shows a typical
depth image of the real world captured by Tango. Due to its precise
indoor positioning, Tango provides an ideal platform for indoor AR.
Other technologies, such as Intel’s RealSense2, will also provide depth
sensing on handheld tablets, and so there are opportunities for further
research in this area.
One of the existing challenges is seamless switching from outdoor to
indoor tracking environments. This is necessary for the development of
ubiquitous tracking systems that work continuously as the user walks in
and out of buildings, or other locations. Piekarski et al. [2003] developed
an early prototype that combined GPS outdoor tracking (accurate to
1
https://www.google.com/atap/projecttango
2
http://www.intel.com/content/www/us/en/architecture-and-technology/
realsense-overview.html
11.2. Interaction 223
50cm) with indoor marker based tracking (accurate to 20 cm), and automatically
switching between the two based on GPS reception. More
recently, Behzadan et al. [2008] developed a system that combined indoor
tracking using Wireless Local Area Networks with GPS tracking
for outdoor position sensing, while Akula et al. [2011] combined GPS
with an inertial tracking system to provide continuous location sensing.
Huber et al. [2007] describe a system architecture for ubiquitous
tracking environments. There needs to be more research conducted in
this area before the goal of ubiquitous tracking will be achieved.
11.2 Interaction
Section 7 provided a review of di erent interaction methods for Augmented
Reality. As was shown, early AR interfaces used input techniques
inspired from desktop interfaces or Virtual Reality, but over
time more innovative methods have been used, such as Tangible AR,
or natural gesture interaction. There is still signiﬁcant opportunity for
on-going research in new interaction methods, especially in the areas of
(1) Intelligent systems, (2) Hybrid user interfaces, and (5) Collaborative
systems, among others. In this section we provide a brief overview
of some opportunities in each of those areas.
Previous research has shown the AR is a very natural way to interact
with virtual content, but in many cases the interfaces themselves
have not been very intelligent with simple interactions that don’t respond
in di erent ways to user input. Over the last twenty years the
ﬁeld of Intelligent User Interface (IU) has emerged [Sullivan, 1991]
which explores how Artiﬁcial Intelligence can be combined with Human
Computer Interaction methods to produce more responsive interfaces.
Unfortunately there has been little research in AR on how to include
IUI methods. Some researcher have begun exploring the user of virtual
characters that display limited intelligence, for example the Welbo interface
was a character seen in the real world that responded to simple
speech commands [Anabuki et al., 2000], and Mr Virtuoso was a handheld
AR interface that used an AR virtual character to teach about art
[Wagner et al., 2006].
224 Research Directions
One topic that has a lot of potential in this area is Intelligent Training
Systems (ITS). Earlier research has shown that both AR and ITS
applications can signiﬁcantly improve training. For example AR technology
allows virtual cues to be overlaid on the workers equipment
and help with performance on spatial tasks. In one case an AR-based
training system improved performance on a vehicle maintenance tasks
by 50% [Henderson and Feiner, 2009]. Similarly, ITS applications allow
people to have a educational experience tailored to their own learning
style, providing intelligent, responsive feedback, and have been shown
to improve learning results of one letter grade or higher, produce signiﬁcantly
faster learning and impressive results in learning transfer [Mitrovic,
2012, VanLehn, 2011]. However there has been little research that
explores how both technologies can be combined together.
Overall, current AR training systems are not intelligent and current
ITS do not use AR for their front end interface. Current AR training
systems typically provide checklists of actions that must be done to
achieve a task, but don’t provide any feedback as to how well the user
has performed that task. One exception to this is the work of Westerﬁeld
et al. [2013] who combined the ASPIRE constraint based ITS
[Mitrovic et al., 2009] with an AR interface for training people on computer
assembly. Using this interface the system would monitor user
performance and automatically provide virtual cues to help them perform
the tasks they were training on (see Figure 11.2). In trials will this
system it was found that the AR with ITS enabled people to complete
the training task faster and score almost 30% higher on a retention
of knowledge test compared to training with the same AR interface
without intelligent support. This is an encouraging result and shows
signiﬁcant opportunity for more research in the area.
A second promising area for research is hybrid AR interfaces and interaction.
The ﬁrst AR systems were stand-alone applications in which
the user focused entirely on using the AR interface and interacting
with the virtual content. However, as we saw in Milgram’s Mixed Reality
continuum and the Metaverse taxonomy, AR technology is complimentary
to other interface technologies and so hybrid interfaces can
be built which combine AR with other interaction methods. For exam-
11.2. Interaction 225
Figure 11.2: Using AR and ITS for training computer assembly. [Westerﬁeld et al.,
2013]
ple, if we consider Rekimoto’s comparison of HCI styles from section 2,
there are artiﬁcial boundaries between the four interface types shown.
As Figure 11.3 shows, interesting research can be explored by crossing
these boundaries.
Considering each of Figures 11.3(b)~(d) separately, we can see
some interesting opportunities for further research. For example, Figures
11.3(b) shows that the boundary between AR and Ubiquitous
Computing can be crossed to create hybrid AR/Ubicomp interfaces.
This could be accomplished by using AR to make visible what is normally
invisible in Ubiquitous Computing applications. Rauhala et al.
[2006] developed a handheld AR interface that used virtual cue to show
the data from humidity sensors hidden behind a wall in a ubiquitous
computer interface (see Figure 11.4). This allows the used to see superimposed
over the real world data that is usually only visible on a
desktop interface remote from the location that it is captured in. In this
case a physical marker placed on the wall was used for mobile phone
tracking and the sensor information is collected by a computer and
wirelessly transmitted to the mobile device for AR visualization. There
are also other examples of how AR can be used to make visible data captured
by ubiquitous computers and sensing systems [Goldsmith et al.,
2008, Veas, 2012, van der Vlist et al., 2013].
Figure 11.3(c) shows boundary crossing between AR and VR interfaces.
One example of this is the MagicBook interface [Billinghurst
et al., 2001] that uses AR to explore the full length of Milgram’s Mixed
226 Research Directions
(a) Original classiﬁca-
tion
(b) AR and Ubicomp (c) AR and VR
(d) AR and GUI (e) AR, VR, GUI, and
Ubicomp
Figure 11.3: Research opportunities in hybrid AR interfaces.
Reality continuum. The system consists of a real book and handheld
AR display. The user can read the book without the use of any technology
(Figure 11.5(a)). However when they look at the book pages
through the handheld display they can see animated AR content overlaid
on the printed page (Figure 11.5(b)). When they see an AR scene
that they like they can touch a button on the handheld display and
transition into a fully immersive VR viewing mode (Figure 11.5(c)).
In the VR view users are free to turn their heads to look around the
virtual environment and also to travel through the space. In this way
the MagicBook uses AR to support seamless transition from Reality
to Virtual Reality. Other interfaces that also allow users to move from
AR to VR include [Nilsen and Looser, 2005], [Grasset et al., 2011], and
[Kiyokawa et al., 1999].
AR interfaces can also be combined with more traditional desktop
interfaces (Figure 11.3(d)). The Emmie system of Butz et al. [1999]
is a system that mixes several display and interaction types together,
11.2. Interaction 227
Figure 11.4: Using mobile AR for visualization ubiquitous computing sensor data.
[Rauhala et al., 2006]
Figure 11.5: The MagicBook interface showing transitioning into a VR experience.
[Billinghurst et al., 2001]
allowing users to view AR content at the same time as using 2D GUI
interfaces. Wearing see-through HMDs, users can look at normal desktop
screens and view and interact with the content on the screen using
a normal mouse-based interface. However, using a 3D wand they can
also pull the content o the screen and see it as a 3D AR object ﬂoating
in space in front of them. In this way they can drag and drop content
between di erent types of interfaces and seamlessly mix GUI and AR
interaction. More recently Benko et al. [2005] developed a hybrid AR
and table top interface that combine gesture interaction in space with
AR objects and touch input on the table surface.
Finally, Figure 11.3(e) shows how AR can be combined with GUI,
VR and Ubiquitous Computing interfaces. In 2006 Kim et al. [2006]
228 Research Directions
Figure 11.6: Ubiquitous-VR interface concept. [Kim et al., 2006]
proposed the idea of Ubiquitous Virtual Reality (U-VR) which they
deﬁned as "..creating ubiquitous VR environments which make VR
pervasive into our daily lives and ubiquitous by allowing VR to meet ..
ubiquitous computing". Their view is that the technologies of Ubiquitous
computing, Virtual Reality and Mixed Reality can be seamlessly
blended together to provide people will constant access to information,
anywhere and at any time using a variety of interface presentations. To
demonstrate this concept they developed the ubiHome [Oh and Woo,
2005] testbed (see Figure 11.6) which showed a number of concepts
such as the handheld AR CAMAR Viewer [Suh et al., 2007], an AR
table, immersive Mixed Reality Window, and smart ubiTV [Oh et al.,
2005], among other objects. These are all connected together with a
smart services infrastructure, so for example, the CAMAR Viewer can
be used to control the ubiTV and content could be presented over both
the AR table and MRWindow.
11.3. Displays 229
11.3 Displays
Although AR display technology has signiﬁcantly improved from Ivan
Sutherland’s original system, current displays are still very far from
Sutherland’s vision of the "Ultimate Display" [Sutherland, 1965]. There
are important research opportunities available in the design of head
mounted displays, in projection technology, in contact-lens displays,
and other areas.
In section 5 we provided an overview of the use of optical-see
through head mounted displays for Augmented Reality. However traditional
optical-see through displays have a number of disadvantages,
such not being able to support a wide ﬁeld of view, or provide true occlusion
of the real world. The ideal display would be one that provides
a wide ﬁeld of view, supports occlusion of the real world, and provides
images at a variety of focal planes, all in a small unencumbering form
factor. Researchers have been exploring each of these areas although
there is still more work to be done.
A number of di erent designs for optical see-through displays have
been developed that address these shortcomings. For the occlusion
problem, Kiyokawa et al. [2001] and others [Cakmakci et al., 2004, Santos
et al., 2008, Gao et al., 2012] have been researching how electronic
masking elements can be added to optical see-through displays. For
example, Cakmakci et al. [2004] have designed an optical see through
display with a spatial light modulator and polarization-based optics
that can block parts of the scene that should be occluded. To address
the challenged of small ﬁeld of view (FOV), Cheng et al. [2011] uses
tiled optical elements to create an optical see-through display with 55
degree diagonal FOV. Liu et al. [2008] uses a liquid lens to vary the
focal length of an optical see-through head mounted display, enabling
users to see virtual content overlaid on real objects at a wide range
of distances. Hu and Hua [2012] address the same problem by using
a wedge-shaped eyepiece and a compensator. However, each of these
display prototypes just addresses one of the shortcomings of optical
see-through display.
The recent work of Maimone and Fuchs [2013] is one of the ﬁrst optical
see-through AR displays that tries to address all of these problems
230 Research Directions
Figure 11.7: Computational AR eyeglasses and view through the display. [Maimone
and Fuchs, 2013]
in one design. They have developed a prototype that uses a computational
displays approach with stacked spatial light modulator (SLM)
layers placed directly in front of the eyes. These SLM are transparent
elements that can provide a per-pixel occlusion of the view of the real
world and also support for multiple focal planes, achieving image overlays
at di erent focal depths. The ﬁrst version of the display is shown
in Figure 11.7. However it has relatively poor image quality, requires
calibration of the stacked layers, and has an occlusion mask with blurry
edges. Thus there is still room for signiﬁcant improvement.
A second area of display research that will o er interesting research
opportunities is in head mounted projection displays. The ﬁrst head
mounted projection systems were developed by Fisher [1996] and Fergason
[1997]], although it was Rolland [Rolland et al., 1998], Kijima [Kijima
and Ojika, 1997] and Inami [Inami et al., 2000] that provided many
of the ﬁrst examples of head mounted projection display (HMPD) used
for AR experiences. Rolland et al. [Rolland 05] provides a summary of
early HMPD research. These early projector systems were bulky and
lacked portability, however recent advances in pico-projector technology
has meant that these limitations can be overcome. For example,
the REFLCT system [Krum et al., 2012] uses a small HMPD and retroreﬂective
material to allow users to see projected AR content without
the need for placing an optical combiner in front of the eyes. The castAR3
system is smaller still, with two micro-projectors directly over the
eyes creating a stereo 3D projected image. In this case the user has to
wear lightweight polarized glasses and look at a retro-reﬂective surface
to see the 3D imagery.
3
http://technicalillusions.com
11.4. Social Acceptance 231
Now that HMPD are as small and lightweight as head mounted displays,
there is still considerable research to be conducted. For example,
a number of people have begun to explore interaction methods should
be used for projected AR content. Hua et al. [2003] explore the use
of datagloves for gesture based interaction, Brown and Hua [2006] use
physical objects as tangible interface widgets, and researchers have developed
projected AR interfaces that use freehand input [Benko et al.,
2012]. Another important problem is exploring how to overcome some
of the limitations of projecting on the retro-reﬂective surface, such as
real objects always occluding the projected AR content, even when the
content is supposed to be in-front of the real object. This may be solved
by creating hybrid HMD and HMPD systems.
One of the most interesting types of displays being currently researched
is contact lens based displays. One of the long-term goals of
AR researchers is to have a head worn display that is imperceptible
to people surrounding the user. This might be achievable through ineye
contact lens displays [Parviz, 2009]. Using MEMS technology and
wireless power and data transfer a contact lens could be fabricated that
contains active pixels. Pandey et al. [2010] developed a single pixel contact
lens display that could be wirelessly powered. This was then tested
in the eye of a rabbit with no adverse e ects [Lingley et al., 2011] (see
Figure 11.8(a)). Since that time researchers have developed a curved
liquid crystal cell that can be integrated into a contact lens [De Smet
et al., 2013] (see Figure 11.8(b)). However there are a number of significant
problems that will need be to solved before such displays become
readily usable, such as adding optics to allow the eye to focus on the
display, ensuring su cient permeability to oxygenate the cornea, and
providing continuous power and data.
11.4 Social Acceptance
As Augmented Reality technology has moved from the lab to the living
room, some of the obstacles that prevent it from being more widely
used are Social rather than Technological, particularly for wearable or
mobile systems.
232 Research Directions
(a) Single pixel contact
lens in rabbit
eye [Lingley et al.,
2011]
(b) Curved liquid crystal display embedded in contact
lens [De Smet, 2014]
Figure 11.8: Prototype contact lens displays.
For example wearable AR systems have shrunk in size from bulky
backpacks to small head worn displays like Google Glass4. Despite this,
there is still considerable social pushback that is preventing devices like
Google Glass from being accepted in some social settings. For example,
in a recent survey of 4,600 US adults, only 12 percent said that they
would be willing to wear "augmented reality glasses" that came from a
brand they trusted5. This reluctance could be for several reasons such
concerns about privacy [Hong, 2013], or fear that using the technology
will make a user look silly, or make the user a target for thieves.
These concerns are not just conﬁned to head worn wearable systems.
For example, if a person is walking through a city with a mobile phone
or tablet and using an AR browser to help them navigate, or viewing
AR content, they may need to hold the phone at eye level in front of
them while walking (see Figure 11.9). This is an unnatural pose that
may make them feel silly, look foolish, or make other people feel that
they are being ﬁlmed.
A number of researchers have raised social acceptance as an important
issue for AR [Azuma et al., 2001, Martínez et al., 2014, Carmigniani
et al., 2011, Narzt et al., 2006], but initially there was been little
4
http://www.google.com/glass
5
http://www.newsfactor.com/story.xhtml?story_id=12100BWEZRVS
11.4. Social Acceptance 233
Figure 11.9: Walking while looking at AR virtual tour content on a tablet. [Lee
et al., 2013a]
research in the ﬁeld. For example, in Zhou’s ISMAR research review
paper, none of the papers published up to 2007 had focused on Social
Acceptance. More recently there has been some work addressing this
issue. For example Nilsson and Johansson [2008] describes an AR training
interface for a medical device in a hospital and evaluation that was
performed in terms of technology usefulness and social acceptance. The
study participants were very positive about the use of AR technology
as a tool for clinical instruction and there were no issues raised in terms
of social acceptance. Similarly Rasimah et al. [2011] report on a study
of the perceived usefulness and acceptance of AR in a university setting
with more than 60 biomedical student subjects. They found that
nearly 95% of the students felt that AR would useful in teaching and
learning, and over 85% said they would use the technology frequently
if they had access to it.
However both of these studies we conducted in professional or educational
settings. In a hospital is it common for medical practitioners to
wear devices to help them perform their tasks, while in a university students
often use a variety of pieces of equipment in labs and classes for
short periods of time. It is very di erent to use a wearable or handheld
AR device in a social setting or in public in front of strangers.
234 Research Directions
There has been little research dedicated to the topic. Olsson points
out ".. despite a large body of AR demonstrators and related research,
the AR research community lacks understanding of what the user experience
with mobile AR services could be, .. especially regarding the emotional
and hedonic elements of user experience." [Olsson et al., 2013].
The closest research to this is the work of Grubert et al. [2011] who
conducted an online survey of 77 people who have used mobile AR
browser applications. They asked them about the social aspects of mobile
AR use and around 20% of users said that they experienced social
issues with using an AR browser application either often or very often.
A similar percentage also said that they had experienced situations in
which in which they refrained from using the application because they
were worried about social issues. This is a relatively small percentage,
but users were using mobile AR applications for just a short period
of a time each task (70% of users reported using the application for
less than 5 minutes at a time), and so were unlikely to draw signiﬁcant
attention.
It would be expected that social acceptance problems will be signiﬁcantly
higher with a person that is wearing a head mounted display
all the time while interacting with AR content. Unfortunately this is
an area that has not been well studied in the AR community, but as
most social acceptance issues seem to arise from the use of mobile or
wearable AR systems, then there is related from the wearable computing
community that could be beneﬁcial. For example, Serrano et al.
[2014] explored the social acceptability of making hand to face gestures
for controlling a head mounted display, ﬁnding that they could be acceptable
in di erent social contexts, but care would needed to taken
around the design of the gestures. However their studies were in a controlled
laboratory environment and not in a public setting where the
wearable computer would be typically used. Rico and Brewster [2010]
have previously evaluated the social acceptability of gestures for mobile
user (although not wearable computers), ﬁnding that location and audience
have a signiﬁcant impact on the type of gestures people will be
willing to perform. They found that gestures that required the participant
to perform large or noticeable actions were the most commonly
11.5. Summary 235
disliked gestures. Other research has explored the social acceptability
of shoe mounted input devices [Bailly et al., 2012], or minimal displays
[Costanza et al., 2006].
This research shows that social acceptability is an important issue
in handheld or wearable AR experiences, especially those that are expected
to be used in a public setting. This is an area that has had
little research to date from the AR community and so could form an
important area for work in the future. Of course as the technology becomes
more unobtrusive with displays and input devices vanishing into
clothing then some social concerns may go away.
11.5 Summary
In this section we have reviewed four areas that will provide signiﬁcant
opportunities for further research in AR in the coming years. However,
there are many other topics that can also be studied in the ﬁeld of AR.
Zhou et al. [2008] identiﬁed a total of 11 categories of research papers
that had been published in the ten years of the ISMAR conference
until 2008, and categories such as Social Acceptance or Collaborative
interfaces were not included amongst these. So it should be accepted
that there will be at least as many research topics going forward. As
the ﬁeld matures, the challenge for AR researchers will be to continue
to identify the most signiﬁcant obstacles that prevent the development
of compelling AR experiences and to ﬁnd ways to overcome them.
12
Conclusion
Nearly ﬁfty years ago Ivan Sutherland imagined an "Ultimate Display"
that would allow people to see digital content become part of their real
worlds [Sutherland, 1965]. In a famous quote he says: "The ultimate
display would, of course, be a room within which the computer can
control the existence of matter. A chair displayed in such a room would
be good enough to sit in. Handcu s displayed in such a room would
be conﬁning, and a bullet displayed in such a room would be fatal.
With appropriate programming such a display could literally be the
Wonderland into which Alice walked."
Although complete Science Fiction at the time, this article has
shown how Augmented Reality is beginning to achieve Sutherland’s
vision. What was once usable only in research labs is now available
to hundreds of millions of people through the technology they have in
their pockets, homes and workplaces. It is easier than ever before to
create content and applications for AR enabled devices, meaning that
even non-programmers can create rich AR experiences. Adoption is
only expected to increase with the availability of inexpensive wearable
computers (e.g. Google Glass) and head mounted displays (e.g. Oculus
Rift), and the growth in AR applications available.
236
237
Signiﬁcant progress has been made in solving many of the fundamental
problems in AR. For example, in section 4 we showed that
there are a number of computer vision libraries that provide robust
tracking from printed images, and there are sensor based systems that
can provide relatively accurate outdoor tracking. A variety of AR display
options were reviewed in section 5. Similarly, section 6 presented a
wide range of authoring tools that can be used to create AR experience.
This means that the research ﬁeld is evolving away from being solely
focused on low-level enabling technology, to exploring AR experiences,
using interaction methods reviewed in section 7, and design guidelines
contained in section 8. section 9 presented a variety of methods that
can be used to evaluate the quality of user experience and showed that
more and more AR papers with evaluation studies in them are being
published. Finally, section 10 presented some case study examples of
AR applications being used today.
Researchers in the ﬁeld have made tremendous advances since the
1960’s, but in many ways the full potential of Augmented Reality is
still to be realized. Section 11 touched on several of the promising areas
for future research such as social acceptance and hybrid AR interfaces,
although there are many other directions people have begun to explore.
The pace of research is dramatically increasing and the developments
to date will seem trivial when we look back in another 50 years time. It
is certain that AR will dramatically change how humans interact with
digital content in the years to come.
Acknowledgements
We would like to thank all those who have kindly granted us to use
ﬁgures from their original work.
238
References
Kyriacos Achilleos. Evolution of lifelogging, 2003. URL http://mms.ecs.
soton.ac.uk/2010/papers/Evolution_of_Lifelogging.pdf.
Matt Adcock, Matthew Hutchins, and Chris Gunn. Augmented reality haptics:
Using artoolkit for display of haptic applications. In Proceedings of
2nd IEEE International Augmented Reality Toolkit Workshop, pages 1–2,
2003.
Kusuma Agusanto, Li Li, Zhu Chuangui, and Ng Wan Sing. Photorealistic
rendering for augmented reality using environment illumination. In Mixed
and Augmented Reality, 2003. Proceedings. The Second IEEE and ACM
International Symposium on, pages 208–216. IEEE, 2003.
Sangchul Ahn, Heedong Ko, and Steven Feiner. Webizing mobile ar contents.
In Proceedings of Virtual Reality 2013 (VR 2013), pages 131–132. IEEE,
2013.
Manu Akula, Suyang Dong, Vineet R Kamat, Lauro Ojeda, Adam Borrell,
and Johann Borenstein. Integration of infrastructure based positioning
systems and inertial navigation for ubiquitous context-aware engineering
applications. Advanced Engineering Informatics, 25(4):640–655, 2011.
Alexandre Alahi, Raphael Ortiz, and Pierre Vandergheynst. Freak: Fast retina
keypoint. In Computer Vision and Pattern Recognition (CVPR), 2012
IEEE Conference on, pages 510–517. IEEE, 2012.
Christopher Alexander, Sara Ishikawa, and Murray Silverstein. A Pattern
Language. Oxford Univ. Press, 1977.
239
240 References
Mahoro Anabuki, Hiroyuki Kakuta, Hiroyuki Yamamoto, and Hideyuki
Tamura. Welbo: An embodied conversational agent living in mixed reality
space. In CHI’00 extended abstracts on Human factors in computing
systems, pages 10–11. ACM, 2000.
Anne H. Anderson, Alison Newlands, Jim Mullin, Anne Marie Fleming,
Gwyneth Doherty-Sneddon, and Jeroen Van der Velden. Impact of videomediated
communication on simulated service encounters. Interacting with
computers, 8(2):193–206, 1996.
Alissa N. Antle and Alyssa F. Wise. Getting down to details: Using theories of
cognition and learning to inform tangible user interface design. Interacting
with Computers, 25(1):1–20, 2013.
Clemens Arth, Daniel Wagner, Manfred Klopschitz, Arnold Irschara, and Dieter
Schmalstieg. Wide area localization on mobile phones. In Mixed and
Augmented Reality, 2009. ISMAR 2009. 8th IEEE International Symposium
on, pages 73–82. IEEE, 2009.
Mark Assad, David J. Carmichael, Daniel Cutting, and Adam Hudson. Ar
phone: Accessible augmented reality in the intelligent environment. In Proceedings
of OZCHI, pages 232–237, 2003.
Thomas Auer and Axel Pinz. Building a hybrid tracking system: Integration
of optical and magnetic tracking. In Augmented Reality, 1999.(IWAR’99)
Proceedings. 2nd IEEE and ACM International Workshop on, pages 13–22.
IEEE, 1999.
Ronald Azuma. Tracking requirements for augmented reality. Communications
of the ACM, 36(7):50–51, 1993.
Ronald Azuma and Gary Bishop. Improving static and dynamic registration
in an optical see-through hmd. In Proceedings of the 21st annual conference
on Computer graphics and interactive techniques, pages 197–204. ACM,
1994.
Ronald Azuma, Yohan Baillot, Reinhold Behringer, Steven Feiner, Simon
Julier, and Blair MacIntyre. Recent advances in augmented reality. Computer
Graphics and Applications, IEEE, 21(6):34–47, 2001.
Ronald T. Azuma. A survey of augmented reality. Presence, 6(4):355–385,
1997.
Ronald T. Azuma, Bruce R. Ho , Howard E. Neely III, Ronald Sarfaty,
Michael J Daily, Gary Bishop, Vern Chi, Greg Welch, Ulrich Neumann,
Suya You, et al. Making augmented reality work outdoors requires hybrid
tracking. In Proceedings of the First International Workshop on Augmented
Reality, volume 1, 1998.
References 241
Gilles Bailly, Jörg Müller, Michael Rohs, Daniel Wigdor, and Sven Kratz.
Shoesense: a new perspective on gestural interaction and wearable applications.
In Proceedings of the SIGCHI Conference on Human Factors in
Computing Systems, pages 1239–1248. ACM, 2012.
Michael Bajura and Ulrich Neumann. Dynamic registration correction in
augmented-reality systems. In Virtual Reality Annual International Symposium,
1995. Proceedings., pages 189–196. IEEE, 1995.
Michael Bajura, Henry Fuchs, and Ryutarou Ohbuchi. Merging virtual objects
with the real world: Seeing ultrasound imagery within the patient. ACM
SIGGRAPH Computer Graphics, 26(2):203–210, 1992.
Martin Bauer, Bernd Bruegge, Gudrun Klinker, Asa MacWilliams, Thomas
Reicher, Stefan Riss, Christian Sandor, and Martin Wagner. Design of
a component-based augmented reality framework. In Augmented Reality,
2001. Proceedings. IEEE and ACM International Symposium on, pages 45–
54. IEEE, 2001.
Herbert Bay, Tinne Tuytelaars, and Luc Van Gool. Surf: Speeded up robust
features. In Computer vision–ECCV 2006, pages 404–417. Springer, 2006.
Herbert Bay, Andreas Ess, Tinne Tuytelaars, and Luc Van Gool. Speeded-up
robust features (surf). Computer vision and image understanding, 110(3):
346–359, 2008.
Paul Beardsley, Jeroen Van Baar, Ramesh Raskar, and Clifton Forlines. Interaction
using a handheld projector. Computer Graphics and Applications,
IEEE, 25(1):39–43, 2005.
Amir H. Behzadan, Zeeshan Aziz, Chimay J. Anumba, and Vineet R. Kamat.
Ubiquitous location tracking for context-speciﬁc information delivery on
construction sites. Automation in Construction, 17(6):737–748, 2008.
Je rey S. Beis and David G. Lowe. Shape indexing using approximate nearestneighbour
search in high-dimensional spaces. In Computer Vision and Pattern
Recognition, 1997. Proceedings., 1997 IEEE Computer Society Conference
on, pages 1000–1006. IEEE, 1997.
Mathilde M. Bekker, Judith S. Olson, and Gary M. Olson. Analysis of gestures
in face-to-face design teams provides guidance for how to use groupware
in design. In Proceedings of the 1st conference on Designing interactive
systems: processes, practices, methods, & techniques, pages 157–166. ACM,
1995.
Hrvoje Benko, Edward W. Ishak, and Steven Feiner. Cross-dimensional gestural
interaction techniques for hybrid immersive environments. In Proceedings
of Virtual Reality 2005 (VR 2005), pages 209–216. IEEE, 2005.
242 References
Hrvoje Benko, Ricardo Jota, and Andrew Wilson. Miragetable: freehand interaction
on a projected augmented reality tabletop. In Proceedings of the
SIGCHI conference on human factors in computing systems, pages 199–208.
ACM, 2012.
Devesh Kumar Bhatnagar. Position trackers for head mounted display systems:
A survey. Technical Report TR93-10, University of North Carolina,
Chapel Hill, 1993.
Mark Billinghurst. Augmented reality in education. New Horizons for Learning,
12, 2002.
Mark Billinghurst and Andreas Dünser. Augmented reality in the classroom.
Computer, 45(7):56–63, 2012. ISSN 0018-9162.
Mark Billinghurst and Hirokazu Kato. Out and about—real world teleconferencing.
BT technology journal, 18(1):80–82, 2000.
Mark Billinghurst, Suzanne Weghorst, and Thomas Furness. Shared space:
Collaborative augmented reality. In Proceedings of CVE’ 96 Workshop,
1996.
Mark Billinghurst, Suzane Weghorst, and Tom Furness III. Wearable computers
for three dimensional cscw. In 2012 16th International Symposium
on Wearable Computers, pages 39–39. IEEE Computer Society, 1997.
Mark Billinghurst, Jerry Bowskill, and Jason Morphett. Wearcom: A wearable
communication space. In Proceedings of CVE, volume 98, 1998.
Mark Billinghurst, Ivan Poupyrev, Hirokazu Kato, and Richard May. Mixing
realities in shared space: An augmented reality interface for collaborative
computing. In Multimedia and Expo, 2000. ICME 2000. 2000 IEEE International
Conference on, volume 3, pages 1641–1644. IEEE, 2000.
Mark Billinghurst, Hirokazu Kato, and Ivan Poupyrev. The magicbook: a
transitional ar interface. Computers & Graphics, 25(5):745–753, 2001.
Mark Billinghurst, Raphael Grasset, and Julian Looser. Designing augmented
reality interfaces. ACM Siggraph Computer Graphics, 39(1):17–22, 2005.
Oliver Bimber and Ramesh Raskar. Spatial augmented reality. In International
Conference on Computer Graphics and Interactive Techniques: ACM
SIGGRAPH 2006 Courses, volume 2006, 2006.
Oliver Bimber, L Miguel Encarnação, and Pedro Branco. The extended virtual
table: An optical extension for table-like projection systems. Presence:
Teleoperators and Virtual Environments, 10(6):613–631, 2001a.
References 243
Oliver Bimber, Bernd Frohlich, Dieter Schmalsteig, and L. Miguel Encarnacao.
The virtual showcase. Computer Graphics and Applications, IEEE,
21(6):48–55, 2001b.
Gabor Blasko, Franz Coriand, and Steven Feiner. Exploring interaction with
a simulated wrist-worn projection display. In Wearable Computers, 2005.
Proceedings. Ninth IEEE International Symposium on, pages 2–9. IEEE,
2005.
Richard A. Bolt. "put-that-there": Voice and gesture at the graphics interface.
SIGGRAPH Computer Graphics, 14(3):262–270, 1980.
Jan Borchers. A Pattern Approach to Interaction Design. Wiley, 2001.
Doug A Bowman, Ernst Kruij , Joseph J. LaViola Jr, and Ivan Poupyrev.
3D user interfaces: theory and practice. Addison-Wesley, 2004.
Elizabeth A. Boyle, Anne H. Anderson, and Alison Newlands. The e ects
of visibility on dialogue and performance in a cooperative problem solving
task. Language and speech, 37(1):1–20, 1994.
José M. Braz and João M. Pereira. Tarcast: Taxonomy for augmented reality
casting with web support. The International Journal of Virtual Reality, 7
(4):47–56, 2008.
David E. Breen, Ross T. Whitaker, Eric Rose, and Mihran Tuceryan. Interactive
occlusion and automatic object placement for augmented reality.
Computer Graphics Forum, 15(3):11–22, 1996.
Jeremy Brooker. The polytechnic ghost: Pepper’s ghost, metempsychosis and
the magic lantern at the royal polytechnic institution. Early Popular Visual
Culture, 5(2):189–206, 2007.
Leonard D. Brown and Hong Hua. Magic lenses for augmented virtual environments.
Computer Graphics and Applications, 26(4):64–73, 2006.
Amy Bruckman and Alisa Bandlow. Human-computer interaction for kids. In
Julie A. Jacko and Andrew Sears, editors, The Human-computer Interaction
Handbook, pages 428–440. L. Erlbaum Associates Inc., 2003.
Grigore Burdea and Philippe Coi et. Virtual reality technology. Presence:
Teleoperators and virtual environments, 12(6):663–664, 2003.
Andreas Butz, Tobias Hollerer, Steven Feiner, Blair MacIntyre, and Cli ord
Beshers. Enveloping users and computers in a collaborative 3d augmented
reality. In Augmented Reality, 1999.(IWAR’99) Proceedings. 2nd IEEE and
ACM International Workshop on, pages 35–44. IEEE, 1999.
244 References
Ozan Cakmakci, Yonggang Ha, and Jannick P. Rolland. A compact optical
see-through head-worn display with occlusion support. In Proceedings of
the 3rd IEEE/ACM International Symposium on Mixed and Augmented
Reality, pages 16–25. IEEE Computer Society, 2004.
Michael Calonder, Vincent Lepetit, Christoph Strecha, and Pascal Fua. Brief:
Binary robust independent elementary features. In Computer Vision–
ECCV 2010, pages 778–792. Springer, 2010.
Christer Carlsson and Olof Hagsand. Dive—a platform for multi-user virtual
environments. Computers & graphics, 17(6):663–669, 1993.
Julie Carmigniani, Borko Furht, Marco Anisetti, Paolo Ceravolo, Ernesto
Damiani, and Misa Ivkovic. Augmented reality technologies, systems and
applications. Multimedia Tools and Applications, 51(1):341–377, 2011.
Robert Castle, Georg Klein, and David W. Murray. Video-rate localization
in multiple maps for wearable augmented reality. In Wearable Computers,
2008. ISWC 2008. 12th IEEE International Symposium on, pages 15–22.
IEEE, 2008.
Thomas P. Caudell and David W. Mizell. Augmented reality: An application
of heads-up display technology to manual manufacturing processes. In System
Sciences, 1992. Proceedings of the Twenty-Fifth Hawaii International
Conference on, volume 2, pages 659–669. IEEE, 1992.
Alphonse Chapanis. Interactive human communication. Scientiﬁc American,
232:36–42, 1975.
Dewen Cheng, Yongtian Wang, Hong Hua, and Jose Sasian. Design of a
wide-angle, lightweight head-mounted display using free-form optics tiling.
Optics letters, 36(11):2098–2100, 2011.
Youngkwan Cho, Jun Park, and Ulrich Neumann. Fast color ﬁducial detection
and dynamic workspace extension in video see-through self-tracking
augmented reality. In Computer Graphics and Applications, 1997. Proceedings.,
The Fifth Paciﬁc Conference on, pages 168–177. IEEE, 1997.
Youngkwan Cho, Jongweon Lee, and Ulrich Neumann. A multi-ring color
ﬁducial system and an intensity-invariant detection method for scalable
ﬁducial-tracking augmented reality. In Proceedings of International Workshop
on Augmented Reality ’98, 1998.
Jinhyuk Choi, Youngsun Kim, Myonghee Lee, Gerard J. Kim, Yanghee Nam,
and Yongmoo Kwon. k-mart: Authoring tool for mixed reality contents.
In Mixed and Augmented Reality (ISMAR), 2010 9th IEEE International
Symposium on, pages 219–220. IEEE, 2010.
References 245
Jan Ciger, Mario Gutierrez, Frederic Vexo, and Daniel Thalmann. The magic
wand. In Proceedings of the 19th spring conference on Computer graphics,
pages 119–124. ACM, 2003.
Adrian Clark and Andreas Dünser. An interactive augmented reality coloring
book. In 3D User Interfaces (3DUI), 2012 IEEE Symposium on, pages
7–10. IEEE, 2012.
Adrian Clark and Thammathip Piumsomboon. A realistic augmented reality
racing game using a depth-sensing camera. In Proceedings of the 10th International
Conference on Virtual Reality Continuum and Its Applications
in Industry, pages 499–502. ACM, 2011.
Philip Cohen, David McGee, Sharon Oviatt, Lizhong Wu, Joshua Clow,
Robert King, Simon Julier, and Lawrence Rosenblum. Multimodal interaction
for 2d and 3d environments. IEEE Computer Graphics and Applications,
19(4):10–13, 1999.
Philip R. Cohen, Mary Dalrymple, Douglas B. Moran, F. C. Pereira, and
Joseph W. Sullivan. Synergistic use of direct manipulation and natural
language. ACM SIGCHI Bulletin, 20(SI):227–233, 1989.
Philip R. Cohen, Michael Johnston, David McGee, Sharon Oviatt, Jay
Pittman, Ira Smith, Liang Chen, and Josh Clow. Quickset: Multimodal
interaction for distributed applications. In Proceedings of the ﬁfth ACM
international conference on Multimedia, pages 31–40. ACM, 1997.
Andrew I. Comport, Éric Marchand, and François Chaumette. A realtime
tracker for markerless augmented reality. In Proceedings of the 2nd
IEEE/ACM International Symposium on Mixed and Augmented Reality,
page 36. IEEE Computer Society, 2003.
Andrew I. Comport, Eric Marchand, Muriel Pressigout, and Francois
Chaumette. Real-time markerless tracking for augmented reality: the virtual
visual servoing framework. Visualization and Computer Graphics,
IEEE Transactions on, 12(4):615–628, 2006.
Sam Corbett-Davies, Andreas Dunser, Richard Green, and Adrian Clark.
An advanced interaction framework for augmented reality based exposure
treatment. In Virtual Reality (VR), 2013 IEEE, pages 19–22. IEEE, 2013.
Maria Francesca Costabile and Maristella Matera. Evaluating wimp interfaces
through the sue approach. In Image Analysis and Processing, 1999.
Proceedings. International Conference on, pages 1192–1197. IEEE, 1999.
246 References
Enrico Costanza, Samuel A. Inverso, Elan Pavlov, Rebecca Allen, and Pattie
Maes. eye-q: Eyeglass peripheral display for subtle intimate notiﬁcations.
In Proceedings of the 8th conference on Human-computer interaction with
mobile devices and services, pages 211–218. ACM, 2006.
Owen Daly-Jones, Andrew Monk, and Leon Watts. Some advantages of video
conferencing over high-quality audio conferencing: ﬂuency and awareness
of attentional focus. International Journal of Human-Computer Studies,
49(1):21–58, 1998.
Fred D. Davis, Richard P. Bagozzi, and Paul R. Warshaw. User acceptance of
computer technology: a comparison of two theoretical models. Management
science, 35(8):982–1003, 1989.
Andrew J. Davison, Ian D. Reid, Nicholas D. Molton, and Olivier Stasse.
Monoslam: Real-time single camera slam. Pattern Analysis and Machine
Intelligence, IEEE Transactions on, 29(6):1052–1067, 2007.
Jelle De Smet. The Smart Contact Lens: From an Artiﬁcial Iris to a Contact
Lens Display. PhD thesis, Faculty of Engineering and Architecture, Ghent
University, Ghent, Belgium, 2014.
Jelle De Smet, Aykut Avci, Pankaj Joshi, David Schaubroeck, Dieter Cuypers,
and Herbert De Smet. Progress toward a liquid crystal contact lens display.
Journal of the Society for Information Display, 21(9):399–406, 2013.
MWM Gamini Dissanayake, Paul Newman, Steve Clark, Hugh F. DurrantWhyte,
and Michael Csorba. A solution to the simultaneous localization
and map building (slam) problem. Robotics and Automation, IEEE Transactions
on, 17(3):229–241, 2001.
Neil A. Dodgson. Autostereoscopic 3d displays. Computer, 38(8):31–36, 2005.
Klaus Dorfmüller. Robust tracking for augmented reality using retroreﬂective
markers. Computers & Graphics, 23(6):795–800, 1999.
Ralf Dörner, Christian Geiger, Michael Haller, and Volker Paelke. Authoring
mixed reality—a component and framework-based approach. In Entertainment
Computing, pages 405–413. Springer, 2003.
David Drascic and Paul Milgram. Positioning accuracy of a virtual stereographic
pointer in a real stereoscopic video world. In Electronic Imaging’91,
San Jose, CA, pages 302–313. International Society for Optics and Photonics,
1991.
David Drascic and Paul Milgram. Perceptual issues in augmented reality.
In Electronic Imaging: Science & Technology, pages 123–134. International
Society for Optics and Photonics, 1996.
References 247
David Drascic, Julius J Grodski, Paul Milgram, Ken Ru o, Peter Wong, and
Shumin Zhai. Argos: A display system for augmenting reality. In Proceedings
of the INTERACT’93 and CHI’93 Conference on Human Factors in
Computing Systems, page 521. ACM, 1993.
Andreas Dünser. Supporting low ability readers with interactive augmented
reality. Annual Review of CyberTherapy and Telemedicine, page 39, 2008.
Andreas Dünser and Eva Hornecker. Lessons from an ar book study. In
Proceedings of the 1st international conference on Tangible and embedded
interaction, pages 179–182. ACM, 2007.
Andreas Dünser, Raphaël Grasset, and Mark Billinghurst. A survey of evaluation
techniques used in augmented reality studies. Technical Report
TR2008-02, The Human Interface Technology Laboratory New Zealand,
University of Canterbury, 2008.
Paul Ekman and Wallace V. Friesen. The repertoire of nonverbal behavior:
Categories, origins, usage, and coding. In Adam Kendon, Thomas A.
Sebeok, and Jean Umiker-Sebeok, editors, Nonverbal communication, interaction,
and gesture, pages 57–106. Mouton Publishers, 1981.
Stephen R. Ellis and Brian M. Menges. Judgments of the distance to nearby
virtual objects: interaction of viewing conditions and accommodative demand.
Presence (Cambridge, Mass.), 6(4):452–460, 1997.
Stephen R. Ellis and Brian M. Menges. Studies of the localization of virtual
objects in the near visual ﬁeld. Lawrence Erlbaum Associates, 2001.
Frédéric Evennou and François Marx. Advanced integration of wiﬁ and inertial
navigation systems for indoor mobile positioning. Eurasip journal on
applied signal processing, 2006:164–164, 2006.
J. Fagard. The development of bimanual coordination, 1990.
Gregg E. Favalora. Volumetric 3d displays and application infrastructure.
IEEE Computer Society, 8:37–44, 2005.
Steven Feiner, Blair MacIntyre, Marcus Haupt, and Eliot Solomon. Windows
on the world: 2d windows for 3d augmented reality. In Proceedings of the 6th
annual ACM symposium on User interface software and technology, pages
145–155. ACM, 1993a.
Steven Feiner, Blair Macintyre, and Dorée Seligmann. Knowledge-based augmented
reality. Communications of the ACM, 36(7):53–62, 1993b.
Steven Feiner, Blair MacIntyre, Tobias Höllerer, and Anthony Webster. A
touring machine: Prototyping 3d mobile augmented reality systems for exploring
the urban environment. Personal Technologies, 1(4):208–217, 1997.
248 References
James L. Fergason. Optical system for a head mounted display using a retroreﬂector
and method of displaying an image, April 15 1997. US Patent
5,621,572.
Mark Fiala. Artag revision 1, a ﬁducial marker system using digital techniques.
Technical report, National Research Council Canada, 2004.
Mark Fiala. Artag, a ﬁducial marker system using digital techniques. In Computer
Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer
Society Conference on, volume 2, pages 590–596. IEEE, 2005a.
Mark Fiala. Comparing artag and artoolkit plus ﬁducial marker systems.
In Haptic Audio Visual Environments and their Applications, 2005. IEEE
International Workshop on, pages 6–pp. IEEE, 2005b.
Martin A. Fischler and Robert C. Bolles. Random sample consensus: a
paradigm for model ﬁtting with applications to image analysis and automated
cartography. Communications of the ACM, 24(6):381–395, 1981.
Ralph W. Fisher. Head-mounted projection display system featuring beam
splitter and method of making same, November 5 1996. US Patent
5,572,229.
Scott S. Fisher, Micheal McGreevy, James Humphries, and Warren Robinett.
Virtual environment display system. In Proceedings of the 1986 workshop
on Interactive 3D graphics, pages 77–87. ACM, 1987.
Paul M. Fitts. The information capacity of the human motor system in controlling
the amplitude of movement. Journal of experimental psychology,
47(6):381, 1954.
Paul M. Fitts and James R. Peterson. Information capacity of discrete motor
responses. Journal of experimental psychology, 67(2):103, 1964.
George W. Fitzmaurice. Situated information spaces and spatially aware
palmtop computers. Communications of the ACM, 36(7):39–49, 1993.
George W. Fitzmaurice and William Buxton. An empirical evaluation of
graspable user interfaces: towards specialized, space-multiplexed input. In
Proceedings of the ACM SIGCHI Conference on Human factors in computing
systems, pages 43–50. ACM, 1997.
Morten Fjeld and Benedikt M. Voegtli. Augmented chemistry: An interactive
educational workbench. In Mixed and Augmented Reality, 2002.
ISMAR 2002. Proceedings. International Symposium on, pages 259–321.
IEEE, 2002.
Eric Foxlin. Inertial head-tracker sensor fusion by a complementary separatebias
kalman ﬁlter. In Virtual Reality Annual International Symposium,
1996., Proceedings of the IEEE 1996, pages 185–194. IEEE, 1996.
References 249
Eric Foxlin, Michael Harrington, and George Pfeifer. Constellation: a widerange
wireless motion-tracking system for augmented reality and virtual
set applications. In Proceedings of the 25th annual conference on Computer
graphics and interactive techniques, pages 371–378. ACM, 1998.
Eric Foxlin, Yury Altshuler, Leonid Naimark, and Mike Harrington. Flighttracker:
a novel optical/inertial tracker for cockpit enhanced vision. In
Proceedings of the 3rd IEEE/ACM International Symposium on Mixed and
Augmented Reality, pages 212–221. IEEE Computer Society, 2004.
Eric M. Foxlin. Head tracking relative to a moving vehicle or simulator platform
using di erential inertial sensors. In AeroSense 2000, pages 133–144.
International Society for Optics and Photonics, 2000.
Henry Fuchs, Andrei State, Etta D. Pisano, William F. Garrett, Gentaro
Hirota, Mark Livingston, Mary C. Whitton, and Stephen M. Pizer. Towards
performing ultrasound-guided needle biopsies from within a headmounted
display. In Visualization in Biomedical Computing, pages 591–600.
Springer, 1996.
Henry Fuchs, Mark A. Livingston, Ramesh Raskar, Kurtis Keller, Jessica R.
Crawford, Paul Rademacher, Samuel H. Drake, Anthony A. Meyer, et al.
Augmented reality visualization for laparoscopic surgery. Springer, 1998.
Anton Fuhrmann, Dieter Schmalstieg, and Werner Purgathofer. Fast calibration
for augmented reality. In Proceedings of the ACM symposium on
Virtual reality software and technology, pages 166–167. ACM, 1999.
Thomas A. Furness III. The application of head-mounted displays to airborne
reconnaissance and weapon delivery. Technical Report TR-69-241, WrightPatterson
Air Force Bose, OH: U.S. Air Force Avionics Laboratory, 1969.
Thomas A. Furness III. The super cockpit and its human factors challenges.
Proceedings of the Human Factors and Ergonomics Society Annual Meeting,
30(1):48–52, 1986.
Susan R. Fussell, Robert E. Kraut, and Jane Siegel. Coordination of communication:
E ects of shared visual context on collaborative work. In Proceedings
of the 2000 ACM conference on Computer supported cooperative
work, pages 21–30. ACM, 2000.
Joseph L. Gabbard and J. Edward Swan. Usability engineering for augmented
reality: Employing user-based studies to inform design. Visualization and
Computer Graphics, IEEE Transactions on, 14(3):513–525, 2008.
Joseph L. Gabbard, Deborah Hix, and J. Edward Swan. User-centered design
and evaluation of virtual environments. Computer Graphics and Applications,
IEEE, 19(6):51–59, 1999.
250 References
Chunyu Gao, Yuxiang Lin, and Hong Hua. Occlusion capable optical seethrough
head-mounted display using freeform optics. In Mixed and Augmented
Reality (ISMAR), 2012 IEEE International Symposium on, pages
281–282. IEEE, 2012.
Christian Geiger, Bernd Kleinnjohann, Christian Reimann, and Dirk Stichling.
Mobile ar4all. In Augmented Reality, 2001. Proceedings. IEEE and
ACM International Symposium on, pages 181–182. IEEE, 2001.
David Gelernter. Mirror worlds, or, The day software puts the universe in
a shoebox–: how it will happen and what it will mean. Oxford University
Press New York, 1991.
James J. Gibson. The theory of a ordances. In Jen Jack Gieseking, William
Mangold, Cindi Katz, Setha Low, and Susan Saegert, editors, The People,
Place, and Space Reader, chapter 9, pages 56–60. Routledge, 1979.
S. Burak Gokturk, Hakan Yalcin, and Cyrus Bamji. A time-of-ﬂight depth
sensor-system description, issues and solutions. In Computer Vision and
Pattern Recognition Workshop, 2004. CVPRW’04. Conference on, pages
35–35. IEEE, 2004.
Daniel Goldsmith, Fotis Liarokapis, Garry Malone, and John Kemp. Augmented
reality environmental monitoring using wireless sensor networks.
In Information Visualisation, 2008. IV’08. 12th International Conference,
pages 539–544. IEEE, 2008.
Stefan Gottschalk and John F. Hughes. Autocalibration for virtual environments
tracking hardware. In Proceedings of the 20th annual conference on
Computer graphics and interactive techniques, pages 65–72. ACM, 1993.
Michael Grabner, Helmut Grabner, and Horst Bischof. Fast approximated
sift. In Computer Vision–ACCV 2006, pages 918–927. Springer, 2006.
Raphael Grasset, Alessandro Mulloni, Mark Billinghurst, and Dieter Schmalstieg.
Navigation techniques in augmented and mixed reality: Crossing the
virtuality continuum. In Handbook of Augmented Reality, pages 379–407.
Springer, 2011.
Jens Grubert, Tobias Langlotz, and Raphaël Grasset. Augmented reality
browser survey. Technical Report 1101, Institute for Computer Graphics
and Vision, University of Technology Graz, Technical Report, 2011.
Chris Gunn, Matthew Hutchins, Matt Adcock, and Rhys Hawkins. Transworld
haptic collaboration. In ACM SIGGRAPH 2003 Sketches & Applications,
pages 1–1. ACM, 2003.
References 251
Taejin Ha, Woontack Woo, Youngho Lee, Junhun Lee, Jeha Ryu, Hankyun
Choi, and Kwanheng Lee. Artalet: tangible user interface based immersive
augmented reality authoring tool for digilog book. In Ubiquitous Virtual
Reality (ISUVR), 2010 International Symposium on, pages 40–43. IEEE,
2010.
Taejin Ha, Mark Billinghurst, and Woontack Woo. An interactive 3d movement
path manipulation method in an augmented reality environment. Interacting
with Computers, 24(1):10–24, 2012.
Michael Haller, Erwin Stauder, and Juergen Zauner. Amire-es: Authoring
mixed reality once, run it anywhere. In Proceedings of the 11th International
Conference on Human-Computer Interaction (HCII), volume 2005, 2005.
Chris Harrison, Hrvoje Benko, and Andrew D. Wilson. Omnitouch: wearable
multitouch interaction everywhere. In Proceedings of the 24th annual ACM
symposium on User interface software and technology, pages 441–450. ACM,
2011.
Alexander G. Hauptmann. Speech and gestures for graphic image manipulation.
ACM SIGCHI Bulletin, 20(SI):241–245, 1989.
Christian Heath and Paul Lu . Disembodied conduct: communication
through video in a multi-media o ce environment. In Proceedings of the
SIGCHI Conference on Human Factors in Computing Systems, pages 99–
103. ACM, 1991.
Gunther Heidemann, Ingo Bax, and Holger Bekel. Multimodal interaction
in an augmented reality scenario. In Proceedings of the 6th international
conference on Multimodal interfaces, pages 53–60. ACM, 2004.
Steven J. Henderson and Steven Feiner. Evaluating the beneﬁts of augmented
reality for task localization in maintenance of an armored personnel carrier
turret. In Mixed and Augmented Reality, 2009. ISMAR 2009. 8th IEEE
International Symposium on, pages 135–144. IEEE, 2009.
Peter Henry, Michael Krainin, Evan Herbst, Xiaofeng Ren, and Dieter Fox.
Rgb-d mapping: Using depth cameras for dense 3d modeling of indoor
environments. In In the 12th International Symposium on Experimental
Robotics (ISER, 2010.
Anders Henrysson and Mark Ollila. Umar: Ubiquitous mobile augmented
reality. In Proceedings of the 3rd international conference on Mobile and
ubiquitous multimedia, pages 41–45. ACM, 2004.
Anders Henrysson, Mark Ollila, and Mark Billinghurst. Mobile phone based
ar scene assembly. In Proceedings of the 4th international conference on
Mobile and ubiquitous multimedia, pages 95–102. ACM, 2005.
252 References
Otmar Hilliges, David Kim, Shahram Izadi, Malte Weiss, and Andrew Wilson.
Holodesk: direct 3d interactions with a situated see-through display. In
Proceedings of the SIGCHI Conference on Human Factors in Computing
Systems, pages 2421–2430. ACM, 2012.
Deborah Hix, Joseph L. Gabbard, J. Edward Swan, Mark A. Livingston,
T. H. Hollerer, Simon J. Julier, Yohan Baillot, Dennis Brown, et al. A
cost-e ective usability evaluation progression for novel interactive systems.
In System Sciences, 2004. Proceedings of the 37th Annual Hawaii International
Conference on, pages 10–pp. IEEE, 2004.
Steve Hodges, Lyndsay Williams, Emma Berry, Shahram Izadi, James Srinivasan,
Alex Butler, Gavin Smyth, Narinder Kapur, and Ken Wood. Sensecam:
A retrospective memory aid. In UbiComp 2006: Ubiquitous Computing,
pages 177–193. Springer, 2006.
H. G. Ho mann. Physically touching virtual objects using tactile augmentation
enhances the realism of virtual environments. In Virtual Reality
Annual International Symposium, 1998. Proceedings., IEEE 1998, pages
59–63. IEEE, 1998.
Tobias Höllerer, Steven Feiner, Tachio Terauchi, Gus Rashid, and Drexel Hallaway.
Exploring mars: developing indoor and outdoor user interfaces to a
mobile augmented reality system. Computers & Graphics, 23(6):779–785,
1999.
Tobias Höllerer, Jason Wither, and Stephen DiVerdi. “anywhere augmentation”:
Towards mobile augmented reality in unprepared environments.
In Location Based Services and TeleCartography, pages 393–416. Springer,
2007.
Jason Hong. Considering privacy issues in the context of google glass. Commun.
ACM, 56(11):10–11, 2013.
Juan Pablo Hourcade. Interaction design and children. Foundations and
Trends in Human-Computer Interaction, 1(4):277–392, 2008.
Xinda Hu and Hong Hua. Design of an optical see-through multi-focal-plane
stereoscopic 3d display using freeform prisms. In Frontiers in Optics, pages
FTh1F–2. Optical Society of America, 2012.
Hong Hua, Leonard D Brown, Chunyu Gao, and Narendra Ahuja. A new
collaborative infrastructure: Scape. In Virtual Reality, 2003. Proceedings.
IEEE, pages 171–179. IEEE, 2003.
References 253
Yetao Huang, Dongdong Weng, Yue Liu, and Yongtian Wang. Key issues
of wide-area tracking system for multi-user augmented reality adventure
game. In Image and Graphics, 2009. ICIG’09. Fifth International Conference
on, pages 646–651. IEEE, 2009.
Manuel Huber, Daniel Pustka, Peter Keitler, Florian Echtler, and Gudrun
Klinker. A system architecture for ubiquitous tracking environments. In
Proceedings of the 2007 6th IEEE and ACM International Symposium on
Mixed and Augmented Reality, pages 1–4. IEEE Computer Society, 2007.
Olivier Hugues, Philippe Fuchs, and Olivier Nannipieri. New augmented reality
taxonomy: Technologies and features of augmented environment. In
Handbook of augmented reality, pages 47–63. Springer, 2011.
Masahiko Inami, Naoki Kawakami, Dairoku Sekiguchi, Yasuyuki Yanagida,
Taro Maeda, and Susumu Tachi. Visuo-haptic display using head-mounted
projector. In Virtual Reality, 2000. Proceedings. IEEE, pages 233–240.
IEEE, 2000.
Piotr Indyk and Rajeev Motwani. Approximate nearest neighbors: towards
removing the curse of dimensionality. In Proceedings of the thirtieth annual
ACM symposium on Theory of computing, pages 604–613. ACM, 1998.
Sylvia Irawati, Scott Green, Mark Billinghurst, Andreas Duenser, and Heedong
Ko. An evaluation of an augmented reality multimodal interface using
speech and paddle gestures. In Advances in Artiﬁcial Reality and TeleExistence,
pages 272–283. Springer, 2006.
Hiroshi Ishii and Brygg Ullmer. Tangible bits: towards seamless interfaces
between people, bits and atoms. In Proceedings of the ACM SIGCHI Conference
on Human factors in computing systems, pages 234–241. ACM,
1997.
Shahram Izadi, David Kim, Otmar Hilliges, David Molyneaux, Richard Newcombe,
Pushmeet Kohli, Jamie Shotton, Steve Hodges, Dustin Freeman,
Andrew Davison, et al. Kinectfusion: real-time 3d reconstruction and interaction
using a moving depth camera. In Proceedings of the 24th annual
ACM symposium on User interface software and technology, pages 559–568.
ACM, 2011.
Seokhee Jeon and Seungmoon Choi. Haptic augmented reality: Taxonomy and
an example of sti ness modulation. Presence: Teleoperators and Virtual
Environments, 18(5):387–408, 2009.
Simon Julier, Marco Lanzagorta, Yohan Baillot, Lawrence Rosenblum, Steven
Feiner, Tobias Hollerer, and Sabrina Sestito. Information ﬁltering for mobile
augmented reality. In Augmented Reality, 2000.(ISAR 2000). Proceedings.
IEEE and ACM International Symposium on, pages 3–11. IEEE, 2000.
254 References
Ed Kaiser, Alex Olwal, David McGee, Hrvoje Benko, Andrea Corradini, Xiaoguang
Li, Phil Cohen, and Steven Feiner. Mutual disambiguation of 3d
multimodal interaction in augmented and virtual reality. In Proceedings
of the 5th international conference on Multimodal interfaces, pages 12–19.
ACM, 2003.
Roy Kalawsky. The science of virtual reality and virtual environments.
Addison-Wesley Longman Publishing Co., Inc., 1993.
Peter Kán and Hannes Kaufmann. Di erential irradiance caching for fast
high-quality light transport between virtual and real worlds. In Mixed
and Augmented Reality (ISMAR), 2013 IEEE International Symposium on,
pages 133–141. IEEE, 2013.
Anantha Kancherla, Mark Singer, and Jannick Rolland. Calibrating seethrough
head-mounted displays. Technical Report TR95-034, Department
of Computer Science, University of North Carolina at Chapel Hill, 1996.
Kenji Kansaku, Naoki Hata, and Kouji Takano. My thoughts through a
robot’s eyes: An augmented reality-brain–machine interface. Neuroscience
research, 66(2):219–222, 2010.
Hirokazu Kato and Mark Billinghurst. Marker tracking and hmd calibration
for a video-based augmented reality conferencing system. In Augmented
Reality, 1999.(IWAR’99) Proceedings. 2nd IEEE and ACM International
Workshop on, pages 85–94. IEEE, 1999.
Hirokazu Kato, Mark Billinghurst, Ivan Poupyrev, Kenji Imamoto, and Keihachiro
Tachibana. Virtual object manipulation on a table-top ar environment.
In Augmented Reality, 2000.(ISAR 2000). Proceedings. IEEE and
ACM International Symposium on, pages 111–119. Ieee, 2000.
Yan Ke and Rahul Sukthankar. Pca-sift: A more distinctive representation for
local image descriptors. In Computer Vision and Pattern Recognition, 2004.
CVPR 2004. Proceedings of the 2004 IEEE Computer Society Conference
on, volume 2, pages II–506. IEEE, 2004.
Lucinda Kerawalla, Rosemary Luckin, Simon Seljeﬂot, and Adrian Woolard.
“making it real”: exploring the potential of augmented reality for teaching
primary school science. Virtual Reality, 10(3-4):163–174, 2006.
Ryugo Kijima and Takeo Ojika. Transition between virtual environment and
workstation environment with projective head mounted display. In Virtual
Reality Annual International Symposium, 1997., IEEE 1997, pages 130–
137. IEEE, 1997.
References 255
Hyemi Kim, Gun A. Lee, Ungyeon Yang, Taejin Kwak, and Ki-Hong Kim.
Dual autostereoscopic display platform for multi-user collaboration with
natural interaction. Etri Journal, 34(3):466–469, 2012.
Sehwan Kim, Youngjung Suh, Youngho Lee, and Woontack Woo. Toward
ubiquitous vr: When vr meets ubicomp. Proc of ISUVR, pages 1–4, 2006.
Kiyoshi Kiyokawa, Haruo Takemura, and Naokazu Yokoya. A collaboration
support technique by integrating a shared virtual reality and a shared augmented
reality. In Systems, Man, and Cybernetics, 1999. IEEE SMC’99
Conference Proceedings. 1999 IEEE International Conference on, volume 6,
pages 48–53. IEEE, 1999.
Kiyoshi Kiyokawa, Haruo Takemura, and Naokazu Yokoya. Seamlessdesign
for 3d object creation. IEEE multimedia, 7(1):22–33, 2000.
Kiyoshi Kiyokawa, Yoshinori Kurata, and Hiroyuki Ohno. An optical seethrough
display for mutual occlusion with a real-time stereovision system.
Computers & Graphics, 25(5):765–779, 2001.
Georg Klein and Tom Drummond. Sensor fusion and occlusion reﬁnement
for tablet-based ar. In Mixed and Augmented Reality, 2004. ISMAR 2004.
Third IEEE and ACM International Symposium on, pages 38–47. IEEE,
2004.
Georg Klein and David Murray. Parallel tracking and mapping for small
ar workspaces. In Mixed and Augmented Reality, 2007. ISMAR 2007. 6th
IEEE and ACM International Symposium on, pages 225–234. IEEE, 2007.
Georg Klein and David W. Murray. Simulating low-cost cameras for augmented
reality compositing. Visualization and Computer Graphics, IEEE
Transactions on, 16(3):369–380, 2010.
Mathias Kolsch, Ryan Bane, Tobias Hollerer, and Matthew Turk. Multimodal
interaction with a wearable augmented reality system. Computer Graphics
and Applications, IEEE, 26(3):62–71, 2006.
Robert E. Kraut, Mark D. Miller, and Jane Siegel. Collaboration in performance
of physical tasks: E ects on outcomes and communication. In
Proceedings of the 1996 ACM conference on Computer supported cooperative
work, pages 57–66. ACM, 1996.
David M. Krum, Evan A. Suma, and Mark Bolas. Augmented reality using
personal projection and retroreﬂection. Personal and Ubiquitous Computing,
16(1):17–26, 2012.
256 References
Oliver Kutter, André Aichert, Christoph Bichlmeier, Jörg Traub, SM Heining,
B Ockert, Ekkehard Euler, and Nassir Navab. Real-time volume rendering
for high quality visualization in augmented reality. In International
Workshop on Augmented environments for Medical Imaging including Augmented
Reality in Computer-aided Surgery (AMI-ARCS 2008), New York,
USA, 2008.
Hideaki Kuzuoka. Spatial workspace collaboration: a sharedview video support
system for remote collaboration capability. In Proceedings of the
SIGCHI conference on Human factors in computing systems, pages 533–
540. ACM, 1992.
Peter Lang, Albert Kusej, Axel Pinz, and Georg Brasseur. Inertial tracking for
mobile augmented reality. In Instrumentation and Measurement Technology
Conference, 2002. IMTC/2002. Proceedings of the 19th IEEE, volume 2,
pages 1583–1587. IEEE, 2002.
Joseph J. LaViola Jr. Whole-hand and speech input in virtual environments.
Master’s thesis, Brown University, 1999.
Gun Lee, Gerard Kim, and Mark Billinghurst. Interaction design for tangible
augmented reality applications. Emerging Technologies of Augmented
Reality: Interfaces and Design, pages 261–282, 2007.
Gun A. Lee and Mark Billinghurst. A component based framework for mobile
outdoor ar applications. In Proceedings of the 12th ACM SIGGRAPH International
Conference on Virtual-Reality Continuum and Its Applications
in Industry, pages 207–210. ACM, 2013.
Gun A. Lee, Gerard Jounghyun Kim, and Chan-Mo Park. Modeling virtual
object behavior within virtual environment. In Proceedings of the ACM
symposium on Virtual reality software and technology, pages 41–48. ACM,
2002.
Gun A. Lee, Mark Billinghurst, and Gerard Jounghyun Kim. Occlusion based
interaction methods for tangible augmented reality environments. In Proceedings
of the 2004 ACM SIGGRAPH international conference on Virtual
Reality continuum and its applications in industry, pages 419–426. ACM,
2004a.
Gun A. Lee, Claudia Nelles, Mark Billinghurst, and Gerard Jounghyun Kim.
Immersive authoring of tangible augmented reality applications. In Proceedings
of the 3rd IEEE/ACM international Symposium on Mixed and
Augmented Reality, pages 172–181. IEEE Computer Society, 2004b.
Gun A. Lee, Gerard J. Kim, and Mark Billinghurst. Immersive authoring:
What you experience is what you get (wyxiwyg). Communications of the
ACM, 48(7):76–81, 2005.
References 257
Gun A. Lee, Ungyeon Yang, and Wookho Son. Layered multiple displays for
immersive and interactive digital contents. In Entertainment ComputingICEC
2006, pages 123–134. Springer, 2006.
Gun A. Lee, Hyun Kang, and Wookho Son. Mirage: A touch screen based
mixed reality interface for space planning applications. In Virtual Reality
Conference, 2008. VR’08. IEEE, pages 273–274. IEEE, 2008a.
Gun A. Lee, Ungyeon Yang, Wookho Son, Yongwan Kim, Dongsik Jo, KiHong
Kim, and Jin Sung Choi. Virtual reality content-based training for
spray painting tasks in the shipbuilding industry. ETRI Journal, 32(5):
695–703, 2010a.
Gun A. Lee, Andreas Dunser, Seungwon Kim, and Mark Billinghurst.
Cityviewar: A mobile outdoor ar application for city visualization. In Mixed
and Augmented Reality (ISMAR-AMH), 2012 IEEE International Symposium
on, pages 57–64. IEEE, 2012.
Gun A. Lee, Andreas Dünser, Alaeddin Nassani, and Mark Billinghurst.
Antarcticar: An outdoor ar experience of a virtual tour to antarctica.
In Mixed and Augmented Reality-Arts, Media, and Humanities (ISMARAMH),
2013 IEEE International Symposium on, pages 29–38. IEEE, 2013a.
Jae Yeol Lee, Gue Won Rhee, and Dong Woo Seo. Hand gesture-based tangible
interactions for manipulating virtual objects in a mixed reality environment.
The International Journal of Advanced Manufacturing Technology,
51(9-12):1069–1082, 2010b.
Minkyung Lee, Richard Green, and Mark Billinghurst. 3d natural hand interaction
for ar applications. In Image and Vision Computing New Zealand,
2008. IVCNZ 2008. 23rd International Conference, pages 1–6. IEEE, 2008b.
Minkyung Lee, Mark Billinghurst, Woonhyuk Baek, Richard Green, and
Woontack Woo. A usability study of multimodal input in an augmented
reality environment. Virtual Reality, 17(4):293–305, 2013b.
Taehee Lee and Tobias Hollerer. Handy ar: Markerless inspection of augmented
reality objects using ﬁngertip tracking. In Wearable Computers,
2007 11th IEEE International Symposium on, pages 83–90. IEEE, 2007.
Alexander Lenhardt and Helge Ritter. An augmented-reality based braincomputer
interface for robot control. In Neural Information Processing.
Models and Applications, pages 58–65. Springer, 2010.
Vincent Lepetit and Pascal Fua. Keypoint recognition using randomized trees.
Pattern Analysis and Machine Intelligence, IEEE Transactions on, 28(9):
1465–1479, 2006.
258 References
Stefan Leutenegger, Margarita Chli, and Roland Yves Siegwart. Brisk: Binary
robust invariant scalable keypoints. In Computer Vision (ICCV), 2011
IEEE International Conference on, pages 2548–2555. IEEE, 2011.
Michael Emmanuel Leventon. A registration, tracking, and visualization system
for image-guided surgery. PhD thesis, Massachusetts Institute of Technology,
1997.
Robert W. Lindeman, John L Sibert, and James K. Hahn. Towards usable vr:
an empirical study of user interfaces for immersive virtual environments.
In Proceedings of the SIGCHI conference on Human Factors in Computing
Systems, pages 64–71. ACM, 1999.
Robert W. Lindeman, Haruo Noma, and Paulo Goncalves de Barros. Hearthrough
and mic-through augmented reality: Using bone conduction to display
spatialized audio. In Proceedings of the 2007 6th IEEE and ACM International
Symposium on Mixed and Augmented Reality, pages 1–4. IEEE
Computer Society, 2007.
Robert W. Lindeman, Gun Lee, Leigh Beattie, Hannes Gamper, Rahul Pathinarupothi,
and Aswin Akhilesh. Geoboids: A mobile ar application for exergaming.
In Mixed and Augmented Reality (ISMAR-AMH), 2012 IEEE
International Symposium on, pages 93–94. IEEE, 2012.
Andrew R. Lingley, Muhammad Ali, Y. Liao, R. Mirjalili, M. Klonner,
M. Sopanen, S. Suihkonen, T. Shen, B. P. Otis, H. Lipsanen, et al. A
single-pixel wireless contact lens display. Journal of Micromechanics and
Microengineering, 21(12):125014, 2011.
Sheng Liu, Dewen Cheng, and Hong Hua. An optical see-through head
mounted display with addressable focal planes. In Mixed and Augmented
Reality, 2008. ISMAR 2008. 7th IEEE/ACM International Symposium on,
pages 33–42. IEEE, 2008.
David G. Lowe. Object recognition from local scale-invariant features. In
Computer vision, 1999. The proceedings of the seventh IEEE international
conference on, volume 2, pages 1150–1157. Ieee, 1999.
David G. Lowe. Distinctive image features from scale-invariant keypoints.
International journal of computer vision, 60(2):91–110, 2004.
Blair MacIntyre. Authoring 3d mixed reality experiences: Managing the relationship
between the physical and virtual worlds. At ACM SIGGRAPH
and Eurographics Campﬁre: Production Process of 3D Computer Graphics
Applications-Structures, Roles and Tools, Snowbird, UT, pages 1–5, 2002.
References 259
Blair MacIntyre, Maribeth Gandy, Steven Dow, and Jay David Bolter. Dart:
a toolkit for rapid design exploration of augmented reality experiences. In
Proceedings of the 17th annual ACM symposium on User interface software
and technology, pages 197–206. ACM, 2004.
Asa MacWilliams, Thomas Reicher, Gudrun Klinker, and Bernd Bruegge.
Design patterns for augmented reality systems. In MIXER, 2004.
Andrew Maimone and Henry Fuchs. Computational augmented reality eyeglasses.
In Mixed and Augmented Reality (ISMAR), 2013 IEEE International
Symposium on, pages 29–38. IEEE, 2013.
Elmar Mair, Gregory D. Hager, Darius Burschka, Michael Suppa, and Gerhard
Hirzinger. Adaptive and generic corner detection based on the accelerated
segment test. In Computer Vision–ECCV 2010, pages 183–196. Springer,
2010.
J. Mandeville, J. Davidson, D. Campbell, A. Dahl, P. Schwartz, and T. Furness.
A shared virtual environment for architectural design review. In CVE
‘96 Workshop Proceedings, pages 19–20, 1996.
Steve Mann. Mediated reality. Technical Report MIT-ML Percom TR-260,
University of Toronto, 1994.
Steve Mann. Mediated reality with implementations for everyday life. Presence
Connect, August, 6, 2002.
Héctor Martínez, Danai Skournetou, Jenni Hyppölä, Seppo Laukkanen, and
Antti Heikkilä. Drivers and bottlenecks in the adoption of augmented reality
applications. Journal of Multimedia Theory and Application, 2(1),
2014.
Andrea H. Mason, Masuma A. Walji, Elaine J. Lee, and Christine L. MacKenzie.
Reaching movements to augmented and graphic objects in virtual environments.
In Proceedings of the SIGCHI conference on Human factors in
computing systems, pages 426–433. ACM, 2001.
John C. McCarthy and Andrew F. Monk. Measuring the quality of computermediated
communication. Behaviour & Information Technology, 13(5):311–
319, 1994.
Chris McDonald and Gerhard Roth. Replacing a mouse with hand gesture in a
plane-based augmented reality system. Technical report, National Research
Council Canada, 2003.
Erin McGarrity and Mihran Tuceryan. A method for calibrating see-through
head-mounted displays for ar. In Augmented Reality, 1999.(IWAR’99) Proceedings.
2nd IEEE and ACM International Workshop on, pages 75–84.
IEEE, 1999.
260 References
David McNeill. Hand and mind: What gestures reveal about thought. University
of Chicago Press, 1992.
Paul Milgram and Fumio Kishino. A taxonomy of mixed reality visual displays.
IEICE TRANSACTIONS on Information and Systems, 77(12):1321–
1329, 1994.
Paul Milgram, Shumin Zhai, David Drascic, and Julius J. Grodski. Applications
of augmented reality for human-robot communication. In Intelligent
Robots and Systems’ 93, IROS’93. Proceedings of the 1993 IEEE/RSJ International
Conference on, volume 3, pages 1467–1472. IEEE, 1993.
Pranav Mistry and Pattie Maes. Sixthsense: a wearable gestural interface. In
ACM SIGGRAPH ASIA 2009 Sketches, page 11. ACM, 2009.
Antonija Mitrovic. Fifteen years of constraint-based tutors: what we have
achieved and where we are going. User Modeling and User-Adapted Interaction,
22(1-2):39–72, 2012.
Antonija Mitrovic, Brent Martin, Pramuditha Suraweera, Konstantin Zakharov,
Nancy Milik, Jay Holland, and Nicholas McGuigan. Aspire: an
authoring system and deployment environment for constraint-based tutors.
International Journal of Artiﬁcial Intelligence in Education, 19(2):155–188,
2009.
D. Mogilev, Kiyoshi Kiyokawa, Mark Billinghurst, and J. Pair. Ar pad: An
interface for face-to-face ar collaboration. In CHI’02 Extended Abstracts on
Human Factors in Computing Systems, pages 654–655. ACM, 2002.
Mathias Mohring, Christian Lessig, and Oliver Bimber. Video see-through ar
on consumer cell-phones. In Proceedings of the 3rd IEEE/ACM International
Symposium on Mixed and Augmented Reality, pages 252–253. IEEE
Computer Society, 2004.
Takuji Narumi, Shinya Nishizaka, Takashi Kajinami, Tomohiro Tanikawa,
and Michitaka Hirose. Meta cookie+: an illusion-based gustatory display.
In Virtual and Mixed Reality-New Trends, pages 260–269. Springer, 2011.
Wolfgang Narzt, Gustav Pomberger, Alois Ferscha, Dieter Kolb, Reiner
Müller, Jan Wieghardt, Horst Hörtner, and Christopher Lindinger. Augmented
reality navigation systems. Universal Access in the Information
Society, 4(3):177–187, 2006.
Nassir Navab, A Bani-Kashemi, and Matthias Mitschke. Merging visible and
invisible: Two camera-augmented mobile c-arm (camc) applications. In
Augmented Reality, 1999.(IWAR’99) Proceedings. 2nd IEEE and ACM International
Workshop on, pages 134–141. IEEE, 1999.
References 261
Nassir Navab, Marco Feuerstein, and Christoph Bichlmeier. Laparoscopic virtual
mirror new interaction paradigm for monitor based augmented reality.
In Virtual Reality Conference, 2007. VR’07. IEEE, pages 43–50. IEEE,
2007.
Ulrich Neumann and Youngkwan Cho. A self-tracking augmented reality
system. In Proceedings of the ACM Symposium on Virtual Reality Software
and Technology, pages 109–115, 1996.
Ulrich Neumann and Jun Park. Extendible object-centric tracking for augmented
reality. In Virtual Reality Annual International Symposium, 1998.
Proceedings., IEEE 1998, pages 148–155. IEEE, 1998.
Richard A. Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux,
David Kim, Andrew J Davison, Pushmeet Kohi, Jamie Shotton, Steve
Hodges, and Andrew Fitzgibbon. Kinectfusion: Real-time dense surface
mapping and tracking. In Mixed and augmented reality (ISMAR), 2011
10th IEEE international symposium on, pages 127–136. IEEE, 2011.
Joseph Newman, Martin Wagner, Martin Bauer, Asa MacWilliams, Thomas
Pintaric, Dagmar Beyer, Daniel Pustka, Franz Strasser, Dieter Schmalstieg,
and Gudrun Klinker. Ubiquitous tracking for augmented reality. In Mixed
and Augmented Reality, 2004. ISMAR 2004. Third IEEE and ACM International
Symposium on, pages 192–201. IEEE, 2004.
Joseph Newman, Gerhard Schall, István Barakonyi, Andreas Schürzinger, and
Dieter Schmalstieg. Wide-area tracking tools for augmented reality. na,
2006.
Trond Nilsen and Julian Looser. Tankwar-tabletop war gaming in augmented
reality. In 2nd International Workshop on Pervasive Gaming Applications,
PerGames, volume 5, 2005.
Susanna Nilsson and Björn Johansson. Acceptance of augmented reality instructions
in a real work setting. In CHI’08 extended abstracts on Human
factors in computing systems, pages 2025–2032. ACM, 2008.
Mark A. Nixon, Bruce C. McCallum, W. Richard Fright, and N. Brent Price.
The e ects of metals and interfering ﬁelds on electromagnetic trackers.
Presence: Teleoperators and Virtual Environments, 7(2):204–218, 1998.
Donald A. Norman. The psychology of everyday things. Basic books, 1988.
Jean-Marie Normand, Myriam Servières, and Guillaume Moreau. A new typology
of augmented reality applications. In Proceedings of the 3rd Augmented
Human International Conference, page 18. ACM, 2012.
262 References
Timothy Nyerges, T. J. Moore, Robb Montejano, and Marcie Compton. Developing
and using interaction coding systems for studying groupware use.
Human-Computer Interaction, 13(2):127–165, 1998.
Christoph Oberhofer, Jens Grubert, and Gerhard Reitmayr. Natural feature
tracking in javascript. In VR, pages 113–114, 2012.
Brid O’Conaill and Steve Whittaker. Characterizing, predicting, and measuring
video-mediated communication: A conversational approach. Videomediated
communication, pages 107–132, 1997.
Ohan Oda, Levi J. Lister, Sean White, and Steven Feiner. Developing an augmented
reality racing game. In Proceedings of the 2nd international conference
on INtelligent TEchnologies for interactive enterTAINment, page 2.
ICST (Institute for Computer Sciences, Social-Informatics and Telecommunications
Engineering), 2008.
Yoosoo Oh and Woontack Woo. A uniﬁed application service model for ubihome
by exploiting intelligent context-awareness. In Ubiquitous Computing
Systems, pages 192–202. Springer, 2005.
Yoosoo Oh, Choonsung Shin, Woojin Jung, and Woontack Woo. The ubitv
application for a family in ubihome. In 2nd Ubiquitous Home workshop,
pages 23–32, 2005.
Toshikazu Ohshima, Kiyohide Satoh, Hiroyuki Yamamoto, and Hideyuki
Tamura. Ar 2 hockey: A case study of collaborative augmented reality.
In Virtual Reality Annual International Symposium, 1998. Proceedings.,
IEEE 1998, pages 268–275. IEEE, 1998.
Thomas Olsson, Else Lagerstam, Tuula Kärkkäinen, and Kaisa VäänänenVainio-Mattila.
Expected user experience of mobile augmented reality services:
a user study in the context of shopping centres. Personal and ubiquitous
computing, 17(2):287–304, 2013.
Alex Olwal, Hrvoje Benko, and Steven Feiner. Senseshapes: Using statistical
geometry for object selection in a multimodal augmented reality system.
In Proceedings of the 2nd IEEE/ACM International Symposium on Mixed
and Augmented Reality, page 300. IEEE Computer Society, 2003.
Claire O’Malley, Steve Langton, Anne Anderson, Gwyneth Doherty-Sneddon,
and Vicki Bruce. Comparison of face-to-face and video-mediated interaction.
Interacting with computers, 8(2):177–192, 1996.
Mustafa Ozuysal, Michael Calonder, Vincent Lepetit, and Pascal Fua. Fast
keypoint recognition using random ferns. Pattern Analysis and Machine
Intelligence, IEEE Transactions on, 32(3):448–461, 2010.
References 263
Jagdish Pandey, Yu-Te Liao, Andrew Lingley, Ramin Mirjalili, Babak Parviz,
and B Otis. A fully integrated rf-powered contact lens with a single element
display. Biomedical Circuits and Systems, IEEE Transactions on, 4(6):454–
461, 2010.
Babak A. Parviz. For your eye only. Spectrum, IEEE, 46(9):36–41, 2009.
Wayne Piekarski and Bruce Thomas. Arquake: the outdoor augmented reality
gaming system. Communications of the ACM, 45(1):36–38, 2002a.
Wayne Piekarski and Bruce H. Thomas. Using artoolkit for 3d hand position
tracking in mobile outdoor environments. In Augmented Reality Toolkit,
The ﬁrst IEEE International Workshop, pages 2–pp. IEEE, 2002b.
Wayne Piekarski and Bruce H Thomas. Augmented reality user interfaces and
techniques for outdoor modelling. In Proceedings of the 2003 symposium
on Interactive 3D graphics, pages 225–226. ACM, 2003.
Wayne Piekarski, Ben Avery, Bruce H. Thomas, and Pierre Malbezin. Hybrid
indoor and outdoor tracking for mobile 3d mixed reality. In Proceedings
of the 2nd IEEE/ACM International Symposium on Mixed and Augmented
Reality, page 266. IEEE Computer Society, 2003.
Thammathip Piumsomboon, Adrian Clark, Mark Billinghurst, and Andy
Cockburn. User-deﬁned gestures for augmented reality. In HumanComputer
Interaction–INTERACT 2013, pages 282–299. Springer, 2013.
Thammathip Piumsomboon, David Altimira, Hyungon Kim, Adrian Clark,
Gun Lee, and Mark Billinghurst. Grasp-shell vs gesture-speech: A comparison
of direct and indirect natural interaction techniques in augmented
reality. In Mixed and Augmented Reality (ISMAR), 2014 IEEE International
Symposium on, pages 73–82. IEEE, 2014.
Timothy Poston and Luis Serra. The virtual workbench: dextrous vr. In
Proceedings of the conference on Virtual reality software and technology,
pages 111–121. World Scientiﬁc Publishing Co., Inc., 1994.
Ivan Poupyrev, Mark Billinghurst, Suzanne Weghorst, and Tadao Ichikawa.
The go-go interaction technique: non-linear mapping for direct manipulation
in vr. In Proceedings of the 9th annual ACM symposium on User
interface software and technology, pages 79–80. ACM, 1996.
Ivan Poupyrev, Numada Tomokazu, and Suzanne Weghorst. Virtual notepad:
handwriting in immersive vr. In Virtual Reality Annual International Symposium,
1998. Proceedings., IEEE 1998, pages 126–132. IEEE, 1998.
Ivan Poupyrev, Desney S. Tan, Mark Billinghurst, Hirokazu Kato, Holger
Regenbrecht, and Nobuji Tetsutani. Developing a generic augmented-reality
interface. Computer, 35(3):44–50, 2002.
264 References
Lulian Radu and Blair MacIntyre. Using children’s developmental psychology
to guide augmented-reality design and usability. In Mixed and Augmented
Reality (ISMAR), 2012 IEEE International Symposium on, pages 227–236.
IEEE, 2012.
Che Mohd Yuso Rasimah, Azlina Ahmad, and Halimah Badioze Zaman.
Evaluation of user acceptance of mixed reality technology. Australasian
Journal of Educational Technology, 27(8):1369–1387, 2011.
Ramesh Raskar, Greg Welch, and Wei-Chao Chen. Table-top spatiallyaugmented
realty: bringing physical models to life with projected imagery.
In Augmented Reality, 1999.(IWAR’99) Proceedings. 2nd IEEE and ACM
International Workshop on, pages 64–71. IEEE, 1999.
Ramesh Raskar, Greg Welch, Kok-Lim Low, and Deepak Bandyopadhyay.
Shader lamps: Animating real objects with image-based illumination. In
StevenJ. Gortler and Karol Myszkowski, editors, Rendering Techniques
2001, Eurographics, pages 89–102. Springer Vienna, 2001. ISBN 978-3-211-
83709-2. . URL http://dx.doi.org/10.1007/978-3-7091-6242-2_9.
Ramesh Raskar, Jeroen van Baar, Paul Beardsley, Thomas Willwacher, Srinivas
Rao, and Clifton Forlines. ilamps: Geometrically aware and selfconﬁguring
projectors. ACM Trans. Graph., 22(3):809–818, 2003.
Malinda Rauhala, Ann-Soﬁe Gunnarsson, and Anders Henrysson. A novel
interface to sensor networks using handheld augmented reality. In Proceedings
of the 8th conference on Human-computer interaction with mobile
devices and services, pages 145–148. ACM, 2006.
Thomas Reicher, Asa MacWilliams, and Bernd Bruegge. Towards a system
of patterns for augmented reality systems. In International Workshop on
Software Technology for Augmented Reality Systems (STARS 2003), 2003.
Gerhard Reitmayr and Tom W. Drummond. Going out: robust model-based
tracking for outdoor augmented reality. In Mixed and Augmented Reality,
2006. ISMAR 2006. IEEE/ACM International Symposium on, pages 109–
118. IEEE, 2006.
Jun Rekimoto. Augmented reality using the 2d matrix code. Interactive
Systems and Software IV. Kindaikagaku-sha, pages 199–208, 1996a.
Jun Rekimoto. Transvision: A hand-held augmented reality system for collaborative
design. In Proc. Virtual Systems and Multimedia, pages 85–90,
1996b.
Jun Rekimoto. Matrix: A realtime object identiﬁcation and registration
method for augmented reality. In Computer Human Interaction, 1998.
Proceedings. 3rd Asia Paciﬁc, pages 63–68. IEEE, 1998.
References 265
Jun Rekimoto and Yuji Ayatsuka. Cybercode: designing augmented reality
environments with visual tags. In Proceedings of DARE 2000 on Designing
augmented reality environments, pages 1–10. ACM, 2000.
Jun Rekimoto and Katashi Nagao. The world through the computer: Computer
augmented interaction with real world environments. In Proceedings
of the 8th annual ACM symposium on User interface and software technology,
pages 29–36. ACM, 1995.
Miguel Ribo, Axel Pinz, and Anton L. Fuhrmann. A new optical tracking
system for virtual and augmented reality applications. In Instrumentation
and Measurement Technology Conference, 2001. IMTC 2001. Proceedings
of the 18th IEEE, volume 3, pages 1932–1936. IEEE, 2001.
Julie Rico and Stephen Brewster. Usable gestures for mobile interfaces: evaluating
social acceptability. In Proceedings of the SIGCHI Conference on
Human Factors in Computing Systems, pages 887–896. ACM, 2010.
Everett M. Rogers. Di usion of innovations. Simon and Schuster, 4 edition,
2010.
Jannick P. Rolland and Henry Fuchs. Optical versus video see-through headmounted
displays in medical visualization. Presence: Teleoperators and
Virtual Environments, 9(3):287–309, 2000.
Jannick P. Rolland, Richard L Holloway, and Henry Fuchs. Comparison of
optical and video see-through, head-mounted displays. In Photonics for
Industrial Applications, pages 293–307. International Society for Optics and
Photonics, 1995a.
Jannick P. Rolland, Jim Parsons, David Poizat, and Dennis Hancock. Conformal
optics for 3d visualization. In International Optical Design Conference,
pages 760–764. International Society for Optics and Photonics, 1998.
Jannick P. Rolland, Larry Davis, and Yohan Baillot. A survey of tracking
technology for virtual environments. Fundamentals of wearable computers
and augmented reality, 1:67–112, 2001.
J.P. Rolland, W. Gibson, and D. Arierly. Towards quantifying depth and size
perception as a function of viewing distance. Presence: Teleoperators and
Virtual Environments, 4:24–49, 1995b.
Paul L. Rosin. Measuring corner properties. Computer Vision and Image
Understanding, 73(2):291–307, 1999.
Edward Rosten and Tom Drummond. Machine learning for high-speed corner
detection. In Computer Vision–ECCV 2006, pages 430–443. Springer, 2006.
266 References
Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary Bradski. Orb: an
e cient alternative to sift or surf. In Computer Vision (ICCV), 2011 IEEE
International Conference on, pages 2564–2571. IEEE, 2011.
Pedro Santos, Thomas Gierlinger, Oliver Machui, and André Stork. The daylight
blocking optical stereo see-through hmd. In Proceedings of the 2008
workshop on Immersive projection technologies/Emerging display technologiges,
page 4. ACM, 2008.
Daniel Scharstein and Richard Szeliski. High-accuracy stereo depth maps
using structured light. In Computer Vision and Pattern Recognition, 2003.
Proceedings. 2003 IEEE Computer Society Conference on, volume 1, pages
I–195. IEEE, 2003.
Fabian Scheer and Stefan Müller. Indoor tracking for large area industrial
mixed reality. In ICAT/EGVE/EuroVR, pages 21–28, 2012.
Reinhold Scherer, Mike Chung, Johnathan Lyon, Willy Cheung, and Rajesh
PN Rao. Interaction with virtual and augmented reality environments
using non-invasive brain-computer interfacing. In 1st International Conference
on Applied Bionics and Biomechanics (October 2010), 2010.
Dieter Schmalstieg and Daniel Wagner. Experiences with handheld augmented
reality. In Mixed and Augmented Reality, 2007. ISMAR 2007. 6th
IEEE and ACM International Symposium on, pages 3–18. IEEE, 2007.
Dieter Schmalstieg and Daniel Wagner. Mobile phones as a platform for
augmented reality. In Proceedings of the IEEE VR 2008 Workshop on
Software Engineering and Architectures for Real Time Interactive Systems
(SEARIS), volume 1, page 3, 2009.
Dieter Schmalstieg, Anton Fuhrmann, Zsolt Szalavári, and Michael Gervautz.
"studierstube" - an environment for collaboration in augmented reality. In
CVE ’96 Workshop Proceedings, 1996.
Dieter Schmalstieg, Anton Fuhrmann, and Gerd Hesina. Bridging multiple
user interface dimensions with augmented reality. In Augmented Reality,
2000.(ISAR 2000). Proceedings. IEEE and ACM International Symposium
on, pages 20–29. IEEE, 2000.
Dieter Schmalstieg, Anton Fuhrmann, Gerd Hesina, Zsolt Szalavári, L Miguel
Encarnaçao, Michael Gervautz, and Werner Purgathofer. The studierstube
augmented reality project. Presence: Teleoperators and Virtual Environments,
11(1):33–54, 2002.
References 267
Johannes Schöning, Michael Rohs, Sven Kratz, Markus Löchtefeld, and Antonio
Krüger. Map torchlight: a mobile augmented reality camera projector
unit. In CHI’09 Extended Abstracts on Human Factors in Computing Systems,
pages 3841–3846. ACM, 2009.
Abigail J Sellen. Remote conversations: The e ects of mediating talk with
technology. Human-computer interaction, 10(4):401–444, 1995.
Marcos Serrano, Barrett M Ens, and Pourang P. Irani. Exploring the use
of hand-to-face input for interacting with head-worn displays. In Proceedings
of the 32nd annual ACM conference on Human factors in computing
systems, pages 3181–3190. ACM, 2014.
Gilles Simon, Andrew W. Fitzgibbon, and Andrew Zisserman. Markerless
tracking using planar structures in the scene. In Augmented Reality,
2000.(ISAR 2000). Proceedings. IEEE and ACM International Symposium
on, pages 120–128. IEEE, 2000.
Asim Smailagic and Daniel P. Siewiorek. The cmu mobile computers: A new
generation of computer systems. In Compcon Spring ’94, Digest of Papers.,
pages 467–473. IEEE, 1994.
John M. Smart, Jamasi Cascio, and Jerry Pa endorf. Metaverse roadmap
overview, 2007.
Thad Starner. Human-powered wearable computing. IBM systems Journal,
35(3.4):618–629, 1996.
Thad Starner, Steve Mann, Bradley Rhodes, Je rey Levine, Jennifer Healey,
Dana Kirsch, Rosalind W. Picard, and Alex Pentland. Augmented reality
through wearable computing. Presence: Teleoperators and Virtual Environments,
6(4):386–398, 1997.
Neal Stephenson. Snow crash. Bantam Books, 1992.
Youngjung Suh, Youngmin Park, Hyoseok Yoon, Yoonje Chang, and Woontack
Woo. Context-aware mobile ar system for personalization, selective
sharing, and interaction of contents in ubiquitous computing environments.
In Human-Computer Interaction. Interaction Platforms and Techniques,
pages 966–974. Springer, 2007.
Joseph W. Sullivan, editor. Intelligent User Interfaces. ACM, New York, NY,
USA, 1991. ISBN 0201606410.
Ivan E. Sutherland. Sketch pad a man-machine graphical communication
system. In Proceedings of the SHARE design automation workshop, pages
6–329. ACM, 1964.
Ivan E. Sutherland. The ultimate display. In Proceedings of the IFIP Congress,
1965.
268 References
Ivan E. Sutherland. A head-mounted three dimensional display. In Proceedings
of the December 9-11, 1968, fall joint computer conference, part I, pages
757–764. ACM, 1968.
J. Edward Swan and Joseph L. Gabbard. Survey of user-based experimentation
in augmented reality. In Proceedings of 1st International Conference
on Virtual Reality, pages 1–9, 2005.
Zsolt Szalavári and Michael Gervautz. The personal interaction panel–a twohanded
interface for augmented reality. Computer Graphics Forum, 16(3):
C335–C346, 1997.
Zsolt Szalavári, Dieter Schmalstieg, Anton Fuhrmann, and Michael Gervautz.
“studierstube”: An environment for collaboration in augmented reality. Virtual
Reality, 3(1):37–48, 1998.
Akinari Takagi, Shoichi Yamazaki, Yoshihiro Saito, and Naosato Taniguchi.
Development of a stereo video see-through hmd for ar systems. In Augmented
Reality, 2000.(ISAR 2000). Proceedings. IEEE and ACM International
Symposium on, pages 68–77. IEEE, 2000.
Susumu Takumai. Surround-sound system. US Patent 8,041,060, October 18
2011.
John C. Tang and Ellen Isaacs. Why do users like video? Computer Supported
Cooperative Work (CSCW), 1(3):163–196, 1993.
Bruce Thomas, Victor Demczuk, Wayne Piekarski, David Hepworth, and
Bernard Gunther. A wearable computer system with augmented reality
to support terrestrial navigation. In Wearable Computers, 1998. Digest of
Papers. Second International Symposium on, pages 168–171. IEEE, 1998.
Bruce Thomas, Wayne Piekarski, and Bernard Gunther. Using augmented
reality to visualise architecture designs in an outdoor environment. International
Journal of Design Computing Special Issue on Design Computing
on the Net (DCNet), 1(4.2), 1999.
Bruce Thomas, Ben Close, John Donoghue, John Squires, Phillip De Bondi,
and Wayne Piekarski. First person indoor/outdoor augmented reality application:
Arquake. Personal and Ubiquitous Computing, 6(1):75–86, 2002.
Michael Tidwell, Richard S Johnston, David Melville, and TA Furness. The
virtual retinal display-a retinal scanning imaging system. Proceedings of
virtual reality World, 95:325–333, 1995.
Engin Tola, Vincent Lepetit, and Pascal Fua. Daisy: An e cient dense descriptor
applied to wide-baseline stereo. Pattern Analysis and Machine
Intelligence, IEEE Transactions on, 32(5):815–830, 2010.
References 269
Marjaana Träskbäack and Michael Haller. Mixed reality training application
for an oil reﬁnery: user requirements. In Proceedings of the 2004 ACM
SIGGRAPH international conference on Virtual Reality continuum and its
applications in industry, pages 324–327. ACM, 2004.
Roger Y Tsai. A versatile camera calibration technique for high-accuracy 3d
machine vision metrology using o -the-shelf tv cameras and lenses. Robotics
and Automation, IEEE Journal of, 3(4):323–344, 1987.
Mihran Tuceryan and Nassir Navab. Single point active alignment method
(spaam) for optical see-through hmd calibration for ar. In Augmented Reality,
2000.(ISAR 2000). Proceedings. IEEE and ACM International Symposium
on, pages 149–158. IEEE, 2000.
Yuko Uematsu and Hideo Saito. Ar registration by merging multiple planar
markers at arbitrary positions and poses via projective space. In Proceedings
of the 2005 international conference on Augmented tele-existence, pages 48–
55. ACM, 2005.
Luca Vacchetti, Vincent Lepetit, and Pascal Fua. Fusing online and o ine
information for stable 3d tracking in real-time. In Computer Vision and
Pattern Recognition, 2003. Proceedings. 2003 IEEE Computer Society Conference
on, volume 2, pages II–241. IEEE, 2003.
James Vallino and Christopher Brown. Haptics in augmented reality. In
Multimedia Computing and Systems, 1999. IEEE International Conference
on, volume 1, pages 195–200. IEEE, 1999.
Bram van der Vlist, Gerrit Niezen, Stefan Rapp, Jun Hu, and Loe Feijs. Conﬁguring
and controlling ubiquitous computing infrastructure with semantic
connections: a tangible and an ar approach. Personal and ubiquitous computing,
17(4):783–799, 2013.
DWF Van Krevelen and R. Poelman. A survey of augmented reality technologies,
applications and limitations. International Journal of Virtual Reality,
9(2):1, 2010.
Kurt VanLehn. The relative e ectiveness of human tutoring, intelligent tutoring
systems, and other tutoring systems. Educational Psychologist, 46
(4):197–221, 2011.
Eduardo E Veas. Augmented reality interfaces for environmental monitoring.
PhD thesis, Graz University of Technology, 2012.
Jonathan Ventura and Tobias Hollerer. Wide-area scene mapping for mobile
visual tracking. In Mixed and Augmented Reality (ISMAR), 2012 IEEE
International Symposium on, pages 3–12. IEEE, 2012.
270 References
Daniel Wagner and Dieter Schmalstieg. First steps towards handheld augmented
reality. In 2012 16th International Symposium on Wearable Computers,
pages 127–127. IEEE Computer Society, 2003.
Daniel Wagner and Dieter Schmalstieg. Artoolkitplus for pose tracking on
mobile devices. In Proceedings of 12th Computer Vision Winter Workshop,
page 8pp, 2007.
Daniel Wagner, Thomas Pintaric, and Dieter Schmalstieg. The invisible train:
a collaborative handheld augmented reality demonstrator. In ACM SIGGRAPH
2004 Emerging technologies, page 12. ACM, 2004.
Daniel Wagner, Thomas Pintaric, Florian Ledermann, and Dieter Schmalstieg.
Towards massively multi-user augmented reality on handheld devices. In
Pervasive Computing, volume 3468 of Lecture Notes in Computer Science,
pages 208–219. Springer Berlin Heidelberg, 2005.
Daniel Wagner, Mark Billinghurst, and Dieter Schmalstieg. How real should
virtual characters be? In Proceedings of the 2006 ACM SIGCHI international
conference on Advances in computer entertainment technology,
page 57. ACM, 2006.
Martin Wagner. Building wide-area applications with the artoolkit. In Augmented
Reality Toolkit, The First IEEE International Workshop, page 7pp.
IEEE, 2002.
Manuela Waldner, Jörg Hauber, Jürgen Zauner, Michael Haller, and Mark
Billinghurst. Tangible tiles: design and evaluation of a tangible user interface
in a collaborative tabletop setup. In Proceedings of the 18th Australia
conference on Computer-Human Interaction: Design: Activities, Artefacts
and Environments, pages 151–158. ACM, 2006.
Jih-fang Wang, Ronald T Azuma, Gary Bishop, Vernon Chi, John Eyles, and
Henry Fuchs. Tracking a head-mounted display in a room-sized environment
with head-mounted cameras. In Orlando’90, 16-20 April, pages 47–57.
International Society for Optics and Photonics, 1990.
Yanqing Wang and Christine L. MacKenzie. The role of contextual haptic
and visual constraints on object manipulation in virtual environments. In
Proceedings of the SIGCHI conference on Human Factors in Computing
Systems, pages 532–539. ACM, 2000.
Mark Ward, Ronald Azuma, Robert Bennett, Stefan Gottschalk, and Henry
Fuchs. A demonstrated optical tracker with scalable work area for headmounted
display systems. In Proceedings of the 1992 symposium on Interactive
3D graphics, pages 43–52. ACM, 1992.
References 271
Jens Weidenhausen, Christian Knoepﬂe, and Didier Stricker. Lessons learned
on the way to industrial augmented reality applications, a retrospective on
arvika. Computers & Graphics, 27(6):887–891, 2003.
Mark Weiser. Hot topics-ubiquitous computing. Computer, 26(10):71–72,
1993.
Greg Welch, Gary Bishop, Leandra Vicci, Stephen Brumback, Kurtis Keller,
et al. The hiball tracker: High-performance wide-area tracking for virtual
and augmented environments. In Proceedings of the ACM symposium on
Virtual reality software and technology, pages 1–10. ACM, 1999.
Dongdong Weng, Yongtian Wang, Yue Liu, Jing Chen, and Dewen Cheng.
Display systems and registration methods for mixed reality applications.
In SPIE Optical Engineering+ Applications, pages 742805–742805. International
Society for Optics and Photonics, 2009.
Giles Westerﬁeld, Antonija Mitrovic, and Mark Billinghurst. Intelligent augmented
reality training for assembly tasks. In Artiﬁcial Intelligence in
Education, pages 542–551. Springer, 2013.
Steve Whittaker and Brid O’Conaill. The role of vision in face-to-face and mediated
communication. In Kathleen E. Finn, Abigail J. Sellen, and Sylvia B.
Wilbur, editors, Video-mediated communication. Computers, cognition, and
work., pages 23–49. Lawrence Erlbaum Associates Publishers, 1997.
Ederyn Williams. Experimental comparisons of face-to-face and mediated
communication: A review. Psychological Bulletin, 84(5):963, 1977.
Rafal Wojciechowski, Krzysztof Walczak, Martin White, and Wojciech Cellary.
Building virtual and augmented reality museum exhibitions. In Proceedings
of the ninth international conference on 3D Web technology, pages
135–144. ACM, 2004.
Eric Woods, Mark Billinghurst, Julian Looser, Graham Aldridge, Deidre
Brown, Barbara Garrie, and Claudia Nelles. Augmenting the science centre
and museum experience. In Proceedings of the 2nd international conference
on Computer graphics and interactive techniques in Australasia and South
East Asia (GRAPHITE 2004), pages 230–236. ACM, 2004.
Harald Wuest, Florent Vial, and D. Strieker. Adaptive line tracking with multiple
hypotheses for augmented reality. In Mixed and Augmented Reality,
2005. Proceedings. Fourth IEEE and ACM International Symposium on,
pages 62–69. IEEE, 2005.
Peta Wyeth and Helen C. Purchase. Using developmental theories to inform
the design of technology for children. In Proceedings of the 2003 conference
on Interaction design and children, pages 93–100. ACM, 2003.
272 References
Yan Xu, Evan Barba, Iulian Radu, Maribeth Gandy, Richard Shemaka, Brian
Schrank, Blair MacIntyre, and Tony Tseng. Pre-patterns for designing
embodied interactions in handheld augmented reality games. In Mixed
and Augmented Reality-Arts, Media, and Humanities (ISMAR-AMH), 2011
IEEE International Symposium On, pages 19–28. IEEE, 2011.
Suya You and Ulrich Neumann. Fusion of vision and gyro tracking for robust
augmented reality registration. In Virtual Reality, 2001. Proceedings. IEEE,
pages 71–78. IEEE, 2001.
Suya You, Ulrich Neumann, and Ronald Azuma. Orientation tracking for outdoor
augmented reality registration. Computer Graphics and Applications,
IEEE, 19(6):36–42, 1999.
Jürgen Zauner and Michael Haller. Authoring of mixed reality applications
including multi-marker calibration for mobile devices. In 10th Eurographics
Symposium on Virtual Environments, EGVE, pages 87–90, 2004.
Jürgen Zauner, Michael Haller, Alexander Brandl, and W Hartman. Authoring
of a mixed reality assembly instructor for hierarchical structures. In
Mixed and Augmented Reality, 2003. Proceedings. The Second IEEE and
ACM International Symposium on, pages 237–246. IEEE, 2003.
Shumin Zhai and Paul Milgram. Telerobotic virtual control system. In
Robotics-DL tentative, pages 311–322. International Society for Optics and
Photonics, 1992.
Zhengyou Zhang. Microsoft kinect sensor and its e ect. MultiMedia, IEEE,
19(2):4–10, 2012.
Feng Zhou, Henry Been-Lirn Duh, and Mark Billinghurst. Trends in augmented
reality tracking, interaction and display: A review of ten years of
ismar. In Proceedings of the 7th IEEE/ACM International Symposium on
Mixed and Augmented Reality, pages 193–202. IEEE Computer Society,
2008.
Dmitry N. Zotkin, Jane Hwang, R. Duraiswaini, and Larry S. Davis. Hrtf
personalization using anthropometric measurements. In Applications of
Signal Processing to Audio and Acoustics, 2003 IEEE Workshop on., pages
157–160. IEEE, 2003.