10/12/2017 1 PA198 Augmented Reality Interfaces Lecture 12 Collaborative AR Applications & Future Fotis Liarokapis liarokap@fi.muni.cz 11th December 2017 Collaborative AR Applications Collaboration • Collaboration is working with others to do a task and to achieve shared goals – A recursive process where two or more people or organizations work together to realize shared goals https://en.wikipedia.org/wiki/Collaboration Collaborative Activities • Collaboration – Business, Entertainment, etc • Computer Supported Collaborative Work (CSCW) • Groupware Collaborative Learning • Collaborative activities are most often based on four principles: – The learner or student is the primary focus of instruction – Interaction and ‘doing’ are of primary importance – Working in groups is an important mode of learning – Structured approaches to developing solutions to real-world problems should be incorporated into learning http://www.cte.cornell.edu/teaching-ideas/engaging-students/collaborative-learning.html http://www.csm.ornl.gov/~geist/java/applets/enote/Slides/sld002.htm 10/12/2017 2 Blooms Taxonomy https://sites.google.com/a/aps.edu/aps-ipad-apps/bloom-s-taxonomy-m-learning Computer-Supported Cooperative Work (CSCW) • The term computer-supported cooperative work (CSCW) was first coined by Irene Greif and Paul M. Cashman in 1984, at a workshop attended by individuals interested in using technology to support people in their work • CSCW is a generic term, which combines the understanding of the way people work in groups with the enabling technologies of computer networking, and associated hardware, software, services and techniques https://en.wikipedia.org/wiki/Computer-supported_cooperative_work CSCW Matrix https://en.wikipedia.org/wiki/Computer-supported_cooperative_work Today’s Technology • Video Conferencing – Lack of spatial cues – Limited participants – 2D collaboration • Collaborative Virtual Environments – Separation from real world – Reduced conversational cues Billinghurst, M. Lecture 6: Collaborative AR Applications, HIT Lab NZ, University of Canterbury Beyond Video Conferencing • 2D Interface onto 3D – VRML, Web3D • Projection Screen – CAVE, WorkBench • Volumetric Display – Scanning laser • Virtual Reality – Natural spatial cues Billinghurst, M. Lecture 6: Collaborative AR Applications, HIT Lab NZ, University of Canterbury 10/12/2017 3 Beyond Virtual Reality • Lessons from CSCW – Seamless – Enhance Reality • Immersive Virtual Reality – Separates from real world – Reduces conversational cues Billinghurst, M. Lecture 6: Collaborative AR Applications, HIT Lab NZ, University of Canterbury Future Collaboration ? • Remote Conferencing • Face to face Conferencing Billinghurst, M. Lecture 6: Collaborative AR Applications, HIT Lab NZ, University of Canterbury AR & Collaboration • Claim: – AR techniques can be used to provide spatial cues that significantly enhance face-to-face and remote collaboration on three-dimensional tasks Billinghurst, M. Lecture 6: Collaborative AR Applications, HIT Lab NZ, University of Canterbury Construct3D [Kaufmann 2000] • Collaborative geometry education tool • Different learning modes – Teacher, student, exam • Tangible interaction – Personal interaction panel Billinghurst, M. Lecture 6: Collaborative AR Applications, HIT Lab NZ, University of Canterbury Construct3D Video https://www.youtube.com/watch?v=tABwBrWL4tc Collaborative AR • Seamless Interaction • Natural Communication • Attributes: – Virtuality – Augmentation – Co-operation – Independence – Individuality Billinghurst, M. Lecture 6: Collaborative AR Applications, HIT Lab NZ, University of Canterbury 10/12/2017 4 Seamless CSCW • Seam – Spatial, temporal, functional discontinuity • Types of Seams – Functional • Between different functional workspaces – Cognitive • Between different work practices Billinghurst, M. Lecture 6: Collaborative AR Applications, HIT Lab NZ, University of Canterbury Functional Seams Billinghurst, M. Lecture 6: Collaborative AR Applications, HIT Lab NZ, University of Canterbury Cognitive Seams Billinghurst, M. Lecture 6: Collaborative AR Applications, HIT Lab NZ, University of Canterbury Effect of Seams • Functional Seams: – Loss of Gaze Information – Degradation of Non-Verbal Cues • Cognitive Seams: – Learning Curve Effects – User Frustration Billinghurst, M. Lecture 6: Collaborative AR Applications, HIT Lab NZ, University of Canterbury Open Research Questions • Does seamlessness enhance performance? • What AR cues can enhance collaboration ? • How does AR collaboration differ ? • What technology is required ? Billinghurst, M. Lecture 6: Collaborative AR Applications, HIT Lab NZ, University of Canterbury Collaborative AR Interfaces • Face to Face Collaboration – WebSpace, Shared Space, Table Top Demo, Interface • Comparison, AR Interface Comparison • Remote Collaboration – SharedView, RTAS, Wearable Info Space, WearCom, AR Conferencing, BlockParty • Transitional Interfaces – MagicBook • Hybrid Interfaces – AR PRISM, GI2VIS Billinghurst, M. Lecture 6: Collaborative AR Applications, HIT Lab NZ, University of Canterbury 10/12/2017 5 Face to Face Collaboration Communication Cues • A wide variety of communication cues used Billinghurst, M. Lecture 6: Collaborative AR Applications, HIT Lab NZ, University of Canterbury Communication Cues . • In computer supported collaboration it is often hard for users to exchange non-verbal communication cues, even when they are co- located Billinghurst, M. Lecture 6: Collaborative AR Applications, HIT Lab NZ, University of Canterbury Differences in Collaboration • Face-to-face collaboration – People surround a table – It is easy to see each other • Computer supported collaboration – People sit side by side – It is hard to see each other Billinghurst, M. Lecture 6: Collaborative AR Applications, HIT Lab NZ, University of Canterbury Shared Space - Table Top Demo • Goal – Create compelling collaborative AR interface usable by novices • Exhibit content – Matching card game – Face to face collaboration – Physical interaction Billinghurst, M. Lecture 6: Collaborative AR Applications, HIT Lab NZ, University of Canterbury Results from Shared Space • 2,500 - 3,000 users • Observations – No problems with the interface • Only needed basic instructions – Physical objects easy to manipulate – Spontaneous collaboration • User study (157 participants) – Users felt they could easily play with other people and interact with objects • Improvements – Reduce lag, improve image quality, better HMD Billinghurst, M. Lecture 6: Collaborative AR Applications, HIT Lab NZ, University of Canterbury 10/12/2017 6 AR Pad • Handheld AR Display – LCD screen – SpaceOrb – Camera – Peripheral awareness Billinghurst, M. Lecture 6: Collaborative AR Applications, HIT Lab NZ, University of Canterbury Support for Collaboration Billinghurst, M. Lecture 6: Collaborative AR Applications, HIT Lab NZ, University of Canterbury Holograms Holography • Holography is the science and practice of making holograms • A hologram is a photographic recording of a light field – Rather than of an image formed by a lens • It is used to display a fully 3D image of the holographed subject – Which is seen without the aid of special glasses or other intermediate optics https://en.wikipedia.org/wiki/Holography Holography . • In its pure form, holography requires the use of laser light for illuminating the subject and for viewing the finished hologram https://en.wikipedia.org/wiki/Holography Reconstructing a Hologram https://en.wikipedia.org/wiki/Holography 10/12/2017 7 Recording a Hologram • To make a hologram, the following are required: – A suitable object or set of objects – A suitable laser beam – Part of the laser beam to be directed so that it illuminates the object beam and another part so that it illuminates the recording medium directly (the reference beam) • Enabling the reference beam and the light which is scattered from the object onto the recording medium to form an interference pattern – A recording medium • Converts this interference pattern into an optical element which modifies either the amplitude or the phase of an incident light beam according to the intensity of the interference pattern – An environment • Provides sufficient mechanical and thermal stability that the interference pattern is stable during the time in which the interference pattern is recorded https://en.wikipedia.org/wiki/Holography CNN Hologram • Elections in 2008, USA • Holographic technology used – First time in TV CNN Hologram Video https://www.youtube.com/watch?v=thOxW19vsTg Basic AR Conferencing • Moves conferencing from the desktop to the workspace Billinghurst, M. Lecture 6: Collaborative AR Applications, HIT Lab NZ, University of Canterbury Features • Hardware – SGI O2 – Virtual i-O HMD – Head mounted camera • Software – Live video – Shared whiteboard – Vision based registration/tracking Billinghurst, M. Lecture 6: Collaborative AR Applications, HIT Lab NZ, University of Canterbury Pilot Study • How does AR conferencing differ ? • Task – Discussing images – 12 pairs of subjects • Conditions – Audio only (AC) – Video conferencing (VC) – Mixed reality conferencing (MR) Billinghurst, M. Lecture 6: Collaborative AR Applications, HIT Lab NZ, University of Canterbury 10/12/2017 8 Results • Paid more attention to pictures • Remote video provided peripheral cues • In AR condition – Difficult to see everything – Remote user distracting – Communication asymmetries Billinghurst, M. Lecture 6: Collaborative AR Applications, HIT Lab NZ, University of Canterbury A Wearable Conferencing Space • Features – Mobile video conferencing – Full size images – Spatial audio/visual cues – Interaction with real world – Dozens of users – Body-stabilized data Billinghurst, M. Lecture 6: Collaborative AR Applications, HIT Lab NZ, University of Canterbury Initial Prototype • Internet Telephony • Spatial Audio/Visuals • See-through HMD • Head Tracking • Wireless Internet • Wearable Computer • Static Images Billinghurst, M. Lecture 6: Collaborative AR Applications, HIT Lab NZ, University of Canterbury Software Architecture • Multicast Groups • Position Broadcasting – 10 kb/s per person • Audio Broadcasting – 172 kb/s per person • Local sound spatialization – DirectSound3D • Graphics Interface – DirectX/Direct3D Billinghurst, M. Lecture 6: Collaborative AR Applications, HIT Lab NZ, University of Canterbury Pilot User Study • Can MR spatial cues aid comprehension? • Task – Recognize words in spoken phrases • Conditions – Number of speakers • 1,3,5 simultaneous speakers – Spatial/Non Spatial Audio – Visual/Non visual cues Billinghurst, M. Lecture 6: Collaborative AR Applications, HIT Lab NZ, University of Canterbury Results Billinghurst, M. Lecture 6: Collaborative AR Applications, HIT Lab NZ, University of Canterbury 10/12/2017 9 Advanced AR Conferencing • Superimpose video of remote person over real world Billinghurst, M. Lecture 6: Collaborative AR Applications, HIT Lab NZ, University of Canterbury System Architecture Billinghurst, M. Lecture 6: Collaborative AR Applications, HIT Lab NZ, University of Canterbury Tangible Manipulation • Using real paddle to move virtual user Billinghurst, M. Lecture 6: Collaborative AR Applications, HIT Lab NZ, University of Canterbury AR Remote Conferencing • Progression – 2D to Spatial Cues to 3D – Increasing realism (visual/audio cues) Billinghurst, M. Lecture 6: Collaborative AR Applications, HIT Lab NZ, University of Canterbury AR Videoconferencing for Social Interaction Video https://www.youtube.com/watch?v=uXPYoOR96OQ CONFETTI Video https://www.youtube.com/watch?v=3ei9PXSZ3B8 10/12/2017 10 Multiscale Collaboration MagicBook Concept • Goal – A collaborative AR interface supporting transitions from reality to virtual reality • Physical Components – Real book • Display Elements – AR and VR content • Interaction Metaphor – Book pages hold virtual scenes Billinghurst, M. Lecture 6: Collaborative AR Applications, HIT Lab NZ, University of Canterbury Milgram’s Reality-Virtuality Continuum • Milgram defined the term ‘Augmented Virtuality’ to identify systems which are mostly synthetic with some real world imagery added such as texture mapping video onto virtual objects Milgram, P., Kishino, A.F. Taxonomy of Mixed Reality Visual Displays, IEICE Transactions on Information and Systems, 1321-1329, 1994. MagicBook Transitions • Interfaces of the future will need to support transitions along the Reality-Virtuality continuum • Augmented Reality is preferred for: – Co-located collaboration • Immersive Virtual Reality is preferred for: – Experiencing world immersively (egocentric) – Sharing views – Remote collaboration Billinghurst, M. Lecture 6: Collaborative AR Applications, HIT Lab NZ, University of Canterbury MagicBook Features • Seamless transition between Reality and Virtuality – Reliance on real decreases as virtual increases • Supports egocentric and exocentric views – User can pick appropriate view • Computer becomes invisible – Consistent interface metaphors – Virtual content seems real • Supports collaboration Billinghurst, M. Lecture 6: Collaborative AR Applications, HIT Lab NZ, University of Canterbury MagicBook Collaboration • Collaboration on multiple levels: – Physical Object – AR Object – Immersive Virtual Space • Egocentric + exocentric collaboration – Multiple multi-scale users • Independent Views – Privacy, role division, scalability Billinghurst, M. Lecture 6: Collaborative AR Applications, HIT Lab NZ, University of Canterbury 10/12/2017 11 MagicBook Video https://www.youtube.com/watch?v=gk_njOc0xwE Conclusions • Face to face collaboration – AR preferred over immersive VR – AR facilitates seamless/natural communication • Remote Collaboration – AR spatial cues can enhance communication – AR conferencing improves video conferencing – Many possible confounding factors • Future – Expect a lot of new AR technologies and apps Future of AR Up to Now • Many years of development – A lot of achievements • Moving from desktop to mobile – New interfaces are required – Research is changing AR Nowadays • 30th November 2015 AR went to space! • New hardware improvements expected • Many companies – > $600 Million USD market • And growing – Thousands of applications (mainly mobile) • A lot of tools exist but no complete solution Current Research in AR • Social Acceptance – Overcome social problems with AR • Cloud Services – Cloud based storage/processing • AR Authoring Tools – Easy content creation for non-experts • Collaborative Experiences – AR teleconferencing 10/12/2017 12 Investments • Big investments by Google and Apple – 29 M Euros Apple (Metaio) – 542 M dollars (Magic Leap) • Facebook invested in VR Facts & Expectations http://www.neosentec.com/news/economic -expectation-of-wearable-devices- augmented-reality-and-google-glass/ Juniper Research http://zugara.com/cmos-select-augmented-reality-future-trend-marketing http://zugara.com/augmented-reality-and-virtual-reality-software-market-projections http://zugara.com/augmented-reality-and-virtual-reality-software-market-projections Device Forecast (2015-2018) http://www.i4u.com/2015/06/92427/augmented-and-virtual-reality-market-be-4-billion-3-years 10/12/2017 13 http://www.pwc.com/us/en/technology-forecast/augmented-reality/augmented-reality-road-ahead.html Areas that Shape the Future of AR http://www.pwc.com/us/en/technology-forecast/augmented-reality/augmented-reality-road-ahead.html Commercial Systems • Ngrain – http://www.ngrain.com/ – Training authoring tool – Model based AR tracking – Focus on industrial applications Billinghurst, M. Augmented Reality: The Next 20 Years, AWE Asia, 18th October 2015. Ngrain Video https://www.youtube.com/watch?v=DKEcQ9uiII0 Commercial Systems . • ScopeAR – http://www.scopear.com/ – Remote assistance – Image based tracking Billinghurst, M. Augmented Reality: The Next 20 Years, AWE Asia, 18th October 2015. ScopeAR Video http://www.scopear.com/remotear/ 10/12/2017 14 Key Enabling Technologies • Augmentation – Display Technology • Real-time interaction – Interaction Technologies • 3D Registration – Tracking Technologies Billinghurst, M. Augmented Reality: The Next 20 Years, AWE Asia, 18th October 2015. Displays Displays Projections • Early years – Bulky HMDs • Nowadays – Handheld, lightweight HMDs • Near Future – Projected AR – Wide FOV see through – Retinal displays • Far Future – Contact lens Projected AR (1-3 years) • Use stereo head mounted projectors – Rollable retro-reflective sheet • • Wide FOV, shared interaction – i.e. CastAR (http://castar.com) • $400 USD, available Q4 2015 Billinghurst, M. Augmented Reality: The Next 20 Years, AWE Asia, 18th October 2015. CastAR Video http://castar.com/ Wide FOV See-Through (3+ years) • Waveguide techniques – Wider FOV – Thin see through – Socially acceptable • Pinlight Displays – LCD panel + point light sources – 110 degree FOV – UNC/Nvidia Maimone, A., Lanman, D., et al. Pinlight displays: wide field of view augmented reality eyeglasses using defocused point light sources, Proc of ACM SIGGRAPH 2014 Emerging Technologies, 20, 2014. Lumus DK40 10/12/2017 15 Retinal Displays (5+ years) • Photons scanned into eye – Infinite depth of field – Bright outdoor performance – Overcome visual defects – True 3D stereo with depth modulation • Microvision (1993-) – Head mounted monochrome • MagicLeap (2013-) – Projecting light field into eye Billinghurst, M. Augmented Reality: The Next 20 Years, AWE Asia, 18th October 2015. Contact Lens (15 + years) • Contact Lens only – Unobtrusive – Significant technical challenges • Power, data, resolution • Contact Lens + Micro-display – Wide FOV – Socially acceptable – Innovega • http://innovega-inc.com/ http://spectrum.ieee.org/biomedical/bionics/augmented-reality-in-a-contact-lens/ Interaction Interaction Projections • Early years – Limited interaction – Viewpoint manipulation • Nowadays – Screen based, simple gesture – Tangible interaction • Future – Natural gesture, Multimodal – Intelligent Interfaces – Physiological/Sensor based Billinghurst, M. Augmented Reality: The Next 20 Years, AWE Asia, 18th October 2015. Natural Gesture (2-5 years) • Freehand gesture input – Depth sensors for gesture capture – Move beyond simple pointing – Rich two handed gestures • i.e. Microsoft Research Hand Tracker – 3D hand tracking, 30 fps, single sensor • Commercial Systems – Meta, Hololens, Occulus, Intel, etc Sharp, T., Keskin, C., et al. Accurate, Robust, and Flexible Real-time Hand Tracking, Proc CHI, Vol. 8, 2015. Smart Glass Hand Interaction • EnvisageAR + Phonevers • RGB-D hand tracking on Android • Natural gesture input for glasses Billinghurst, M. Augmented Reality: The Next 20 Years, AWE Asia, 18th October 2015. 10/12/2017 16 Multimodal Input (5+ years) • Combine gesture and speech input – Gesture good for qualitative input – Speech good for quantitative input – Support combined commands – “Put that there” + pointing • HIT Lab NZ multimodal input – 3D hand tracking, speech – Multimodal fusion module – Complete tasks faster with MMI, less errorsBillinghurst, M. Piumsomboon, T., et al. Hands in Space: Gesture Interaction with Augmented-Reality Interfaces, IEEE computer graphics and applications, (1), 77-80, 2014. Intelligent Interfaces (10+ years) • Move to Implicit Input vs. Explicit – Recognize user behaviour – Provide adaptive feedback – Support scaffolded learning – Move beyond check-lists of actions • Eg AR + Intelligent Tutoring – Constraint based ITS + AR – PC Assembly – 30% faster, 25% better retention Westerfield, G., Mitrovic, A., & Billinghurst, M. Intelligent Augmented Reality Training for Motherboard Assembly, International Journal of Artificial Intelligence in Education, 25(1), 157-172, 2015. Tracking Tracking Projections • Early years – Location based, marker based, – Magnetic/mechanical • Nowadays – Image based, hybrid tracking • Future – Ubiquitous – Model based – Environmental Billinghurst, M. Augmented Reality: The Next 20 Years, AWE Asia, 18th October 2015. Model Based Tracking (1-3 yrs) • Track from known 3D model – Use depth + colour information – Match input to model template – Use CAD model of targets • Recent innovations – Learn models online – Tracking from cluttered scene – Track from deformable objects Hinterstoisser, S., Lepetit, V., et al. Model based training, detection and pose estimation of texture-less 3D objects in heavily cluttered scenes, Computer Vision–ACCV 2012, Springer Berlin Heidelberg, 548-562, 2013. Environmental Tracking (3+ yrs) • Environment capture – Use depth sensors to capture scene & track from model • InifinitAM – Real time scene capture on mobiles (dense or sparse) – Dynamic memory swapping allows large environment capture – Cross platform, open source library available Billinghurst, M. Augmented Reality: The Next 20 Years, AWE Asia, 18th October 2015. 10/12/2017 17 InifinitAM Video http://www.robots.ox.ac.uk/~victor/infinitam/ Wide Area AR Tracking (5+ yrs) • Using panorama imagery • Processed into a point cloud dataset • Used for AR localisation Ventura, J., Hollerer, T. Wide-area scene mapping for mobile visual tracking, Proc. of the International Symposium on Mixed and Augmented Reality 2012, (ISMAR), IEEE Computer Society, 3-12, 2012. Conclusions • 30th November 2015 AR went to space! • New hardware improvements expected • Many companies – > $600 Million USD market • And growing – Thousands of applications (mainly mobile) • A lot of tools exist but no complete solution Questions