20/02/2018 1 PA201 Virtual Environments Lecture 1 Introduction to Virtual Reality Fotis Liarokapis liarokap@fi.muni.cz 20th February 2018 Fotis Liarokapis • PhD in Computer Engineering – University of Sussex, UK • MSc in Computer Graphics and Virtual Environments – University of Hull, UK • BSc in Computer Systems Engineering – University of Sussex, UK My Research • Research areas: – Computer Graphics – Virtual Reality – Augmented Reality – Procedural Modeling – Interactive Environments – Serious Games – User studies Contact Details • Email: – liarokap@fi.muni.cz • Telephone: – 549493948 • Office Location: – C411 • Office Hour: – Tuesday 13:00 to 14:00 Course Details • Prerequisites – Knowledge of computer graphics fundamentals • Lectures – Every Tuesday – Time: 14:00 to 16:00 – Location: A218 • Lab/Seminar – Every Tuesday – Time: 16:00 to 17:00 – Location: A215 Course Objectives • Demonstrate an understanding of the main mathematical concepts, hardware and software technologies used in immersive virtual environments • Evaluate different approaches, methodologies and tools focused on virtual reality • Propose virtual environments for both indoor and outdoor environments • Design virtual reality applications for various application domains 20/02/2018 2 Syllabus • Introduction to Virtual Environments • Head Mounted Displays • Other Visualisation Technologies • Virtual reality interaction techniques • Design guidelines • Tracking technologies • Physical simulation • Haptics • Perception Issues for Virtual Environments • Psychological aspects of Virtual Environments (i.e. motion sickness, nausea) • Collaborative Virtual Environments • Software for Virtual Environments (i.e. VR toolkits) • Mobile Virtual Environments • Application Domains • Future of Virtual Environments Teaching Methods • Delivery of the material will be based on – Expositional lectures – Reinforced by computer demonstrations of the application of the material – Video demonstrations Assessment Methods • An assignment about designing, implementing and testing a virtual environment – An essay – Practical assignment Plagiarism and Cheating • If you use an external resource cite it clearly! • Don’t do things that would be considered dishonest... if in doubt ask • Cheating earns you: – Fail in the class – Getting reported to the University – No exceptions Assignment - Overview • The aim of the VR application is to raise people’s archaeological knowledge and cultural awareness • It will provide immersive technologies to increase interaction time in an underwater archaeological site, both for the public, as well as, for researchers and scholars • Users can experience an immersive virtual underwater visit using off-the-self VR headsets • Apart from their visit, they can also get some information about the archaeological artefacts (i.e. textual descriptions, videos and sounds) • Focus is not on simulating swimming but on underwater excavation techniques based on archaeological procedures Assignment - Objectives • The main objective is: – Design and implement an underwater virtual reality excavation experience • Focus can be given on: – Interactive storytelling – Realistic exploration – Real-time excavation – Multimodal Interaction – Collaboration – Tasks/Scenarios 20/02/2018 3 Assignment - Environment • Environment – Terrain – Vegetation – Fish – Effects • Game objects: – Ceramics (amphorae, galley wares etc) – Wood (hull components, ship’s gear or other artefacts on-board) – Pieces of anchors – Other organic material (fruit pits, leather, baskets etc) Implementation • Make use of immersive VR to create an underwater excavation game – Oculus Rift, HTC Vive • Implementation in Unity – Easy to port to HMDs • Emphasis will be given on the interaction and visualisation techniques • Deadline end of the term! Details • The application should be focused on indoor environments – So not mobile! • The topic is focused on designing a game/tool to assist archaeologists to learn underwater excavation • Visualisation – All types of multimedia information can be superimposed Intrusive Techniques • Probing • Taking Samples • Loosening or Removing Sediment http://www.unesco.org/new/fileadmin/MULTIMEDIA/HQ/CLT/images/630X300/UNIT10.pdf Probing • Probing is a physical attempt to locate structures beneath the surface layers and is usually carried out in a systematic manner (along a line at fixed distances), to understand the extent of a site and depth of burial • The results of a probing survey are dependent on feel and are difficult to accurately measure Taking Samples • A sample is a representative amount of material that has been collected from an archaeological or natural context • Samples are collected for a range of reasons, though usually for material identification of artefacts dating, environmental analysis and comparative background assessment, such as typological analysis http://www.unesco.org/new/fileadmin/MULTIMEDIA/HQ/CLT/images/630X300/UNIT10.pdf 20/02/2018 4 Loosening or Removing Sediment • The choice of tool to excavate a site depends on the nature of the sediments covering a site, and the condition and material type of objects being excavated • Since the aim of archaeological excavation is to meticulously excavate and document artefacts, the need to rapidly remove sediment (other than sterile overburden) is not a primary concern – Have a look at the AirLift http://www.unesco.org/new/fileadmin/MULTIMEDIA/HQ/CLT/images/630X300/UNIT10.pdf AirLift • Airlifts can be constructed in various ways • A common feature is a long discharge pipe with a diver controlled on/off tap – Which can also be used to regulate the airflow and strength of the suction http://www.unesco.org/new/fileadmin/MULTIMEDIA/HQ/CLT/images/630X300/UNIT10.pdf VR Airlift Example • Working with the dredging device (airlift) to remove a layer of sand VR Stratigraphical Excavation • After removal of sand, lower (lighter-colored) layer of sand should be exposed 3D Scuba Diving Simulator • Zenobia game gives you the real diving experience to plan your upcoming dive • The app is developed with realistic concepts, which gives you the real diving experience http://www.softwaredriveus.com/2017/01/scuba-dive-simulator-zenobia-game-that.html Videos • PowerRay™ - Change the fishing world – https://www.youtube.com/watch?v=LUhPl-6X-H0 • Titanic VR – https://www.kickstarter.com/projects/1436197736/ titanic-vr?ref=card • VISAS - Virtual Dive Experience with Totem and HMD – https://www.youtube.com/watch?v=fGIOgUrzO2Q& feature=youtu.be 20/02/2018 5 Report Structure • Title page • Contents • Abstract (or summary) • Introduction • Background theory • Methodology and results • Conclusions • References • Appendices VR History First VR Interpretation Stereoscopic Photos & Viewers • In 1838 Charles Wheatstone’s research demonstrated that the brain processes the different 2D images from each eye into a single object of three dimensions – Viewing two side by side stereoscopic images or photos through a stereoscope gave the user a sense of depth and immersion – The later development of the popular View-Master stereoscope (patented 1939), was used for “virtual tourism” – The design principles of the Stereoscope is used today for the popular Google Cardboard and low budget VR head mounted displays for mobile phones http://www.vrs.org.uk/virtual-reality/history.html Stereoscopic Photos & Viewers . • 1838 : The stereoscope (Charles Wheatstone) http://www.vrs.org.uk/virtual-reality/history.html Stereoscope and View-Master • 1849: The lenticular stereoscope (David Brewster) • 1939: The View-Master (William Gruber) http://www.vrs.org.uk/virtual-reality/history.html 20/02/2018 6 Link Trainer • In 1929 Edward Link created the “Link trainer” (patented 1931) probably the first example of a commercial flight simulator, which was entirely electromechanical • Controlled by motors – That linked to the rudder and steering column to modify the pitch and roll – A small motor-driven device mimicked turbulence and disturbances http://www.vrs.org.uk/virtual-reality/history.html Link Trainer . • US military bought six of these devices for $3500 (~ $50,000 today's money) – During World War II over 10,000 “blue box” Link Trainers were used by over 500,000 pilots for initial training and improving their skills http://www.vrs.org.uk/virtual-reality/history.html Science Fiction Story Predicted VR • In the 1930s, Stanley G. Weinbaum (Pygmalion’s Spectacles) proposed a pair of goggles that let the wearer experience a fictional world through holographics, smell, taste and touch http://www.vrs.org.uk/virtual-reality/history.html Sensorama • In mid 1950s Morton H Eilig built a single user console called Sensorama that included a stereoscopic display, fans, or emitters, stereo speakers and a moving chair – This enabled the user watch television in 3D Telesphere Mask • In 1960, Morton Heilig invented the Telesphere Mask (patented 1960) – First example of a head-mounted display (HMD), albeit for the noninteractive film medium without any motion tracking – The headset provided stereoscopic 3D and wide vision with stereo sound http://www.vrs.org.uk/virtual-reality/history.html Headsight HMD • In 1961, Philco Corporation engineers developed the first Head-Mounted Display (HMD) – Known as the ‘Headsight’ • The helmet consisted of a video screen along with a tracking system – Also linked to a closed circuit camera system • Similar HMD was used later for helicopter pilots 20/02/2018 7 Ultimate Display • In 1965, Ivan Sutherland proposed the ‘Ultimate Display’ – After using this display a person imagines the virtual world very similar to the real world • During 1966, he built an HMD – Was tethered to a computer system Ultimate Display HMD GROPE • In 1967, Brooks developed force feedback GROPE system VR History 1997-1994 • 1977 – Sandin and Sayre invent a bend-sensing glove • 1979 – Raab et al: Polhemus tracking system • 1994 – VR Society formed Virtual Reality Birth • In 1987, Jaron Lanier, founder of the visual programming lab (VPL), coined the term ‘virtual reality’ and developed a range of VR gear – Dataglove (along with Tom Zimmerman) – EyePhone head mounted display • A major development in the area of virtual reality haptics http://www.vrs.org.uk/virtual-reality/history.html Virtual Visual Environmental Display • The first complete VR system was developed by NASA – “Virtual Visual Environmental Display” • Became “Virtual Interface Environment Workstation” (VIEW) 1989 Sega VR Headset • In 1993, Sega announced the Sega VR headset for the Sega Genesis console – The wrap-around protoype glasses had head tracking, stereo sound and LCD screens in the visor • A commercial failure – Developed 4 games for this product but naver made it to the market due to technical development difficulties http://www.vrs.org.uk/virtual-reality/history.html 20/02/2018 8 Nintendo Virtual Boy • In 1995, Nintendo Virtual Boy (known as VR-32) was the first ever portable console that could display true 3D graphics • A commercial failure – The following year they discontinued its production and sale • Reasons for this failure were – Lack of colour in graphics (games were in red and black) – Lack of software support – Difficult to use the console http://www.vrs.org.uk/virtual-reality/history.html VR Nowadays Current State of VR • Nowadays VR is moving from the research laboratories to the working environment by replacing ergonomically limited HMD’s with – Projective displays • CAVE and Responsive Workbench – Mobile devices • PDAs, smartphones, tablets, etc – Interaction devices • Wii, Kinect, many others VR Nowadays • $3-5 Billion VR business – Around $150 Billion Graphics Industry – Visualization, simulation, gaming, CAD/CAE, multimedia, graphics arts • Closely aligned with computer games/video games and other apps VR Nowadays . • A number of different expensive devices exist • Target to get full immersion Google Cardboard • Launched in 2014 • SDK is available for the Android and iOS • Over 5 million Cardboard viewers • Over 1,000 compatible applications https://vr.google.com/daydream/ 20/02/2018 9 Open Source Virtual Reality • The Open Source Virtual Reality (OSVR) introduced on January 2015 • VR headset that claims to be open-source hardware and use open-source software – Developed by Razer and Sensics https://www.razerzone.com/osvr Samsung Gear VR • Released on November 27, 2015 in collaboration with Oculus • Designed to work with Samsung’s flagship smartphones http://www.samsung.com/global/galaxy/gear-vr/ Google Daydream • The Daydream View headset is made with lightweight fabric and launched in 2016 • Paired with a controller equipped with smart sensors to understand movements and gestures https://vr.google.com/daydream/ PlayStation VR • Project Morpheus launched on October 13, 2016 • Designed to be fully functional with the PlayStation 4 • Tracks 360 degree head movement based on nine positional LEDs on its surface for the PlayStation Camera https://www.playstation.com/en-us/explore/playstation-vr/ Oculus Rift • Released on March 28, 2016 • The Rift has a stereoscopic OLED display, 1080×1200 resolution per eye, a 90 Hz refresh rate, and 110° field of view https://www.oculus.com/ HTC Vive • Released on 5 April 2016 • Refresh rate of 90 Hz, two screens, one per eye, each having a display resolution of 1080x1200 • Laser position sensors, 4.6 by 4.6 m tracking space http://store.steampowered.com/app/358040/ 20/02/2018 10 FOVE • FOVE is the first virtual reality headset that utilizes eye tracking – Accuracy of 1/20th of a degree – Will utilize SteamVR's Lighthouse technology for positional tracking https://www.getfove.com/ VR Definitions Virtual Reality Definition • “The computer-generated simulation of a 3D image or environment that can be interacted with in a seemingly real or physical way by a person using special electronic equipment, such as a helmet with a screen inside or gloves fitted with sensors” • Some popular related terms include: – Virtual Environments (VE), Artificial Reality, Telepresence and Cyberspace http://www.oxforddictionaries.com/definition/english/virtual-reality Telepresence • The use of various technologies to produce the effect of placing the user in another location Artificial Reality • Responsive Environment – An environment where human behavior is perceived by a computer which interprets what it observes and responds through intelligent visual and auditory displays Burdea’s 3 I’s of VR • Interactivity – User impacts world • Immersion – Believing you are there • Imagination – User ‘buying’ into the experience 20/02/2018 11 Zeltzer’s Cube https://www.cg.tuwien.ac.at/research/publications/1996/mazuryk-1996-VRH/TR-186-2-96-06Paper.pdf VR Typical Architecture User (programmer, trainee, etc.) Modern VR Systems Basic Components https://www.cg.tuwien.ac.at/research/publications/1996/mazuryk-1996-VRH/TR-186-2-96-06Paper.pdf VR Rendering • The visual display transformation for VR is much more complex than in standard computer graphics https://www.cg.tuwien.ac.at/research/publications/1996/mazuryk-1996-VRH/TR-186-2-96-06Paper.pdf Stereoscopy • Stereo vision relies on depth-cues based on retinal disparity and therefore may greatly increase the feeling of immersion https://www.cg.tuwien.ac.at/research/publications/1996/mazuryk-1996-VRH/TR-186-2-96-06Paper.pdf Stereoscopic depth-cues: (a) eye convergence, (b) retinal disparity 20/02/2018 12 VR Types Characteristic Types • Windows on World • Immersive • Telepresence • Distributed • Mixed Reality Windows on World (WoW) • Also called monitorbased VR – Similar to games • Using a conventional computer monitor (or mobile device) to display the 3D virtual world Immersive VR • Completely immerse the user's personal viewpoint inside the virtual 3D world • The user has no visual contact with the physical word • Often equipped with a Head Mounted Display (HMD) or use of CAVE displays Telepresence • A variation of visualizing complete computer generated worlds • Links remote sensors in the real world with the senses of a human operator • The remote sensors might be located on a robot • Useful for performing operations in dangerous environments Teleoperating https://www.cg.tuwien.ac.at/research/publications/1996/mazuryk-1996-VRH/TR-186-2-96-06Paper.pdf 20/02/2018 13 Distributed VR • A simulated world runs on several computers which are connected over network and the people are able to interact in real time, sharing the same virtual world Mixed Reality • Milgram coined the term ‘Augmented Virtuality’ to identify systems which are mostly synthetic with some real world imagery added such as texture mapping video onto virtual objects Paul Milgram and Fumio Kishino, 1994 VR Paramount VR Immersion • In a typical VR system the user’s natural sensory information is completely replaced with digital information • The user’s experience of a computer-simulated environment is called immersion • As a result, VR systems can completely immerse a user inside a synthetic environment by blocking all the signals of the real world VR Paramount • Paramount for the sensation of immersion into virtual reality are – High frame rate • At least 95 fps – Low latency – Pixel persistence lower than 3 ms is required • If not, users will feel sick when moving their head around https://en.wikipedia.org/wiki/Virtual_reality VR Paramount Diagram https://en.wikipedia.org/wiki/Virtual_reality 20/02/2018 14 VR Technologies VR Tracking • Many technologies exist – Optical and magnetic and the most dominant • Of course it depends on the application domain • Immersive VR requires the position and orientation of the viewer’s head – Needed for the proper rendering of images • Additionally other parts of body may be tracked e.g., hands – to allow interaction, chest or legs – to allow the graphical user representation etc VR Tracking . • In general there are two kinds of trackers: – Those that deliver absolute data (total position/orientation values) – Those that deliver relative data (i.e. a change of data from the last state) • More in another lecture ... Emitter and receiver units of Polhemus Fastrak https://www.cg.tuwien.ac.at/research/publications/1996/mazuryk-1996-VRH/TR-186-2-96-06Paper.pdf Eye Tracking • Eye-tracking techniques may be incorporated to determine the gaze direction • Also, useful for experiments https://www.cg.tuwien.ac.at/research/publications/1996/mazuryk-1996-VRH/TR-186-2-96-06Paper.pdf NAC Eye Mark eye tracker Complete Body Experience • VPL Research DataSuit developed in 1989 • Full-body outfit with sensors for measuring the movement of: – Arms – Legs – Trunk https://en.wikipedia.org/wiki/Virtual_reality VR Interfaces • Keyboard, Mouse, Joystick – 3D Pointing Devices – Spaceball – CyberWand – Ring Mouse – EGG 20/02/2018 15 VR Interfaces . • Whole-hand and body input – 5th glove – Handmaster – ArmMaster – TCAS Dataware Auditory Interfaces • Auralization – 3D simulation of a complex acoustic field • Sonification – Audible display of data • Speech Recognition Haptics • Haptic, from the Greek ἁφή (Haphe), means pertaining to the sense of touch • Haptic technology refers to technology which interfaces the user via the sense of touch by applying forces, vibrations and/or motions to the user Augmented Reality Introduction to Augmented Reality • Augmented Reality – Abbreviation (AR) • Czech Translation… – Rozšířená realita Some Definitions • “A technology that superimposes a computergenerated image on a user’s view of the real world, thus providing a composite view.” • “An enhanced image or environment as viewed on a screen or other display, produced by overlaying computer-generated images, sounds, or other data on a real-world environment.” http://dictionary.reference.com/browse/augmented+reality http://www.oxforddictionaries.com/definition/english/augmented-reality 20/02/2018 16 Another Definition • “… Augmented Reality is a type of virtual reality that aims to duplicate the world's environment in a computer. An augmented reality system generates a composite view for the user that is the combination of the real scene viewed by the user and a virtual scene generated by the computer that augments the scene with additional information. The virtual scene generated by the computer is designed to enhance the user's sensory perception of the virtual world they are seeing or interacting with. The goal of Augmented Reality is to create a system in which the user cannot tell the difference between the real world and the virtual augmentation of it. Today Augmented Reality is used in entertainment, military training, engineering design, robotics, manufacturing and other industries.” http://www.webopedia.com/TERM/A/Augmented_Reality.html Invisible Interfaces Jun Rekimoto, Sony CSL Augmented Reality Concept • The concept of AR is the opposite of the closed world of virtual spaces since users can perceive both virtual and real information • Most AR systems use more complex software approaches compared to VR systems – So, it is harder! Augmented Reality Concept . • The basic theoretical principle is to superimpose digital information directly into a user’s sensory perception rather than replacing it with a completely synthetic environment as VR systems do – In some cases we want it to be very realistic, in some other cases not! VR and AR Similarities • Both technologies process and display the same digital information and often make use of the same dedicated hardware • For example, both an VR and an AR system may be equipped with a head-mounted display (HMD) to visualize the same 3D computer generated model VR and AR Differences • An AR system uses the real world instead of trying to replace it • On the other hand, in a VR system the whole environment is synthetic • The user is completely immersed within a virtual world trying to mimic reality • A VR simulated world does not always have to obey all laws of nature 20/02/2018 17 VR and AR Differences . • The most common problems of VR systems are of emotional and psychological nature including motion sickness, nautia, and other symptoms, which are created by the high degree of immersiveness of the users • Although AR systems are influenced by the same factors the amount of influence is much less than in VR since only a portion of the environment is virtual VR and AR Technology Requirements Billinghurst, M., Clark, A. Lee, G. A Survey of Augmented Reality, Foundations and Trends in Human-Computer Interaction, Vol. 8, No. 2-3 2014 Mann’s Mediated Reality Billinghurst, M., Clark, A. Lee, G. A Survey of Augmented Reality, Foundations and Trends in Human-Computer Interaction, Vol. 8, No. 2-3 2014 Metaverse • Neal Stephenson’s “SnowCrash” • The Metaverse is the convergence of: – virtually enhanced physical reality – physically persistent virtual space • Metaverse Roadmap – http://metaverseroadmap.org/ Metaverse Dimensions • Augmentation technologies that layer information onto our perception of the physical environment. • Simulation refers to technologies that model reality • Intimate technologies are focused inwardly, on the identity and actions of the individual or object • External technologies are focused outwardly, towards the world at large Metaverse Components • Four Key Components – Virtual Worlds – Augmented Reality – Mirror Worlds – Lifelogging 20/02/2018 18 Mirror Worlds • Mirror worlds are informationally-enhanced virtual models of the physical world – Google Earth, MS Street View, Google Maps LifeLogging • Technologies record and report the intimate states and life histories of objects and users – Nokia LifeBlog, Nike+, FitBits Human Factors Contribution of Human Senses • The contribution of each of the five human senses: – Sight................. 70 % – Hearing.............. 20 % – Smell ..................5 % – Touch..................4 % – Taste ...................1 % https://www.cg.tuwien.ac.at/research/publications/1996/mazuryk-1996-VRH/TR-186-2-96-06Paper.pdf Human Factors in VR Human Performance Efficiency Health and Safety Societal Implications Stanney et al., 1998 Human Factors in VR . Will the user get sick in VR? Which tasks are most suitable for users in VR? Which user characteristics will influence VR performance? Will there be negative societal impact from user’s misuse of the technology? What kind of designs will enhance user’s performance in VR? How much feedback from VR can the user process? Will the user perceive system limitations? How should VR technology be improved to better meet the user’s needs? ? Stanney et al., 1998 20/02/2018 19 Simulator Sickness • There are potentially many sources of simulator sickness • Hardware imperfection may contribute to the generation of sickness feeling, because it fails to provide perfect stimuli to human senses • However, there are other crucial design issues: – System latency – Frame rate variations https://www.cg.tuwien.ac.at/research/publications/1996/mazuryk-1996-VRH/TR-186-2-96-06Paper.pdf Dangers of VR/Video Games • Excessive game play can be fatal – Can be worst with immersive VR! • In Korea, where 30% of the population subscribes to online multiplayer games, one man died in 2005 after playing 50 hours (almost non-stop) StarCraft • 3 Chinese died in 2007 after playing more than 50 hours, and 2 died in 2005 Dangers of VR/Video Games . • EverQuest is a 3D online game played by more than 400,000 people • Games can lead to isolation and suicide • Hudson Wooley, an epileptic who was playing 12-hours per day, eventually committed suicide Advantages of VR/Games • People regularly exposed to video-games have improved: – Visual and Spatial attention (C. S. Green, D. Bavelier, Nature, 2003) – Memory (J. Feng et al., Psychol. Sci., 2007) – Mental rotation abilities – Enhanced sensorimotor learning (D. G. Gozli, et al.,Hum. Mov. Sci., 2014) • Extensive video-game practice has also been shown to improve the efficiency of – Movement control brain networks – Visuomotor skills (J. A. Granek, et al., Nerv. Syst. Behav., 2010) Applications Entertainment • In last years W-Industry has successfully brought to the market networked multi-player game systems https://www.cg.tuwien.ac.at/research/publications/1996/mazuryk-1996-VRH/TR-186-2-96-06Paper.pdf Virtuality 1000DS from W-Industries 20/02/2018 20 Medicine https://www.cg.tuwien.ac.at/research/publications/1996/mazuryk-1996-VRH/TR-186-2-96-06Paper.pdf VR in medicine: (left) eye surgery (right) leg surgery Simulations https://www.cg.tuwien.ac.at/research/publications/1996/mazuryk-1996-VRH/TR-186-2-96-06Paper.pdf Advanced flight simulator of Boeing 777: (left) outside view, (right) inside view And Many More • Military • Archaeology • Geography • Etc ... Videos Cyber Tennis (1997) Video https://www.youtube.com/watch?v=Ci6gL0da4R0 Social Phobia Video https://www.youtube.com/watch?v=-4x1t6-UNjo 20/02/2018 21 Virtual Guide Video https://www.youtube.com/watch?v=b6MkXsGViB4 Questions