17/02/2020 1 PA201 Virtual Environments Lecture 1 Introduction to Virtual Reality Fotis Liarokapis liarokap@fi.muni.cz 17th February 2020 Fotis Liarokapis • PhD in Computer Engineering – University of Sussex, UK • MSc in Computer Graphics and Virtual Environments – University of Hull, UK • BSc in Computer Systems Engineering – University of Sussex, UK My Research • Research areas: – Computer Graphics – Virtual Reality – Augmented Reality – Procedural Modeling – Interactive Environments – Serious Games – User studies Contact Details • Email: – liarokap@fi.muni.cz • Telephone: – 549493948 • Office Location: – C411 • Office Hour: – Monday 13:00 to 14:00 Course Details • Prerequisites – Knowledge of computer graphics fundamentals • Lectures – Every Monday – Time: 14:00 to 16:00 – Location: B411 • Lab/Seminar – Every Monday – Time: 16:00 to 17:00 – Location: B311 Course Objectives • Demonstrate an understanding of the main mathematical concepts, hardware and software technologies used in immersive virtual environments • Evaluate different approaches, methodologies and tools focused on virtual reality • Propose virtual environments for both indoor and outdoor environments • Design virtual reality applications for various application domains 17/02/2020 2 Syllabus • Introduction to Virtual Environments • Head Mounted Displays • Other Visualisation Technologies • Virtual reality interaction techniques • Design guidelines • Tracking technologies • Physical simulation • Haptics • Perception Issues for Virtual Environments • Psychological aspects of Virtual Environments (i.e. motion sickness, nausea) • Collaborative Virtual Environments • Software for Virtual Environments (i.e. VR toolkits) • Mobile Virtual Environments • Application Domains • Future of Virtual Environments Teaching Methods • Delivery of the material will be based on – Expositional lectures – Reinforced by computer demonstrations of the application of the material – Video demonstrations Assessment Methods • An assignment about designing, implementing and testing a virtual environment – An essay – Practical assignment • 4 credits awarded Plagiarism and Cheating • If you use an external resource cite it clearly! • Don’t do things that would be considered dishonest... if in doubt ask • Cheating earns you: – Fail in the class – Getting reported to the University – No exceptions Report Structure • Title page • Contents • Abstract (or summary) • Introduction • Background theory • Methodology and results • Conclusions • References • Appendices VR History 17/02/2020 3 First VR Interpretation http://www.bradshawfoundation.com/news/cave_art_paintings.php?id=The-Spanish-Cave-of-Altamira-opens-with-politics Stereoscopic Photos & Viewers • In 1838 Charles Wheatstone’s demonstrated that the brain processes the different 2D images from each eye into a single object of three dimensions – Viewing two side by side stereoscopic images or photos through a stereoscope gave the user a sense of depth and immersion – The later development of the popular View-Master stereoscope (patented 1939), was used for “virtual tourism” – The design principles of the Stereoscope is used today for the popular Google Cardboard and low budget VR head mounted displays for mobile phones http://www.vrs.org.uk/virtual-reality/history.html Stereoscopic Photos & Viewers . • 1838 : The stereoscope (Charles Wheatstone) http://www.vrs.org.uk/virtual-reality/history.html Stereoscope and View-Master • 1849: The lenticular stereoscope (David Brewster) • 1939: The View-Master (William Gruber) http://www.vrs.org.uk/virtual-reality/history.html Link Trainer • In 1929 Edward Link created the “Link trainer” probably the first example of a commercial flight simulator, which was entirely electromechanical – Patented 1931 • Controlled by motors – That linked to the rudder and steering column to modify the pitch and roll – A small motor-driven device mimicked turbulence and disturbances http://www.vrs.org.uk/virtual-reality/history.html Link Trainer . • US military bought six of these devices for $3500 (~ $50,000 today's money) – During World War II over 10,000 “blue box” Link Trainers were used by over 500,000 pilots for initial training and improving their skills http://www.vrs.org.uk/virtual-reality/history.html 17/02/2020 4 Science Fiction Story Predicted VR • In the 1930s, Stanley G. Weinbaum (Pygmalion’s Spectacles) proposed a pair of goggles that let the wearer experience a fictional world through holographics, smell, taste and touch http://www.vrs.org.uk/virtual-reality/history.html Sensorama (1950) • In mid 1950s Morton H Eilig built a single user console called Sensorama that included a stereoscopic display, fans, or emitters, stereo speakers and a moving chair – This enabled the user watch television in 3D Telesphere Mask (1960) • In 1960, Morton Heilig invented the Telesphere Mask (patented 1960) – First example of a head-mounted display (HMD), albeit for the noninteractive film medium without any motion tracking – The headset provided stereoscopic 3D and wide vision with stereo sound http://www.vrs.org.uk/virtual-reality/history.html Headsight HMD (1961) • In 1961, Philco Corporation engineers developed the first Head-Mounted Display (HMD) – Known as the ‘Headsight’ • The helmet consisted of a video screen along with a tracking system – Also linked to a closed circuit camera system • Similar HMD was used later for helicopter pilots Ultimate Display (1965) • In 1965, Ivan Sutherland proposed the ‘Ultimate Display’ – After using this display a person imagines the virtual world very similar to the real world • During 1966, he built an HMD – Was tethered to a computer system Ultimate Display HMD GROPE (1967) • In 1967, Brooks developed force feedback GROPE system 17/02/2020 5 Sayre Glove (1977) • Sandin and Sayre invent a bendsensing glove – The first wired glove • The Sayre Glove, created by Electronic Visualization Laboratory in 1977 – Used light based sensors with flexible tubes with a light source at one end and a photocell at the other – As the fingers were bent, the amount of light that hit the photocells varied, thus providing a measure of finger flexion https://www.evl.uic.edu/entry.php?id=2162 Polhemus FASTRAK Tracker (1979) • FASTRAK tracks the position (X, Y, and Z Cartesian coordinates) and orientation (azimuth, elevation, and roll) • Uses electro-magnetic fields to determine the position and orientation of a remote object https://polhemus.com/ Virtual Reality Birth (1987) • In 1987, Jaron Lanier, founder of the visual programming lab (VPL), coined the term ‘virtual reality’ and developed a range of VR gear – Dataglove (along with Tom Zimmerman) – EyePhone head mounted display • A major development in the area of virtual reality haptics http://www.vrs.org.uk/virtual-reality/history.html VIEW (1989) • The first complete VR system was developed by NASA – “Virtual Visual Environmental Display” • Became “Virtual Interface Environment Workstation” (VIEW) 1989 Sega VR Headset • In 1993, Sega announced the Sega VR headset for the Sega Genesis console – The wrap-around prototypeglasses had head tracking, stereo sound and LCD screens in the visor • Commercial failure! – Developed 4 games for this product – Never made it to the market due to technical development difficulties http://www.vrs.org.uk/virtual-reality/history.html Nintendo Virtual Boy • In 1995, Nintendo Virtual Boy (known as VR-32) was the first ever portable console that could display true 3D graphics • A commercial failure – The following year they discontinued its production and sale • Reasons for this failure were – Lack of colour in graphics (games were in red and black) – Lack of software support – Difficult to use the console http://www.vrs.org.uk/virtual-reality/history.html 17/02/2020 6 The Rig (1999) • Philip Rosedale founds Linden Lab and focuses on developing ‘The Rig’ – A VR device users could wear on their shoulders • Development proves impossible • Linden Labs starts developing virtual worlds – Including ‘Second Life’ https://vrroom.buzz/vr-news/lifestyle/second-life-virtual-real-relationships VR Nowadays Current State of VR • Nowadays VR is moving from the research laboratories to the working environment by replacing ergonomically limited HMD’s with – Projective displays • CAVE and Responsive Workbench – Mobile devices • PDAs, smartphones, tablets, etc – Interaction devices • Wii, Kinect, many others VR Nowadays • $3-5 Billion VR business – Around $150 Billion Graphics Industry – Visualization, simulation, gaming, CAD/CAE, multimedia, graphics arts • Closely aligned with computer games/video games and other apps VR Nowadays . • A number of different expensive devices exist • Target to get full immersion Forecast unit shipments of AR and VR headsets from 2019-2023 (in millions) https://www.statista.com/statistics/653390/worldwide-virtual-and-augmented-reality-headset-shipments/ 17/02/2020 7 https://www.forbes.com/sites/solrogers/2019/06/21/2019-the-year-virtual-reality-gets-real/#40e6d63c6ba9 https://venturebeat.com/2019/10/24/superdata-facebook-sold-180000-oculus-quests-in-q3-2019/ https://nwn.blogs.com/nwn/2019/06/vr-headset-shipments-superdata-2022-forecast.html VR Headsets Google Cardboard • Launched in 2014 • SDK is available for the Android and iOS • Over 5 million Cardboard viewers • Over 1,000 compatible applications https://vr.google.com/daydream/ Open Source Virtual Reality • The Open Source Virtual Reality (OSVR) introduced on January 2015 • VR headset that claims to be open-source hardware and use open-source software – Developed by Razer and Sensics https://www.razerzone.com/osvr 17/02/2020 8 Samsung Gear VR • Released on November 27, 2015 in collaboration with Oculus • Designed to work with Samsung’s flagship smartphones http://www.samsung.com/global/galaxy/gear-vr/ Google Daydream • The Daydream View headset is made with lightweight fabric and launched in 2016 – Discontinued 2019 • Paired with a controller equipped with smart sensors to understand movements and gestures https://vr.google.com/daydream/ PlayStation VR • Project Morpheus launched on October 13, 2016 • Designed to be fully functional with the PlayStation 4 • Tracks 360 degree head movement based on nine positional LEDs on its surface for the PlayStation Camera https://www.playstation.com/en-us/explore/playstation-vr/ Oculus Rift • Released on March 28, 2016 • The Rift has a stereoscopic OLED display, 1080×1200 resolution per eye, a 90 Hz refresh rate, and 110° field of view https://www.oculus.com/ HTC Vive • Released on 5 April 2016 • Refresh rate of 90 Hz, two screens, one per eye, each having a display resolution of 1080x1200 • Laser position sensors, 4.6 by 4.6 m tracking space http://store.steampowered.com/app/358040/ FOVE • FOVE is the first virtual reality headset that utilizes eye tracking – Accuracy of 1/20th of a degree – Will utilize SteamVR's Lighthouse technology for positional tracking https://www.getfove.com/ 17/02/2020 9 Oculus Go • All-in-one headset that’s portable and easy to use – No need to use mobile phones • Lacks cameras and can't track a whole room, or even part of one – Not as powerful as the PC version https://www.oculus.com/go/ HTC VIVE Pro Eye • Improves input and navigation • Eye tracking – Foveated Rendering • Perfect for VR experiments https://www.vive.com/eu/product/vive-pro-eye/ Oculus Quest • Best mobile VR solution to date • Quest' users have more freedom of movement and can virtually (thus physically) walk around with the headset on https://www.oculus.com/quest/ https://www.reddit.com/r/OculusGo/comments/adx2vg/differences_between_oculus_go_quest_and_rift_from/ VR Definitions Virtual Reality Definition • “The computer-generated simulation of a 3D image or environment that can be interacted with in a seemingly real or physical way by a person using special electronic equipment, such as a helmet with a screen inside or gloves fitted with sensors” • Some popular related terms include: – Virtual Environments (VE), Artificial Reality, Telepresence and Cyberspace http://www.oxforddictionaries.com/definition/english/virtual-reality 17/02/2020 10 Telepresence • The use of various technologies to produce the effect of placing the user in another location Artificial Reality • Responsive Environment – An environment where human behavior is perceived by a computer which interprets what it observes and responds through intelligent visual and auditory displays Burdea’s 3 I’s of VR • Interactivity – User impacts world • Immersion – Believing you are there • Imagination – User ‘buying’ into the experience Zeltzer’s Cube https://www.cg.tuwien.ac.at/research/publications/1996/mazuryk-1996-VRH/TR-186-2-96-06Paper.pdf VR Typical Architecture User (programmer, trainee, etc.) Modern VR Systems 17/02/2020 11 Basic Components https://www.cg.tuwien.ac.at/research/publications/1996/mazuryk-1996-VRH/TR-186-2-96-06Paper.pdf VR Rendering • The visual display transformation for VR is much more complex than in standard computer graphics https://www.cg.tuwien.ac.at/research/publications/1996/mazuryk-1996-VRH/TR-186-2-96-06Paper.pdf Stereoscopy • Stereo vision relies on depth-cues based on retinal disparity and therefore may greatly increase the feeling of immersion https://www.cg.tuwien.ac.at/research/publications/1996/mazuryk-1996-VRH/TR-186-2-96-06Paper.pdf Stereoscopic depth-cues: (a) eye convergence, (b) retinal disparity VR Types Characteristic Types • Windows on World • Immersive • Telepresence • Distributed • Mixed Reality • Extended Reality Windows on World (WoW) • Also called monitorbased VR – Similar to games • Using a conventional computer monitor (or mobile device) to display the 3D virtual world 17/02/2020 12 Immersive VR • Completely immerse the user's personal viewpoint inside the virtual 3D world • The user has no visual contact with the physical word • Often equipped with a Head Mounted Display (HMD) or use of CAVE displays Telepresence • A variation of visualizing complete computer generated worlds • Links remote sensors in the real world with the senses of a human operator • The remote sensors might be located on a robot • Useful for performing operations in dangerous environments Teleoperating https://www.cg.tuwien.ac.at/research/publications/1996/mazuryk-1996-VRH/TR-186-2-96-06Paper.pdf Distributed VR • A simulated world runs on several computers which are connected over network and the people are able to interact in real time, sharing the same virtual world Mixed Reality • Milgram coined the term ‘Augmented Virtuality’ to identify systems which are mostly synthetic with some real world imagery added such as texture mapping video onto virtual objects Paul Milgram and Fumio Kishino, 1994 Extended Reality (XR) • Referring to all real-andvirtual combined environments and humanmachine interactions generated by computer technology and wearables – Includes augmented reality (AR), mixed reality (MR) and virtual reality (VR) 17/02/2020 13 VR Paramount VR Immersion • In a typical VR system the user’s natural sensory information is completely replaced with digital information • The user’s experience of a computersimulated environment is called immersion • As a result, VR systems can completely immerse a user inside a synthetic environment by blocking all the signals of the real world VR Paramount • Paramount for the sensation of immersion into virtual reality are – High frame rate • At least 95 fps – Low latency – Pixel persistence lower than 3 ms is required • If not, users will feel sick when moving their head around https://en.wikipedia.org/wiki/Virtual_reality VR Paramount Diagram https://en.wikipedia.org/wiki/Virtual_reality VR Technologies VR Tracking • Many technologies exist – Optical and magnetic and the most dominant • Of course it depends on the application domain • Immersive VR requires the position and orientation of the viewer’s head – Needed for the proper rendering of images • Additionally other parts of body may be tracked – Hands: to allow interaction – Chest or legs: to allow the graphical user representation 17/02/2020 14 VR Tracking . • In general there are two kinds of trackers: – Those that deliver absolute data (total position/orientation values) – Those that deliver relative data (i.e. a change of data from the last state) • More in another lecture ... Emitter and receiver units of Polhemus Fastrak https://www.cg.tuwien.ac.at/research/publications/1996/mazuryk-1996-VRH/TR-186-2-96-06Paper.pdf Eye Tracking • Eye-tracking techniques may be incorporated to determine the gaze direction • Useful for interaction and experiments https://www.cg.tuwien.ac.at/research/publications/1996/mazuryk-1996-VRH/TR-186-2-96-06Paper.pdf NAC Eye Mark eye tracker Complete Body Experience • VPL Research DataSuit developed in 1989 • Full-body outfit with sensors for measuring the movement of: – Arms – Legs – Trunk https://en.wikipedia.org/wiki/Virtual_reality VR Interfaces • Keyboard, Mouse, Joystick – 3D Pointing Devices – Spaceball – CyberWand – Ring Mouse – EEG – ECG VR Interfaces . • Whole-hand and body input – 5th glove – Handmaster – ArmMaster – TCAS Dataware Auditory Interfaces • Auralization – 3D simulation of a complex acoustic field • Sonification – Audible display of data • Speech Recognition 17/02/2020 15 Haptics • Haptic, from the Greek ἁφή (Haphe), means pertaining to the sense of touch • Haptic technology refers to technology which interfaces the user via the sense of touch by applying forces, vibrations and/or motions to the user 360-degree VR • An audiovisual simulation of an altered, augmented or substituted environment that surrounds the user, allowing them to look around them in all directions, just as they can in real life https://photo360tours.com/venice-in-italy-virtual-tour/ Augmented Reality Introduction to Augmented Reality • Augmented Reality – Abbreviation (AR) • Czech Translation… – Rozšířená realita Some Definitions • “A technology that superimposes a computergenerated image on a user’s view of the real world, thus providing a composite view.” • “An enhanced image or environment as viewed on a screen or other display, produced by overlaying computer-generated images, sounds, or other data on a real-world environment.” http://dictionary.reference.com/browse/augmented+reality http://www.oxforddictionaries.com/definition/english/augmented-reality Another Definition • “… Augmented Reality is a type of virtual reality that aims to duplicate the world's environment in a computer. An augmented reality system generates a composite view for the user that is the combination of the real scene viewed by the user and a virtual scene generated by the computer that augments the scene with additional information. The virtual scene generated by the computer is designed to enhance the user's sensory perception of the virtual world they are seeing or interacting with. The goal of Augmented Reality is to create a system in which the user cannot tell the difference between the real world and the virtual augmentation of it. Today Augmented Reality is used in entertainment, military training, engineering design, robotics, manufacturing and other industries.” http://www.webopedia.com/TERM/A/Augmented_Reality.html 17/02/2020 16 Invisible Interfaces Jun Rekimoto, Sony CSL Augmented Reality Concept • The concept of AR is the opposite of the closed world of virtual spaces since users can perceive both virtual and real information • Most AR systems use more complex software approaches compared to VR systems – So, it is harder! Augmented Reality Concept . • The basic theoretical principle is to superimpose digital information directly into a user’s sensory perception rather than replacing it with a completely synthetic environment as VR systems do – In some cases we want it to be very realistic, in some other cases not! VR and AR Similarities • Both technologies process and display the same digital information and often make use of the same dedicated hardware • For example, both an VR and an AR system may be equipped with a head-mounted display (HMD) to visualize the same 3D computer generated model VR and AR Differences • An AR system uses the real world instead of trying to replace it • On the other hand, in a VR system the whole environment is synthetic • The user is completely immersed within a virtual world trying to mimic reality • A VR simulated world does not always have to obey all laws of nature VR and AR Differences . • The most common problems of VR systems are of emotional and psychological nature including motion sickness, nautia, and other symptoms, which are created by the high degree of immersiveness of the users • Although AR systems are influenced by the same factors the amount of influence is much less than in VR since only a portion of the environment is virtual 17/02/2020 17 VR and AR Technology Requirements Billinghurst, M., Clark, A. Lee, G. A Survey of Augmented Reality, Foundations and Trends in Human-Computer Interaction, Vol. 8, No. 2-3 2014 Mann’s Mediated Reality Billinghurst, M., Clark, A. Lee, G. A Survey of Augmented Reality, Foundations and Trends in Human-Computer Interaction, Vol. 8, No. 2-3 2014 Metaverse • Neal Stephenson’s “SnowCrash” • The Metaverse is the convergence of: – virtually enhanced physical reality – physically persistent virtual space • Metaverse Roadmap – http://metaverseroadmap.org/ Metaverse Dimensions • Augmentation technologies that layer information onto our perception of the physical environment. • Simulation refers to technologies that model reality • Intimate technologies are focused inwardly, on the identity and actions of the individual or object • External technologies are focused outwardly, towards the world at large Metaverse Components • Four Key Components – Virtual Worlds – Augmented Reality – Mirror Worlds – Lifelogging Mirror Worlds • Mirror worlds are informationally-enhanced virtual models of the physical world – Google Earth, MS Street View, Google Maps 17/02/2020 18 LifeLogging • Technologies record and report the intimate states and life histories of objects and users – Nokia LifeBlog, Nike+, FitBits Human Factors Contribution of Human Senses • The contribution of each of the five human senses: – Sight................. 70 % – Hearing.............. 20 % – Smell ..................5 % – Touch..................4 % – Taste ...................1 % https://www.cg.tuwien.ac.at/research/publications/1996/mazuryk-1996-VRH/TR-186-2-96-06Paper.pdf Human Factors in VR Human Performance Efficiency Health and Safety Societal Implications Stanney et al., 1998 Human Factors in VR . Will the user get sick in VR? Which tasks are most suitable for users in VR? Which user characteristics will influence VR performance? Will there be negative societal impact from user’s misuse of the technology? What kind of designs will enhance user’s performance in VR? How much feedback from VR can the user process? Will the user perceive system limitations? How should VR technology be improved to better meet the user’s needs? ? Stanney et al., 1998 Simulator Sickness • There are potentially many sources of simulator sickness • Hardware imperfection may contribute to the generation of sickness feeling, because it fails to provide perfect stimuli to human senses • However, there are other crucial design issues: – System latency – Frame rate variations https://www.cg.tuwien.ac.at/research/publications/1996/mazuryk-1996-VRH/TR-186-2-96-06Paper.pdf 17/02/2020 19 Dangers of VR/Video Games • Excessive game play can be fatal – Can be worst with immersive VR! • In Korea, where 30% of the population subscribes to online multiplayer games, one man died in 2005 after playing 50 hours (almost non-stop) StarCraft • 3 Chinese died in 2007 after playing more than 50 hours, and 2 died in 2005 Dangers of VR/Video Games . • EverQuest is a 3D online game played by more than 400,000 people • Games can lead to isolation and suicide • Hudson Wooley, an epileptic who was playing 12-hours per day, eventually committed suicide Advantages of VR/Games • People regularly exposed to video-games have improved: – Visual and Spatial attention (C. S. Green, D. Bavelier, Nature, 2003) – Memory (J. Feng et al., Psychol. Sci., 2007) – Mental rotation abilities – Enhanced sensorimotor learning (D. G. Gozli, et al.,Hum. Mov. Sci., 2014) • Extensive video-game practice has also been shown to improve the efficiency of – Movement control brain networks – Visuomotor skills (J. A. Granek, et al., Nerv. Syst. Behav., 2010) Applications Entertainment • In last years W-Industry has successfully brought to the market networked multi-player game systems https://www.cg.tuwien.ac.at/research/publications/1996/mazuryk-1996-VRH/TR-186-2-96-06Paper.pdf Virtuality 1000DS from W-Industries Medicine https://www.cg.tuwien.ac.at/research/publications/1996/mazuryk-1996-VRH/TR-186-2-96-06Paper.pdf VR in medicine: (left) eye surgery (right) leg surgery 17/02/2020 20 Simulations https://www.cg.tuwien.ac.at/research/publications/1996/mazuryk-1996-VRH/TR-186-2-96-06Paper.pdf Advanced flight simulator of Boeing 777: (left) outside view, (right) inside view Cultural Heritage https://imareculture.eu/ And Many More ! • Military • Manufacturing • Geography • Education • Etc ... Videos Cyber Tennis (1997) Video https://www.youtube.com/watch?v=Ci6gL0da4R0 Social Phobia Video https://www.youtube.com/watch?v=-4x1t6-UNjo 17/02/2020 21 Virtual Guide Video https://www.youtube.com/watch?v=b6MkXsGViB4 More videos • https://www.youtube.com/watch?v=b6MkXsG ViB4 • https://www.youtube.com/watch?v=eCvs_pk RAl4 • https://www.youtube.com/watch?v=rDjYa5LP n5s Questions