David Kuťák, 27th of March 2024 Speaker Introduction •PhD-in-progress-soon-to-be-finished •Here at FI MUNI •Virtual reality & molecular visualization •Especially DNA nanotechnology :) • •Three main areas of interest •Virtual reality •Molecular visualization & modeling •Games (development) • 2 • Speaker Introduction in Images 3 • A picture containing text Description automatically generated Fun fact •Two headsets owned now •PlayStation VR2 •Meta Quest 3 • •(1) One for games •(2) Other for (game) development • •Point (2) did not yet happen :D • 4 • Outline 5 • A person wearing a garment Description automatically generated with low confidence A person wearing a virtual reality headset Description automatically generated A person wearing a virtual reality headset Description automatically generated with medium confidence A person wearing a virtual reality headset Description automatically generated with medium confidence •Images generated with craiyon.com Q&A •Raise your hand if you have experience with VR 6 • Feel free to share what you think about it 7 • • Why to talk about VR? 8 • Young businessman sitting palm on face Young businessman thumbs up •It can help you or other people… Why to talk about VR? 9 • Young businessman sitting palm on face •… or it may not, as it is no magical formula. Young businessman sitting palm on face I was just sad, now I am also poor. A green alligator with red teeth Description automatically generated Why to talk about VR? 10 • •VR is a technology and we believe that this technology may benefit us. If this is not true, why to even bother? Woman using virtual reality headset in neon lights What is a Virtual Reality? 11 What is a Virtual Reality? • 12 • What is a Virtual Reality? •Digital experience feeling like reality both audiovisually and interaction-wise 13 • What is not a Virtual Reality? 14 • University lecture What is not a Virtual Reality? 15 • Counter Strike on a desktop PC. Or could it be? “Virtual reality is a simulated experience that employs pose tracking and 3D near-eye displays to give the user an immersive feel of a virtual world.” (Wikipedia) “images and sounds created by a computer that seem almost real to the user, who can interact with them by using sensors” (Oxford Learner’s Dictionary) “a computer-generated digital environment that can be experienced and interacted with as if that environment were real.” (The VR Book by Jerald) “an artificial environment which is experienced through sensory stimuli (such as sights and sounds) provided by a computer and in which one's actions partially determine what happens in the environment” (Merriam-Webster) Badge Cross with solid fill Badge Tick1 with solid fill Badge Question Mark with solid fill Badge Question Mark with solid fill What is not a Virtual Reality? 16 • This is dangerous. Badge Question Mark with solid fill What is not a Virtual Reality? 17 • •There are also other „realities“ •The definitions may differ • •Augmented reality •Augments digital data on top of real world data • •Mixed reality •Blends physical and digital worlds •Sometimes just used to refer to „both VR and AR“ • •Extended reality •Umbrella term for all „non-real realities“ In this talk, we focuses solely on virtual reality. Immersion x Presence •Two common „VR terms“ •Wikipedia definition of VR: „(…) immersive feel of a virtual world“ • •Immersion •Technical aspect •The extent to which the technology makes you feel surrounded by the virtual world •Aids to achieve the „presence“ •Presence •Subjective aspect •The feeling of „being there“ at the given moment • 18 • Immersion x Presence 19 • Immersion x Presence 20 • Immersion x Presence 21 •But what if… Beetle with solid fill Beetle with solid fill Beetle with solid fill Beetle with solid fill Beetle with solid fill Beetle with solid fill Beetle with solid fill Motion sickness •You may feel sick or dizzy (or even vomit) •Occurs when there is a conflict between senses •Eyes: “I am moving” •Body/inner ear: “I am sitting” •à problem •You may experience when using VR •Depends on each individual, •and the application. 22 • CDN media Virtual Reality – Hardware through time •Head-mounted displays throughout the history 23 • Virtual Reality – Sword of Damocles •1968 •Attached to the ceiling •Head-tracking •Weight 24 • Virtual Reality – VIEW by NASA •1980s •Steroscopic display •Proper „HMD“ •Interaction gloves •3D sound •Voice recognition 25 • Virtual Reality – Sega & Nintendo •1990s •“Failures“ •Sega à prototype •Nintendo à bad sales 26 • 1*h8qwxQYmzMhz6kQ8wb0Usw.webp (1400×1778) Virtual Reality – Nintendo Virtual Boy Commercial 27 •1995 at its best :) (1995) Nintendo Virtual Boy Commercial •I recommend to watch at home J Nintendo continued •2019 •Another „shot“ at VR with Nintendo Switch •VR is part of „Labo“ cardboard kit •Very cheap, 3DOF •Accepted more positively than Virtual Boy… •… but no „major breakthrough“ • • 28 • Virtual Reality – CAVE 29 • Cave at the beach Virtual Reality – CAVE •1990s •Projections on walls •Stereoscopic glasses •Used to be popular •à natural collaboration •à being replaced by HMDs 30 •Cave Automatic Virtual Environment • Virtual Reality Today •VR „exploded“ in 2014 when Facebook (Meta) bought Oculus 31 •(for most people, as well as this lecture) fee_325_225_png (325×225) Really exploded? Virtual Reality Today – Market •Various data about actual users, reliability is questionable •Right plot: estimate of active users on Steam 32 • Virtual Reality Today – HMDs since 2013 • 33 • Source: VR.Space Virtual Reality Today – Components 34 • Virtual Reality Today – Components •Display(s) – count, type, resolution, refresh rate 35 • Lenses FD234FF9ABBE23A32687F7809C4512B5C80FFE02 (2000×1333) Virtual Reality Today – Components 36 • •Types of controll(er)s Worth remembering: if you develop a VR application for users with different HMDs, you need to take into account their different interaction possibilities. Virtual Reality Today – Components 37 • Virtual Reality Today – Components 38 • Virtual Reality Today – Components 39 • Outline 40 • A person wearing a garment Description automatically generated with low confidence A person wearing a virtual reality headset Description automatically generated A person wearing a virtual reality headset Description automatically generated with medium confidence A person wearing a virtual reality headset Description automatically generated with medium confidence •Images generated with craiyon.com Joke Time 41 • • VR … what is it even good for? 42 • VR in automotive industry •VR since 1990s •HTC Vive since 2015 •Combined with physical model of car‘s interior •Faster iterations than with real cars •Easy to simulate real driving experience 43 •BMW use case VR in automotive industry 44 •Honda Virtual Reality Exposure Therapy (VRET) •Treatment of mental health issues •Phobias •Posttraumatic stress disorder (PTSD) •Social anxiety •Brain injuries •Etc. • •Landstuhl Regional Medical Center •VRET treatment of soldiers with PTSD •„(…) extremely effective treatment because it is a patient's personalized reality that they learn to process, control and regulate.“ 45 • Virtual Reality Exposure Therapy (VRET) 46 •Targeting arachnophobia Arachnophobia VR in architecture •VR is popular in this area for visualization of buildings and their designs • •Ennead Architects company •Shanghai Planetarium visualization •Not just space itself but also most-lit locations • • 47 • VR in training •Used by multiple companies to train their employees • • Bank of America •VR training for 50,000 employees •Five different VR simulations •Relationships with clients •Navigating difficult conversations •Responding with empathy 48 • VR in training 49 • VR in training •Military or law enforcement training • • 50 • VR in entertainment •YouTube, Disney, … 51 •360° Movies VR in entertainment 52 •Games VR in entertainment •Large VR market (unsurprisingly) 53 •Games VR in entertainment 54 •Games • VR in entertainment 55 •Games • VR in education •Very „tempting“ domain for VR 56 • VR in education 57 • • •Reported applications and motivations of papers applying VR to education •Increased immersion •Increased motivation •Deeper learning •Increased enjoyment •Collaboration • •Kavanagah et al. (2017) VR for collaboration •Higher immersion à more personal and natural experience •Became more relevant during COVID19 •distance learning, online conferences, … 58 •Events, meetings, and conferences VR for animals aka „true creativity“ •Turkish farmer use(s|d) VR with his cows as it seemed (call it a „preliminary data“) to increase the amount of milk produced 59 •Because … why not Why These Cows Are Wearing VR Glasses Bio(Vis) and Med(Vis) examples 60 • •Several existing approaches and applications •Molecular structure related to function •Increased depth perception may help to better understand some characteristics Molecular Visualization in VR 61 • Molecular Modeling in VR •Tries to combine VR visualization possibilities with its (potentially) more natural interactions 62 • Molecular Visualization & Modeling in VR 63 • Molecular Dynamics in VR •Seeing how molecules behave •Possibly also modeling supported by interactive MD 64 • 64225415-0c58-6ef2-4b29-f6e78a01e460_3 (1259×756) Molecular Dynamics in VR Mixed reality manipulation of neuraminidase + oseltamivir using Narupa iMD-VR 65 •Interacting with MD & docking (in Narupa) Learning about molecules in VR •Biology education is a large area in VR MolVis 66 • Learning about molecules in VR Peppy VR v059 demo reel V2 67 •Peppy – learning about polypeptide structure Nice feature of Peppy: works both with and without VR Collaboration, molecules, and VR •Similar to general VR collaboration •may facilitate discussions about the topic, and •increase the learning or research outcomes. 68 • Surgical planning 69 •CT, MRI, ... scans visualized in VR (by ImmersiveView) • Anatomy education 70 • Generic (volumetric) medical data visualization 71 •Imaging Reality Imaging Reality Collaborative medical data visualization 72 • • Benefits and drawbacks of (HMD) VR 73 •When compared to regular „desktop interface“ Q&A: Can you come up with additional benefits or disadvantages? How to develop for VR? •I would suggest to consider any of these three ways: 74 • How to develop for VR? •Good support for majority of modern VR headsets •Unity & C# are quite user friendly :) •XR Interaction Toolkit •Developed by Unity •Good starting point 75 •Unity How to develop for VR? •Also good support for majority of modern VR headsets •Not as user friendly as Unity •Unreal licensing is more „clear“ 76 •Unreal How to develop for VR? •Web framework •Built on WebXR and Three.js •Web app à No installation of application nedded •However, WebXR support depends on the user‘s browser •Performance and some features might be limited 77 •A-FRAME VR in the future •Two key factors: •Hardware and its price •Availability of content •Makes sense to further explore where it is good (and where not) •How to interact with 2D content or represent it more “3D friendly“? •How to handle text input? •Build large „VR only“ platforms or expand current applications with „VR viewer“? Or find a good combination? •à Converting concepts to real usable applications 78 • Concluding this section … •VR has advantages •VR has disadvantages •You should try it :) •But please do not try the explosive headset • 79 • Core thing to remember: pros >> cons to convince people to actually use it Thank you for your attention! References •The VR Book: Human-Centered Design for Virtual Reality (Jason Jerald) •https://teslasuit.io/blog/history-of-virtual-reality-ultimate-guide/ •https://veative.com/degrees-of-freedom-3dof-vs-6dof/ •https://www.press.bmwgroup.com/global/article/detail/T0258902EN/bmw-opts-to-incorporate-htc-vive-v r-headsets-and-mixed-reality-into-the-development-of-new-vehicle-models-computer-images-instead-of- laboriously-constructed-draft-models:-greater-flexibility- •https://www.wired.com/2016/11/vr-totally-changing-architects-dream-buildings/ •https://www.army.mil/article/144048/virtual_reality_exposure_therapy_helps_resolve_ptsd •https://vrscout.com/news/vr-simulation-allows-police-to-train-using-real-tasers/ •https://www.nytimes.com/2017/06/22/technology/personaltech/virtual-reality-video-games.html •https://is.muni.cz/auth/th/s4fso/PhD_Thesis_Proposal_oneside.pdf •https://github.com/davous267/molecular-visualization-in-virtual-environments#purely-immersive-vr •https://link.springer.com/article/10.1007/s10278-021-00480-z •https://link.springer.com/chapter/10.1007/978-3-030-62655-6_11 •https://ieeexplore.ieee.org/document/9288058 81 • Applications you may try … •Nanome (molvis/molmod) 82 • Medicalholodeck (medvis) Alternatively: MedicalImagingVR UnityMol (molvis) CellPaint VR (drawing of cell env.) GORN (brutal game) Blocks (creativity)