Taking (BioMedical) Visualization Off the Screen Renata Georgia Raidou Research Unit of Computer Graphics Institute of Visual Computing & Human-Centered Technology TU Wien, Austria 2Source: Siemens Healthcare The Anatomy Lesson of Dr. Nicolaes Tulp – Rembrandt (c. 1632) 2D 3D Da Vinci (15th c.) Vesalius (16th c.) Ou Xi Fang (10th c.) 2D 5 Anatomical Venus (18th c.) 3D Moulagen (18th c.) Wax Sculpture (18th c.) The Mother (18th c.) Papier-Mâché Brain (18th c.) 6 3D De Fragonard (18th c.) Von Hagens Body World Exhibit (21st c.) 16th century 1895 X-rays 1970s CT, MRI MedVis DataPhys 9Source: Siemens Healthcare Data Physicalization is… …a rich and vast research area that studies the use of physical artifacts to convey data. Ang et al. 2019, C&G Dragicevic et al. 2021, HCI Handbook http://dataphys.org/list/gallery/ Not only for medical data! 11 Categories Ambient Display Pixel Sculpture Object Augmentation Wearable Visualization Data Sculpture Modality Display 12 VirtualiTee Gillet et al. 2005, Structure Watson and Crick, DNA Structure Discovery 1953 A 3D printed heart (costs around $700-1000) expensive complex time-costly From Data To Sculpture • Extension of info vis pipeline (Jansen et al. 2013, CHI) • Adds information extraction on user side: • How is information processed? • What interactions are possible? Two Recent Examples of Data Sculptures Schindler et al. 2020, IEEE Vis Schindler et al. 2022, CGF 18 The Anatomical Edutainer Marwin Schindler, Hsiang-Yun Wu, Renata G. Raidou Best Short Paper at Motivation 20 Motivation 21 1st Concept: Light Absorption 22 1st Concept : Light Absorption 23 2nd Concept: Papercrafts 24 Benefits 25 26 27 28 2D with foils 3D with lights 29 internal light external light 30 Nested Papercrafts for Anatomical and Biological Edutainment Marwin Schindler, Thorsten Korpitsch, Renata G. Raidou, and Hsiang-Yun Wu Nested Papercrafts 32 Motivation 33 Amao Chan Art Studio https://amaochan.work/a Contribution 34 • Workflow for Computer-Aided Generation of Nested Papercraft Physicalizations Contribution 35 • Workflow for Computer-Aided Generation of Nested Papercraft Physicalizations • Optimal Visibility of Nested Structures 36 • Workflow for Computer-Aided Generation of Nested Papercraft Physicalizations • Optimal Visibility of Nested Structures • Strategy for Generating Realizable Papercrafts Contribution Workflow - Requirements • (R1) No Domain Knowledge 37 Workflow - Requirements • (R1) No Domain Knowledge • (R2) 3D Mesh Model Input 38 Workflow - Requirements • (R1) No Domain Knowledge • (R2) 3D Mesh Model Input • (R3) No Complex User Interaction 39 Workflow - Requirements • (R1) No Domain Knowledge • (R2) 3D Mesh Model Input • (R3) No Complex User Interaction • (R4) Easy-to-Assemble Engaging Physical Twin 40 Workflow - Requirements • (R1) No Domain Knowledge • (R2) 3D Mesh Model Input • (R3) No Complex User Interaction • (R4) Easy-to-Assemble Engaging Physical Twin • (R5) Affordable and Available Resources 41 Workflow - Input • Registered Meshes from Anatomical and Biological Models 42 BodyParts3D: 3D Structure Database for Anatomical Concepts, Mitsuhashi et al., 2009. Workflow - Approximation • Feasibility of Reconstruction • Limit Complexity • Approximation by Subdivision of Bounding Box (BBox) • Calculate BBox • Subdivide BBox • Move Vertices to Closest Point on Surface 43 Workflow - Hierarchy • Calculate Hierarchy Using Approximated Input • Check if All Vertices of Mesh A are Inside Mesh B • 3D Iso-oriented Box Intersection Tests on Edges 44 Workflow - Viewpoint • Provide Optimal Viewpoint on Inner Levels • Maximum Viewpoint Entropy 45 Viewpoint Selection Using Viewpoint Entropy, Vazquez et al., 2001. Workflow - Cutting • Subtract Inner Mesh • Cut Each Level Using the Optimal Viewpoint 46 Workflow - Stability • Subtract Inner Mesh • Cut Each Level Using the Optimal Viewpoint • Ensure a Feasible and Stable Result 47 Forward Dynamics - The Articulated-Body Method. Featherstone, 1987. Workflow - Projection • Create a Texture for Each Anatomical Structure 48 Workflow - Projection • Create a Texture for Each Anatomical Structure • Inflate Inner Parts • Clip Distant Features • Project on BBox 49 Inflation Clipping BBox Workflow - Projection • Create a Texture for Each Anatomical Structure • Blend Textures for Each Level 50 The Anatomical Edutainer, Schindler et al., 2020. Workflow - Unfolding • Calculate Dual Graph • Calculate Minimum Spanning Tree 51 Simulated Annealing to Unfold 3D Meshes and Assign Glue Tabs, Korpitsch et al., 2020. Workflow - Unfolding • Calculate Dual Graph • Calculate Minimum Spanning Tree • Add Minimum Number of Gluetabs 52 Simulated Annealing to Unfold 3D Meshes and Assign Glue Tabs, Korpitsch et al., 2020. 53 Results: Plant Cell • 60 Faces - 29 Minutes 54 Results: Human Anatomy • 98 Faces - 42 Minutes 55 Evaluation • Initial User Study with 10 Participants • Constructing the Head Model 56 Evaluation • Initial User Study with 10 Participants • Constructing the Head Model • Additional Indicators for Mountain/ Valley Folds 57 Evaluation • Initial User Study with 10 Participants • Constructing the Head Model • Additional Indicators for Mountain/ Valley Folds • New Insights in an Entertaining Way 58 Evaluation • Initial User Study with 10 Participants • Constructing the Head Model • Additional Indicators for Mountain/ Valley Folds • New Insights in an Entertaining Way • Unprofessional in Patient Communication 59 Evaluation • Initial User Study with 10 Participants • Constructing the Head Model • Additional Indicators for Mountain/ Valley Folds • New Insights in an Entertaining Way • Unprofessional in Patient Communication • Children Education 60 Limitations Distortions From Mesh Abstraction 61 Limitations Size of inner substructures too small 62 Limitations Configuration of Inner Structures 63 Limitations Construction Time 64 Future Work • Further Domains of Use • Machine Assisted Construction • Full-size Papercrafts • Evaluation of the Entire Fabrication Process 65 Thank you! This work would have not been possible without: Marwin Schindler Thorsten Korpitsch Hsiang-Yun Wu Daniel Pahr Eduard Gröller 66