Visualization of Medical Data Katarína Furmanová PA214 / Visualization II Goals of Medical Visualization • Education • Diagnosis • Treatment planning • Treatment guidance • Doctor-patient communication 2De Humani Corporis Fabrica by Vesalius, 1543 In This Lecture • A brief tour through the zoo of medical data and their visualization! Saginaw Future Inc., CC BY 2.0, via Wikimedia Commons 3 Scales Atoms Genes Organs PopulationsOrganisms Cells Tissue Molecules Interactions 4 Tissue Samples of Human Tissue • Excretion: urine, stool, mucus, vomit, saliva • Excision: • Puncture: blood, lung fluid, amniotic fluid, … • Scraping: cheek, mouth, uterine cervix, … • Biopsy: • Taking out piece of tissue as a whole • Bone marrow, brain, skin, liver, …. 6 Resulting Data • Images (e.g., histopathology) • Tables (e.g., blood values) CC BY 2.0, https://commons.wikimedia.org/w/index.php?curid=437645 Human lung tissue stained with hematoxylin and eosin 7 Organ Medical Imaging Martin Tornai, CC BY 4.0, via Wikimedia Commons 9 X-ray • Radiation: electromagnetic waves (ionizing!) • Varying tissue absorption leads to image contrast • Standard, Fluoroscopy, Angiography, Mammography CardioNetworks: Jer5150, CC BY-SA 3.0, via Wikimedia CommonsMikael Häggström, M.D. CC0, via Wikimedia Commons 10 Computed Tomography (CT) • X-ray tube rotating around the body • Reconstruction: stack of 2D slices • Intensity: Hounsfield Units (HU) Dr1rrb, CC BY-SA 4.0, via Wikimedia Commons Tissue HU Air -1000 Fat -120 to -90 Water 0 Bone (cortical) 500 to 1900 11 Magnetic Resonance Imaging (MRI) • Magnetic field, gradient, and radiofrequency pulse (non-ionizing!) • fMRI, MR Spectroscopy, MR angiography, … Ofirglazer at English Wikipedia, CC BY-SA 3.0, via Wikimedia Commons Wei-yuan Huang, Gang Wu, Feng Chen, Meng-meng Li and Jian-jun Li, CC BY 4.0, via Wikimedia Commons 12 Medical Ultrasound • High frequency sound waves (inaudible) • Pulses of ultrasound sent through tissue and echo caught • 2D, time-varying, 3D, Doppler (blood flow), … Fruehaufsteher2, CC BY-SA 3.0, via Wikimedia CommonsMme Mim, CC BY-SA 3.0, via Wikimedia Commons 13 Nuclear Imaging: PET/SPECT • Patient generates radiation through radioactive tracer injection • Scintigraphy (2D), Single-Photon Emission Computed Tomography SPECT (3D), Positron Emission Tomography (PET) (3D) • Functional imaging: metabolic processes, blood flow, regional chemical composition, absorption… CT PET Rauscher, Isabel; Maurer, Tobias; Fendler, Wolfgang P.; Sommer, Wieland H.; Schwaiger, Markus; Eiber, Matthias, CC BY 4.0, via Wikimedia Commons 14 Hybrid: PET/CT, PET/MR, SPECT/CT, … • Integrated hardware to acquire multiple imaging modalities Myohan (talk) (Uploads), CC BY 3.0, via Wikimedia Commons 15 Organism Whole-Body MRI, CT, PET • Not very commonly done • Indications: trauma or metastases detection 17 Cryosection • Very large scale histology • Frozen embedded sections • No microscopes needed! This image was created by a US government project in the National Library of Medicine, a branch of NIH. The original image was modified by user:Looie496, Public domain, via Wikimedia Commons 18 Visible Korean Human female 19 Resolution: 5616 x 2300 0.2 mm between every slice Electronic Health Records (EHR) • Collection of patient health information • For example: medical history, medication, test results, allergies, radiology images, personal statistics, billing information 20 Population Screening • Discovering disease among population without symptoms • Systematic testing of individuals at risk to benefit from further investigation or preventative measures • Examples: cholesterol measurements (cardiovascular disease risk), mammography (breast cancer) 22 Cohort Studies • Medical research studying a large number of subjects over time (longitudinal) • Cohort: a group of people who share a defining characteristic • Examples: cohort people born in Rotterdam in 1980-1985, cohort of gynecological cancer patients, … • Data: self-reported interviews, medical examinations, imaging Preim, Bernhard, Paul Klemm, Helwig Hauser, Katrin Hegenscheid, Steffen Oeltze, Klaus Toennies, and Henry Völzke. "Visual analytics of image-centric cohort studies in epidemiology." In Visualization in medicine and life sciences III, pp. 221-248. Springer, Cham, 2016. 23 Public Health • Studying the population in order to prevent disease • Aim: encouraging behavior and policy chance, limit acute disease outbreak, reduce chronic diseases and injuries • Data over time and locations (spatio-temporal) Bernhard Preim and Kai Lawonn. "A survey of visual analytics for public health." In Computer Graphics Forum, vol. 39, no. 1, pp. 543-580. 2020. 24 Visualization Approaches for Spatial Data Spatial Data Visualization Challenges in Medicine • Artifacts, noise, uncertainty • Integration with data at different scales • When combining modalities: registration, … • Occlusions The steps of the workflow of radiotherapy planning [Raidou et al. 2019] Spatial Data Visualization Approaches 27 Surface, Spatial Features Volume Rendering IllustrativeReformation, Abstraction Slices Slices 28 Axial Sagittal Coronal [Lesjak et al., 2018] • Baseline visualization Slices 29 [Raidou et al., 2016] • Additional data: colormaps, glyphs • Problem with occlusions – linked views [Mörth et al., 2020] Surface – Construction 30 • Result of thresholding or segmentation • Marching cubes, Lorensen & Cline 1987 Marching cubes for iso-surface generation Jmtrivial via Wikimedia Commons Dake via Wikimedia Commons CC BY SA 2.5 Surface – Smoothing 31 • Basic low pass filters lead to shrinkage -> Imprecision Comparison of multiple smoothing algorithms applied to MC (a) [Wei et al., 2015] Surface – Construction/Smoothing 32 • Constrained elastic surface nets, Gibson 1998 • Start with binary mask border –> relaxation constrained to size of one voxel Marching cubes vs. Surface nets [Bruin et al., 2000] Gibson, 1998 Surface – Construction 33 • Reconstruction from camera (e.g., endoscopy) • Point cloud extraction + triangulation/fitting [Ma et al., 2021] Surface – Mapping Additional Data 34 [Iterrante et al., 1997] Tumor + iso-surface of radiation dose with curvature-oriented glyphs Pelvic organs – shape variability • Colormaps, glyphs, limited by occlusions & shading Comparison of 2 faces Illustrative Approaches [Lawonn et al., 2013] [Carl et al., 2019] • Hatching, stippling, silhouettes Projections and Reformations 36[Raidou et al., 2018] • Flattening to deal with occlusion, better comparison [Neugebauer et al., 2009] Projections and Reformations 37[Mörth et al., 2019] • Pose standardization Vasculature – Depth Perception 38 Chromadepth Psedo-Chromadepth Hatching, outlines, shadows Fog [Behrendt et al., 2017] [Kreiser et al., 2021] [Titov, 2020] [Lawonn et al., 2015] Void space surfaces 39 Composition (d) of stress & distance mappings (a, b) using a mask based on Fresnel reflection effect (c) [Behrendt et al., 2017] Wall shear stress on a vessel Vasculature – Additional Data 40 [Meuschke et al., 2016] Vasculature – Blood Flow • Fluid dynamics simulations, 4D PC-MRI • Arrows, streamlines, stream surfaces with textures or glyphs 41 Projections and Reformations Ofirglazer at English Wikipedia, CC BY-SA 3.0, via Wikimedia Commons [Kanitsar et al., 2002] • Curved planar reformations • Projection, stretching, straightening Projections and Reformations 42 [Angelelli & Hauser, 2011] [Borkin et al., 2011] Brain Connectivity 43 • Based on diffusion MRI • Glyph based representations vs. fibers tracking Glyph representations of diffusion tensor [Schultz & Vilanova, 2018] Brain Connectivity – Fiber Tracking 44 • Streamlines/stream tubes Line rendering colored according to local tangent direction Illuminated lines Ambient Occlusion [Eichelbaum et al., 2013] Fiber tract contraction + white halos [Everts et al., 2013] Neuronal Connectivity 45 [Al-Awami et al., 2014] Subway map metaphor for neuronal connectivity • Abstraction Direct Volume Rendering 46 • Projecting 3D volume onto 2D screen • Typically using ray-casting • Basic approach – maximum/average intensity projection – not very “3D” Thetawave via Wikimedia Commons CC-BY-SA-3.0 Mikael Häggström via Wikimedia Commons CC-BY-SA-2.1 Transfer Functions 47 • Define opacity and color of each voxel • 1D – classification based on material density (intensity) of the voxel • Multidimensional – additional attributes, e.g., gradient or curvature [Ljung et al., 2016] Illustrative Techniques 48 [Ganglberger et al., 2019] [Bruckner & Gröller, 2007] • Silhouettes, hatching , style transfer Depth Perception 49 [Schott et al., 2011] [Svakhine et al., 2009] Depth of field simulation Depth dependant silhouettes and blurring Lighting – Cinematic Rendering 50 [Dappa et al., 2016] • Conventional VR – local illumination • Cinematic rendering – simulates physical light scattering Smart Visibility 51 • Peel aways • Exploded views • Importance driven TFs • Lenses [Stoppel & Bruckner, 2018] [Bruckner & Gröller, 2006] [Bruckner et al., 2005] Visualization Approaches for Non-Spatial Data Visualization Challenges • Datasets too large for visual inspection of all individuals/data points • Combination with statistical and/or machine-learning based analysis • Data quality: missing values, standardization, … 53 Electronic Health Records 54 • Visualizing patient histories – LifeLines by Plaisant & Schneiderman, 2003 LifeLines [Wongsuphasawat et al., 2011] Electronic Health Records 55 • Visualizing medical texts Doccurate [Sultanum et al. 2018] Public Health 56 Martin Krzywinski: A pandemic of bad charts https://www.youtube.com/watch?v=_YGmfsKL8N8&ab_channel=HelenaKlaraJambor [Afzal et al., 2020] Cohort Studies 57 • Feature extraction, dimensionality reduction, clustering, … [Alemzadeh et al., 2017] Visual Analytics – Bringing It All Together 59Image by Renata Raidou 60 Spatial data, abstractions, infovis techniques, statistical analysis, clustering, …. Visual Analystics Closing Remarks Evaluation & Translation to Practice • Appropriateness depends on target user and task • Requirement analysis • Task completion evaluation • E.g., which of two points on blood vessel tree is closer • Correctness – essential for decision making • Time • User engagement – e.g., for communication tasks • Long-term evaluation might be necessary but rarely done • Limited adoption to clinical practice 62Dooffy, CC0, via Wikimedia Commons https://gdpr-info.eu/issues/personal-data/ Open Challenges • Bridging data across multiple scales (omics + medical data) • Uncertainty in medical data • Data size • Explainable AI • Patient empowerment • Constant adaptation to new data/technologies 63 Medical Data = Sensitive Personal Data • Remember that this belongs to people, treat it with respect • Follow regulations, check if you aren’t sure • Medical collaborators may have additional restrictions 64Dooffy, CC0, via Wikimedia Commons https://gdpr-info.eu/issues/personal-data/ Acknowledgement • Noeska Smit, University of Bergen, Norway (Medical Data Overview) 65