1 VISUALIZATION OF MOLECULAR STRUCTURES – CURRENTLY USED METHODS AND FUTURE CHALLENGES Barbora Kozlíková 30. 03. 2020 Visualization II Barbora Kozlíková INTRODUCTION • Molecular visualization is one of the oldest branches of data visualization – Builds up on pre-computer era depictions and models of molecules • Molecular visualization is a vast and diverse field of research  We will focus on – Interactive 3D Visualization of – Biomolecules (DNA, proteins, lipids etc.) described by – Classical Models (no quantum effects, atoms depicted by hard spheres) 2 Barbora Kozlíková INTRODUCTION 3 Barbora Kozlíková INTRODUCTION 4 Barbora Kozlíková BIOMOLECULES • Molecules – Atoms (117 chemical elements) • Oxygen, carbon, nitrogen, hydrogen – Bonds (e.g., covalent, disulfide, hydrogen) • Small molecules & ions – Lipids (membranes) – Ligands/metabolites – Solvent molecules (e.g., water) – etc. http://en.wikipedia.org/wiki/Phospholipid 7 Barbora Kozlíková BIOMOLECULES • Proteins – Building blocks of the „machinery of life“ – Consist of amino acids • One or more linear chains of amino acids that form a functional complex – Secondary structure (helix, sheet, turn, coil) http://en.wikipedia.org/wiki/Protein_structurehttp://en.wikipedia.org/wiki/Amino_acid 8 Barbora Kozlíková BIOMOLECULES • DNA & RNA – DNA stores the “genetic code” • Blueprint for proteins – Chain of nucleotides • Sugar backbone • Phosphate • Nucleobase – cytosine, guanine, adenine, thymine/uracil) • 3 nucleotides encode 1 amino acid http://en.wikipedia.org/wiki/DNA 9 Barbora Kozlíková TAXONOMY 10 Interactive simulations Flexibility Aggregation Molecular representations Atomistic depictions of mesoscopic data Reaction networks Level Of Detail AtomisticAbstract coarse-grainedmesoscopicatomistic Data scale inter- molecular intra- molecular Visualization static geometry Interactive animation ~100 µm ~0.1 nm Barbora Kozlíková TAXONOMY Interactive simulations Flexibility Aggregation Molecular representations Atomistic depictions of mesoscopic data Reaction networks Level Of Detail AtomisticAbstract 11 coarse-grainedmesoscopicatomistic Data scale inter- molecular intra- molecular Visualization static geometry Interactive animation Barbora Kozlíková MOLECULAR REPRESENTATION MODELS • Atomistic Representations – Bond-centric Models – Surface Models • Abstract and Illustrative Representations – Representations of Molecular Architecture – Surface Abstractions • Structural Level of Detail 12 Barbora Kozlíková TAXONOMY Interactive simulations Flexibility Aggregation Molecular representations Atomistic depictions of mesoscopic data Reaction networks Level Of Detail AtomisticAbstract 13 coarse-grainedmesoscopicatomistic Data scale inter- molecular intra- molecular Visualization static geometry Interactive animation Barbora Kozlíková ATOMISTIC REPRESENTATIONS • Molecular models that show the position of the atoms • Bond-centric Models – Bonds define the topology of the molecule – Lines, Sticks, Balls-and-Sticks  spheres and cylinders 14 Barbora Kozlíková GPU-BASED GLYPH RAY CASTING • State-of-the-art for rendering implicit objects – Upload implicit description of object to GPU – Proxy geometry that covers the object in Vertex/Geometry Shader – Object/ray intersection in Fragment Shader Image plane & proxy geometry camera IEEE Vis 2015 Tutorial: „Interactive GPU-based Visualization of Large Dynamic Particle Data“ 15 Barbora Kozlíková MOLECULAR SURFACES • Show molecular properties • Depict boundary 16 Barbora Kozlíková VDW AND SAS SURFACE • Van der Waals (vdW) surface – vdW radius: distance between non-bonded atoms – Molecular volume – Does not consider ligands or solvent molecules • Solvent Accessible Surface (SAS) – Surface with respect to a certain solvent radius • Interior not reachable by solvent – Theory: Rolling probe (radius rp) – Practice: Inflation of vdW radius by rp • Rendering via GPU ray casting vdW surface SAS Probe 17 Barbora Kozlíková VDW AND SAS SURFACE • Van der Waals (vdW) surface – vdW radius: distance between non-bonded atoms – Molecular volume – Does not consider ligands or solvent molecules • Solvent Accessible Surface (SAS) – Surface with respect to a certain solvent radius • Interior not reachable by solvent – Theory: Rolling probe (radius rp) – Practice: Inflation of vdW radius by rp • Rendering via GPU ray casting vdW surface SAS 18 Barbora Kozlíková SOLVENT EXCLUDED SURFACE • Defined by rolling probe of radius rp – Probe surface traces out SES • Smooth, tight surface – Boundary with respect to solvent – No inflation (molecular volume is preserved) • Three types of patches – Concave spherical triangles – Convex spherical patches – Saddle-shaped toroidal patches • Parallel computation – Interactive for 100k atoms – CPU [Lindow et al. 2010] or GPU [Krone et al. 2011] SES vdW surface SAS Probe 19 Barbora Kozlíková LIGAND EXCLUDED SURFACE • Recent extension of the SES [Lindow et al. 2014] – Shows a more accurate contact surface with respect to a specific ligand • No analytic computation (yet?) – Computationally expensive, grid-based sampling method 21 Barbora Kozlíková LIGAND EXCLUDED SURFACE • Recent extension of the SES [Lindow et al. 2014] – Shows a more accurate contact surface with respect to a specific ligand • No analytic computation (yet?) – Computationally expensive, grid-based sampling method Solvent Excluded Surface Ligand Excluded SurfaceDifference 22 Barbora Kozlíková GAUSSIAN SURFACES a1 a2 SES Gaussian surface Two atoms with radial symmetric Gaussian density kernels Images: [Krone et al. 2012] 23 Barbora Kozlíková GAUSSIAN SURFACES • Interactive Rendering – Direct ray casting using depth peeling (~1M atoms) [Kanamori et al. 2008] – Grid-based sampling of the density (GPU-parallelized: ~10M atoms) • Isosurface extraction via Volume Ray Marching or Marching Cubes/Tetrahedra – Interactive image-based method for molecular dynamics [Bruckner, 2019] Marching Cubes / Tetrahedra a1 a2 Isosurface a1 a2 Images: [Krone et al. 2012] 24 Barbora Kozlíková TAXONOMY Interactive simulations Flexibility Aggregation Molecular representations Atomistic depictions of mesoscopic data Reaction networks Level Of Detail AtomisticAbstract 25 coarse-grainedmesoscopicatomistic Data scale inter- molecular intra- molecular Visualization static geometry Interactive animation Barbora Kozlíková ABSTRACT AND ILLUSTRATIVE REPRESENTATIONS • Representations of Molecular Architecture – Show functional structure (derived from atom positions) – Cartoon Representation for DNA and proteins • Seamless transition [van der Zwan et al. 2011] http://tobias.isenberg.cc/VideosAndDemos/Zwan2011IMV 26 Barbora Kozlíková ABSTRACT AND ILLUSTRATIVE REPRESENTATIONS • Cartoon Rendering – Complex shapes  no ray casting – GPU-acceleration polygonal rendering • Vertex shader [Wahle et al. 2011] • Geometry shader [Krone et al. 2008] 27 Barbora Kozlíková ABSTRACT AND ILLUSTRATIVE REPRESENTATIONS • Surface Abstractions – Coarsening of Gaussian surfaces (LoD, bounding spheres) [Krone at al. 2012] – Smoothing of high-frequency surfaces like SES [Cipriano, Gleicher 2007] – Mapping of molecular surface to a sphere (e.g., [Rahi, Sharp 2014]) Images: [Krone et al. 2012] 28 Barbora Kozlíková ABSTRACT AND ILLUSTRATIVE REPRESENTATIONS • Surface Abstractions – Coarsening of Gaussian surfaces (LoD, bounding spheres) [Krone at al. 2012] – Smoothing of high-frequency surfaces like SES [Cipriano, Gleicher 2007] – Mapping of molecular surface to a sphere (e.g., [Rahi, Sharp 2014]) Images: [Cipriano, Gleicher 2007] 29 Barbora Kozlíková ABSTRACT AND ILLUSTRATIVE REPRESENTATIONS • Surface Abstractions – Coarsening of Gaussian surfaces (LoD, bounding spheres) [Krone at al. 2012] – Smoothing of high-frequency surfaces like SES [Cipriano, Gleicher 2007] – Mapping of molecular surface to a sphere (e.g., [Rahi, Sharp 2014]) Images: [Rahi, Sharp 2014] 30 Barbora Kozlíková TAXONOMY Interactive simulations Flexibility Aggregation Molecular representations Atomistic depictions of mesoscopic data Reaction networks Level Of Detail AtomisticAbstract 31 coarse-grainedmesoscopicatomistic Data scale inter- molecular intra- molecular Visualization static geometry Interactive animation Barbora Kozlíková STRUCTURAL LEVEL OF DETAIL • Derive all-atom representation from coarse-grained simulations – Cellular environment  many instances of the same molecules – Special GPU-accelerated rendering methods – Interactive rendering of up to 10 billion particles [Falk et al. 2013] [LeMuzic et al. 2014] 32 Barbora Kozlíková MOLECULAR RENDERING • Enhances – Image quality – Perception of geometric shapes and depth complexity • Achieved by – Shading – Depth cues • Computable for dynamic data in real-time 33 Barbora Kozlíková COLOR type of atoms chains hydrophobicity partial charge 34 Barbora Kozlíková CEL SHADING • Artistic or non-photorealistic renderings with a comic-like look • Resembles hand-drawn illustrations Mycoplasma cell B-Raf protein rendered in MegaMol [Goodsell] [Grottel et al. 2015] 36 Barbora Kozlíková CEL SHADING • cellVIEW – Aiming to resemble hand-drawn illustrations of David Goodsell 37 Barbora Kozlíková FEATURE LINES AND HATCHING cartoon representations molecular surfaces [Lawonn et al. 2014] [van der Zwan et al. 2011] [Weber 2009] space filling models [van der Zwan et al. 2011] 38 Barbora Kozlíková DEPTH CUE TECHNIQUES • Sihouettes, halos, depth darkening • Ambient Occlusion Real-time Ambient Occlusion • Depth of Field 39 Barbora Kozlíková ORDINAL DEPTH CUES • Silhouettes Computed in image space in postprocessing • Halos Extended from the object boundaries • Depth darkening Visually separates distant overlapping objects [Tarini et al. 2006] [Krone et al. 2009] 40 Barbora Kozlíková RELATIVE DEPTH CUES – AMBIENT OCCLUSION • Mimicking the transport of diffuse light between objects • Local shadowing, increases depth perception Local lighting Ambient Occlusion Combined • Computationally expensive, accelerated approaches developed 41 Barbora Kozlíková REAL-TIME AMBIENT OCCLUSION • Screen-Space Ambient Occlusion – Image space technique, approximates the effects in postprocessing – Considers the visible neighborhood of fragments • Object-Space Ambient Occlusion – Considers the entire local neighborhood of atoms Local lighting Screen Space AO [Kajalin 2009] Object Space AO [Grottel et al. 2012] Images: [Grottel et al. 2012] 42 Barbora Kozlíková DEPTH OF FIELD • Separating foreground from background • Image-space and object-space based approaches • Draw the attention to a specific region or semantic properties Region-based [Falk et al. 2013] Semantic-based [Kottravel et al. 2015] 43 Barbora Kozlíková VISUALIZATION OF MOLECULAR DYNAMICS 44 Barbora Kozlíková VISUALIZATION OF MOLECULAR DYNAMICS Interactive simulations Flexibility Aggregation Molecular representations Atomistic depictions of mesoscopic data Reaction networks Level Of Detail AtomisticAbstract coarse-grainedmesoscopicatomistic Data scale inter- molecular intra- molecular Visualization static geometry Interactive animation Reaction networks 45 Barbora Kozlíková TAXONOMY Interactive simulations Flexibility Aggregation Molecular representations Atomistic depictions of mesoscopic data Reaction networks Level Of Detail AtomisticAbstract coarse-grainedmesoscopicatomistic Data scale inter- molecular intra- molecular static geometry Interactive animation 46 Barbora Kozlíková VISUALIZATION OF FLEXIBILITY • Probability distribution depicting the varying molecular conformations Modulated tube [Lv et al. 2013] Normal Mode Analysis [Bryden et al. 2012] 47 Barbora Kozlíková TAXONOMY Interactive simulations Flexibility Aggregation Molecular representations Atomistic depictions of mesoscopic data Reaction networks Level Of Detail AtomisticAbstract coarse-grainedmesoscopicatomistic Data scale inter- molecular intra- molecular Visualization static geometry Interactive animation 48 Barbora Kozlíková AGGREGATION • Spatial Aggregating atom densities using property grids [Rozmanov et al. 2014] 49 Barbora Kozlíková AGGREGATION • Temporal • Aggregated diffusional motion • Combination of temporal and spatial aggregation [Chavent et al. 2014] [Ertl et al. 2014] 50 Barbora Kozlíková TAXONOMY Interactive simulations Flexibility Aggregation Molecular representations Atomistic depictions of mesoscopic data Reaction networks Level Of Detail AtomisticAbstract coarse-grainedmesoscopicatomistic Data scale inter- molecular intra- molecular Visualization static geometry Interactive animation 51 Barbora Kozlíková INTERACTIVE SIMULATIONS • Visualization has to be interactive  simulation performance has to be the limiting factor • Haptic rendering – 1000 Hz refresh rates • Cheaper and better hardware  haptic steering is very attractive Applied to systems with more than 1 million atoms [Dreher et al. 2013] 52 Barbora Kozlíková TAXONOMY Interactive simulations Flexibility Aggregation Molecular representations Atomistic depictions of mesoscopic data Reaction networks Level Of Detail AtomisticAbstract coarse-grainedmesoscopicatomistic Data scale inter- molecular intra- molecular Visualization static geometry Interactive animation 54 Barbora Kozlíková MOLECULAR REACTIONS • Several existing tools for the visualization of reaction networks • Particle simulations are very crowded Methods visually emphasizing interesting aspects of simulations particle trajectory [Falk et al. 2009] 55 Barbora Kozlíková MOLECULAR REACTIONS • Several existing tools for the visualization of reaction networks • Particle simulations are very crowded Methods visually emphasizing interesting aspects of simulations focus on reactions [Le Muzic et al. 2014] 56 Barbora Kozlíková MOLECULAR REACTIONS 57 Barbora Kozlíková MOLECULAR REACTIONS • Visualization of polymerization • Visualization of molecular orbitals, electron densities, bonds [Kolesár et al. 2014] [Stone et al. 2009] [Haranczyk, Gutowski 2008] 58 Barbora Kozlíková MOLECULAR VISUALIZATION SYSTEMS 59 Barbora Kozlíková MOLECULAR VISUALIZATION SYSTEMS 60 Barbora Kozlíková FUTURE CHALLENGES • Recent trend is to use GPU based rendering and computations Programmable GPUs and multi-core GPUs enable parallelization • Increasing amount of captured data sets in terms of particle numbers and time steps • Complexity of data will require new visual representations Visual analysis • Quantum mechanics simulations will require novel visualization methods • Visual language for biomolecules 61 Barbora Kozlíková DENSE AND MULTISCALE ENVIRONMENTS 62 • Modeling and visualizing large structures or molecular systems, often without the knowledge about their constitution • Modeling based on a “recipe” [Johnson et al., cellPACK] Barbora Kozlíková INSTANT CONSTRUCTION OF MODELS 63 [Klein et al., IEEE TVCG, 24(1), 2018] Barbora Kozlíková LIPID MEMBBRANE POPULATION 64 [Klein et al., IEEE TVCG, 24(1), 2018] Barbora Kozlíková INNER (SOLUBLE) COMPONENTS 65 [Klein et al., IEEE TVCG, 24(1), 2018] Barbora Kozlíková INNER FIBROUS STRUCTURES 66 [Klein et al., IEEE TVCG, 24(1), 2018] Barbora Kozlíková Thank You for Your Attention 67