Faculty of Informatics Masaryk University Object Tracking in Bioimage Data Habilitation Thesis (Collection of Articles) Martin Maška April 2020 Abstract Object tracking plays an important role in understanding the principles of intracellular and intercellular processes, with particles, cells, and cell membrane protrusions being typical targets of interest. Subsequent downstream analyses of their spatiotemporal lineages help to quantitatively characterize their migratory patterns, morphological changes, or mutual interactions and relationships, with the primary aim of revealing novel insights on the behavior of these targets under different physiological and pathological conditions. The advances in optical microscopy and fluorescent reporters have facilitated observations of these targets at unprecedented spatiotemporal resolutions, at the expense of acquiring massive amounts of bioimage data, calling for robust and automatic bioimage analysis workflows to substitute for subjective, highly error-prone, and extremely laborious manual analyses. In this thesis, the author surveys his concerted efforts made to develop robust and fully automatic workflows for tracking particles, cells, and filopodial protrusions in multidimensional bioimage data. This involves not only novel algorithmic contributions, but also establishing of heterogeneous benchmark datasets and rigorous evaluation protocols to identify the limits and to assess the parameter sensitivity of these workflows. The thesis is structured as a comprehensive commentary to a selected collection of three conference papers and nine journal articles, highlighting the most important results achieved and their contributions to the state of the art. i Acknowledgments First of all, the author would like to express his gratitude to Pavel Matula, Petr Matula, Michal Kozubek, Carlos Ortiz de Solorzano, and Arrate Munoz-Barrutia who all were his mentors on the way to becoming a scientist. Special thanks also go to individual co-authors of the papers included in this collection and to other colleagues and collaborators for fruitful discussions and open-mindedness. Last but not least, the author thanks his family for their patience and love. iii Contents I Commentary 1 Introduction 2 Particle Tracking 3 Cell Tracking 4 Filopodium Tracking 5 Conclusion Bibliography Parti Commentary Chapter 1 Introduction The state-of-the-art bioimaging instruments routinely produce multidimensional bioimage data of enormous quantities [42], calling for robust and automatic bioimage analysis workflows to substitute for subjective, highly error-prone, and extremely laborious manual analyses [15, 40, 7]. Object tracking is instrumental in understanding the principles of various intracellular and intercellular processes, with particles, cells, and cell membrane protrusions being common targets of interest [52,16, 21]. The spatiotemporal lineages produced by bioimage object trackers form the solid ground for subsequent downstream analyses that shed more light on migratory patterns, morphological changes, and mutual interactions and relationships of the targets under investigation, thus providing important clues on their behavior in both tissue development and disease [1, 41, 22]. Unsurprisingly, the truly informative and unbiased value of such conclusions highly depends on the correctness of image analysis results and their possible curation. Therefore, there has been an increasing demand for evaluating bioimage analysis workflows to objectively assess their performance [25], leading to the boom in organizing biomedical competitions [14, 39, 49, 58, 50, 7], which in turn has called for control mechanisms and best practices to guarantee their fairness [32,48], and also for publicly available repositories of heterogeneous, annotated bioimage data. To this end, image-based simulations [59], allowing straightforward in-depth performance assessments due to the inherent availability of complete reference annotations and natural capability of generating bioimage data of the same targets of interest under varying environmental and imaging setups, have greatly been exploited [14, 39, 49, 58, 50]. Purpose and focus of the thesis The primary objective of this thesis is to comprehensively summarize the author's main algorithmic contributions as well as benchmarking efforts toward developing robust and automatic workflows for tracking particles, cells, and filopodial protrusions in multidimensional bioimage data. This is a research topic he has been attending to since receiving his Ph.D. in 2011 on segmenting fluorescent cells using fast level set-like algorithms. Apart from co-organizing two renowned competitions, Particle Tracking Challenge and Cell Tracking Challenge, that evolved into publicly available benchmarks for objective comparison of particle and cell tracking methods [14, 39, 58], the author was awarded a three-year grant from the Czech Science Foundation on tracking filopodial cells and received the Masaryk University Rector's Award for Outstanding Research Results Achieved by Young Scientists under 35 in 2016. As of April 2020, he has 3 co-authored 18 publications on object tracking in bioimage data, with more than 500 citations received according to the Web of Science database. Their representative subset, consisting of three conference papers and nine journal articles, is included in this thesis and is accompanied by a comprehensive commentary, highlighting the most important results achieved and their contributions to the state of the art. Structure of the thesis In the following three chapters, the state of the art of particle tracking (Chapter 2), cell tracking (Chapter 3), and filopodium tracking (Chapter 4) is briefly surveyed, being complemented with the author's contributions and with the lists of co-authored, representative papers relevant to individual tracking tasks. Finally, a short personal perspective on some of the actual research gaps and possible near-future research directions toward their addressing is presented in Chapter 5. In case of the printed version of the thesis, all selected papers in the collection are attached. However, they are purposely excluded from an online, publicly available version of the thesis to avoid copyright infringement. 4 Chapter 2 Particle Tracking Particle tracking is of fundamental importance in diverse quantitative analyses of dynamic intracellular processes using time-lapse microscopy [52]. The particle can be anything from a single molecule to a macromolecular complex, organelle, virus, or microsphere manifesting itself as a small dot in the image data. The problem of particle tracking can be formulated as having a recorded time-lapse sequence of moving dot-like objects, one is interested in spatiotemporal positions of individual objects. Due to the frequent impracticability of tracking particles manually, a number of fully automatic particle trackers have been developed over past decades [29], carrying out the tracking task in two subsequent phases: (1) particle detection and (2) particle linking. First, individual particles are detected separately in every frame of a given time-lapse sequence. Second, the detected particles are linked into tracks, a set of which forms a linear oriented forest (LOF) in the terminology of graph theory. Common approaches to particle detection range from simple thresholding or local-maxima finding to more advanced linear and nonlinear model fitting and centroid estimation schemes, whereas the linking of detected particles ranges from simple nearest-neighbor to multiframe association with or without the explicit guidance derived from motion models and state estimation [23, 41, 12, 13, 19, 30, 51]. In line with the widespread use of deep learning across different computer vision and bioimage analysis tasks [26,42], the self-learning capabilities of deep neural networks have also been exploited for both particle detection [43, 62] as well as particle linking [63, 54] over recent years. Contributions Bearing calls for open science [8] and reproducible research [9] in mind, the author co-organized the Particle Tracking Challenge (PTC, http : //bioimageanalysis . org/track/) in collaboration with the Biomedical Imaging Group Rotterdam (Erasmus University Medical Center, the Netherlands) and Quantitative Image Analysis Unit (Institut Pasteur, France). By gathering the bioimage informatics community and coordinating an open competition hosted by the 2012 IEEE International Symposium on Biomedical Imaging, the challenge organizers established a novel benchmark for objective and systematic comparison of particle tracking algorithms [14]. Using a completely annotated repository of computer-generated image data, consisting of four biologically driven scenarios with various particle appearance, dynamics, density, and signal (Fig. 2.1), and a diverse set of performance evaluation criteria, 14 particle trackers were objectively compared and clear differences between them, leading to important practical conclusions 5 Vesicles Microtubules Receptors Viruses Density Figure 2.1: Samples of the Particle Tracking Challenge image data, with varying particle (a-d) appearance, (e-h) dynamics, (i-k) density, and (l-o) signal. The figure was reproduced from [14]. for developers and users, were revealed. As of April 2020, this has been demonstrated not only by receiving more than 250 citations according to the Web of Science database and by tagging the paper as highly cited there, but also by continually exploiting the PTC dataset repository for further algorithmic improvements [30, 63, 54, 62]. Subsequent in-depth analyses of the PTC evaluation protocol revealed that its performance assessment strongly depends on the algorithm linking capabilities [36]. Indeed, it establishes particle correspondences at the level of individual tracks, and thus produces possibly inconsistent scoring for identical configurations of tracking errors with different temporal contexts. Furthermore, it provides neither users nor developers with any direct information about individual linking errors committed by the algorithm, which may complicate its parameter fine-tuning and further algorithmic developments. To address these difficulties, we developed a new evaluation protocol based on the Linear Oriented Forests Matching (LOFM) measure [36]. It establishes correspondences between a reference LOF and an algorithm-generated LOF at the level of particle detections and measures the difficulty of transforming an algorithm-generated LOF to a reference LOF by normalizing a weighted sum of the lowest number of allowed graph operations needed to make both LOFs identical. The new evaluation protocol allows one to assess detection and linking performance of the algorithm in an isolated, unbiased manner, thus consistently penalizing identical, time-varying configurations of tracking errors, and to directly identify pre-defined graph operations needed to transform the algorithm-generated LOF to the reference one, thus recognizing individual tracking errors committed by the algorithm. By re- 6 Scenario Density Microtubules Low Mid High Low Fteceptor; Mid High Low Vesicles Mid High Low Viruses Mid High SNR 4 7 4 7 4 7 4 7 4 7 4 7 ■ 7 4 7 4 7 4 7 4 7 4 7 Detection JSC #2 #3 5 2 4 2 4 2 3 3 3 3 3 3 5 7 7 7 7 7 1 1 5 1 2 2 11 4 11 4 11 4 11 11 11 11 11 11 7 3 3 3 3 1 2 2 5 5 4 5 5 11 5 11 1 1 1 9 1 1 3 5 1 1 1 5 2 2 1 5 1 1 #1 Linking #2 #3 4 4 4 4 4 4 11 11 3 3 3 3 5 1 1 1 1 1 1 1 5 2 2 2 8 5 2 2 2 2 1 1 11 11 11 11 11 8 8 8 3H 5 5 2 1 5 5 5 2 5 5 11 5 3 8 1 5 1 1 3 5 3 3 5 8 2 2 1 5 1 1 Scenario 1 Microtubules 1 tecepton 1- m 1 1 1 De isity Low Mid High Low Mid High ■ ffi SNR Detection LOFMD #1 1 1 1 1 12 1 1 li 3 3 3 3 7 7 7 7 7 7 5 1 2 1 2 2 #2 8 12 12 12 1 12 11 3 11 11 11 1 5 11 3 3 1 1 l| 5 5 2 1 1 #3 11 4 5 2 5 2 3 112 1 12 5 11 11 3 l| 11 11 5 2 2 1 5 5 12 ,. .. #1 Linking LOFML #2 #3 5 4 4 4 4 4 3 3 3 3 11 11 3 3 1 1 8 7 1 1 5 5 5 2 11 2 5 2 8 2 11 11 11 11 3 3 1 1 3 3 3 1 12 5 2 1 2 5 13 5 8 11 11 11 1 9 1 8 2 1 12 12 7 7 1 2 5 8 1 8 8 8 Figure 2.2: The top three best-performing algorithms in terms of particle detection and linking according to the Particle Tracking Challenge evaluation protocol (top panel) and the Linear Oriented Forests Matching (LOFM) measure (bottom panel). The figure was reproduced from [36]. analyzing the results of all 14 particle trackers that competed in PTC, it has been shown that the LOFM-based evaluation protocol compiles substantially different rankings from those reported previously using the PTC evaluation protocol (Fig. 2.2). In addition to performance-oriented evaluation of particle trackers, the symmetric nature of established vertex correspondences allows one to use the LOFM measure also for determining differences between multiple manual annotations of point-like targets [56,11], thus finding its application in annotation fusing too. Articles in collection (in chronological order) • N. Chenouard, I. Smal, F. de Chaumont, M. Maska, I. F. Sbalzarini, Y. Gong, J. Cardinale, C. Carthel, S. Coraluppi, M. Winter, A. R. Cohen, W. J. Godinez, K. Rohr, Y. Kalaidzidis, L. Liang, J. Duncan, H. Shen, Y. Xu, K. E. G. Magnusson, J. Jalden, H. M. Blau, P. Paul-Gilloteaux, P. Roudot, C. Kervrann, F. Waharte, J.-Y. Tinevez, S. L. Shorte, J. Willemse, K. Celler, G. P. van Wezel, H.-W. Dan, Y.-S. Tsai, C. Ortiz-de-Solorzano, J.-C. Olivo-Marin, and E. Meijering. Objective comparison of particle tracking methods. Nature Methods, ll(3):281-289, 2014. [14] Author's contribution: preparation of image data; evaluation and analysis of results; paper editing • M. Maska and P. Matula. Particle tracking accuracy measurement based on comparison of linear oriented forests. In IEEE International Conference on Computer Vision Workshops, pages 11-17,2017. [36] Author's contribution: method design, development, and implementation; evaluation and analysis of results; paper writing 7 Chapter 3 Cell Tracking Cell tracking is a key step in understanding a large variety of nontrivial biological processes [1]. Given a time-lapse sequence of motile cells, this task consists in identifying and segmenting individual cell instances and in establishing their temporal relationships. Because cells frequently undergo division events, during which mother cells are split into two or even more daughter cells in case of abnormal divisions, linear oriented forests underrepresent such situations, making acyclic oriented graphs the preferred representation of cell lineages [34]. Cell tracking approaches can broadly be classified into two categories [41,58]: tracking by detection [27,45,31,6] and tracking by model evolution [18,17, 35, 4]. The former paradigm, being followed also when tracking particles, involves two subsequent steps. First, a segmentation routine identifies and delineates individual cells in the entire time-lapse sequence separately for each frame. Second, the delineated cells are linked into tracks, mostly by optimizing a probabilistic objective function that evaluates the information from two successive frames, multiple successive frames, or even all frames at once. On the contrary, the latter paradigm solves both steps simultaneously, either under the instrumental assumption of unambiguous, spatiotemporal overlap between the corresponding cell regions using parametric or implicit active contour models, or without this limiting assumption using a Bayesian inference of dynamic models. Unsurprisingly, there is a recent rise of deep-learning-based workflows for tracking cells [46, 3, 28] in this field too. Contributions Inspired by the great success of PTC that was designed as a one-time event, the author has been co-organizing the Cell Tracking Challenge (CTC, http: //celltrackingchallenge . net) since 2012, initially in close collaboration with the University of Navarra (Spain) and Erasmus University Medical Center (the Netherlands), and nowadays also with the Carlos III University of Madrid (Spain), Max Planck Institute of Molecular Cell Biology and Genetics (Dresden, Germany), and University of New South Wales (Australia). Being organized as five open competitions held under the auspices of the 2013, 2014,2015, 2019, and 2020 IEEE International Symposium on Biomedical Imaging and running online since February 2017, it constitutes a renowned platform for objective and systematic comparison of cell tracking algorithms [39, 58]. Apart from a set of rigorous performance evaluation measures to assess detection, segmentation, and tracking accuracy, it offers a unique, highly heterogeneous, annotated repository of real and synthetic bioimage data, covering different imaging modalities (fluorescence, phase-contrast, 9 Figure 3.1: Samples of the Cell Tracking Challenge image data. The figure was inspired by [58] and extended accordingly by adding new samples of the datasets included in 2019 and 2020. differential-interference-contrast, and brightfield microscopy), staining (nuclear and cellular), dimensionality (2D and 3D), and cell lines and organisms (C. elegans, Drosophila Melanogaster, and Tribolium Castaneum), as shown in Fig. 3.1. The key published outputs of this initiative, [39] and [58], lay down the principles of this challenge, summarize the results received within the first challenge edition [39] and over the first three challenge editions [58], and draw important practical conclusions for developers and users. As of April 2020, both papers together received more than 175 citations according to the Web of Science database and the number of objectively evaluated algorithms within the challenge has nearly doubled, from 21 algorithms submitted by 17 participating groups thoroughly analyzed in [58] to 50 algorithms submitted by 40 participating groups included in the online version of the challenge. This tremendous increase can be attributed not only to the recent boom of deep-learning methods that frequently populate top positions in the leaderboard (Fig. 3.4), but also to the possibility of evaluating segmentation results only in reaction to many requests received from the community in this regard. Indeed, since 2019 the Cell Tracking Challenge has simultaneously been running two benchmarks, Cell Tracking Benchmark and Cell Segmentation Benchmark, to objectively evaluate cell segmentation and tracking algorithms for time-lapse microscopy image data. In contrast to widely used overlap-driven measures, such as the Dice or Jaccard coefficients, 10 I I t © False negative vertices IX] False positive vertices O True positive vertices | J Non-split vertices x Edges inherently deleted along with split or delete vertex operation : t = 0 t=l t = 2 -a - t = 3 -#- t = 5 -O- * = 6 i = 6 o- o • o * o \ Ö Operations » Split vertex (NS) © Add vertex (FN) |^ Delete vertex (FP) X Delete edge (ED) Add edge (EA) I Alter edge semantics (EC) Figure 3.2: A schematic example of the Acyclic Oriented Graphs Matching measure calculation for a reference graph (top left) and an algorithm-generated graph (top right), with individual types of vertices and graph operations needed to transform the algorithm-generated graph to the reference graph being highlighted by different colors and symbols (bottom). The figure was reproduced from [34]. for evaluating cell segmentation performance, there was the lack of standardized measures for evaluating cell tracking performance in 2012, when the preparatory steps toward organizing the very first CTC edition were initiated. Therefore, we developed the Acyclic Oriented Graphs Matching (AOGM) measure [34] that penalizes all possible errors in cell tracking results and aggregates them into a single value, thus accounting for all important cellular events within a captured field of view, such as migration, division, death, and transit through it, within a single evaluation protocol (Fig. 3.2). The measure assesses the difficulty of transforming an algorithm-generated graph into a reference graph by calculating a weighted sum of the lowest number of allowed graph operations needed to make both graphs identical. Because the weights fairly reflect the effort needed to perform individual graph operations, the AOGM measure is flexible and universal. A different configuration of weights can serve purposes other than objectively comparing the performance of multiple cell tracking algorithms. It may also be useful to tune the parameters of individual modules involved in cell tracking algorithms under development, such as cell detection or cluster separation modules. Not being concentrated on a single, often application-limited aspect of cell tracking, it allows one to evaluate the tracking performance of any cell tracking algorithm irrespective of its nature because it works with its final output. 11 In [35], we developed a fast and robust approach to tracking cells in multidimensional fluorescence microscopy image data, which follows the tracking by model evolution paradigm. It consists in reducing the amount of noise and enhancing flow-like structures using a coherence-enhancing diffusion filter, and in detecting cell boundaries by minimizing the Chan-Vese model in the fast level set-like (FLS) and graph cut (GC) frameworks, both integrated with a topological prior to allow simultaneous tracking of multiple cells over time. By quantitatively as well as qualitatively comparing both frameworks, it has been shown that the GC framework is slightly more accurate and produces smoother boundaries (Fig. 3.3), but at the expense of substantially slower and more memory-demanding execution than the FLS framework. In March 2014, the FLS variant was objectively evaluated within the second Cell Tracking Challenge edition. As of April 2020, it has still been occupying the top position in terms of segmentation performance and the second position in terms of overall performance for the Fluo-N3DH-CHO dataset in both the Cell Segmentation Benchmark and Cell Tracking Benchmark (Fig. 3.4), being tagged as MU-CZ there. By contrast, the GC variant was exploited for quantifying cancer cell migration in collagen-Matrigel scaffolds [2], providing the solid ground for subsequent downstream analyses that revealed new insights on the mesenchymal and lobopodial migratory patterns of lung adenocarcinoma cells in reaction to the different amount of Matrigel in the composition of hydrogels. Indeed, in the pure collagen hydrogels, the cells showed clear polarization of their cell bodies with well-defined lamellipodia at their leading edges and rear ends, whereas they protruded numerous short-lifetime blebs along their surface after the addition of Matrigel into the hydrogels. Articles in collection (in chronological order) • M. Maška, O. Daněk, S. Garasa, A. Rouzaut, A. Munoz-Barrutia, and C. Ortiz-de-Solorza-no. Segmentation and shape tracking of whole fluorescent cells based on the Chan-Vese model. IEEE Transactions on Medical Imaging, 32(6):995-1006, 2013. [35] Author's contribution: method design, development, and implementation; evaluation and analysis of results; paper writing • M. Maška, V. Ulman, D. Svoboda, P. Matula, P. Matula, C. Ederra, A. Urbiola, T. Espana, S. Venkatesan, D. M. W. Balak, P. Karas, T. Bolcková, M. Streitová, C. Carthel, S. Coraluppi, N. Harder, K. Rohr, K. E. G. Magnusson, J. Jaldén, H. M. Blau, O. Dzyubachyk, P. Křížek, G. M. Hagen, D. Pastor-Escuredo, D. Jimenez-Carretero, M. J. Ledesma-Carbayo, A. Munoz-Barrutia, E. Meijering, M. Kozubek, and C. Ortiz-de-Solórzano. A benchmark for comparison of cell tracking algorithms. Bioinformatics, 30(11):1609-1617, 2014. [39] Author's contribution: challenge co-organization; handling, evaluation, and analysis of submissions; implementation of evaluation and annotation software; supervision of annotations and creation of the consensual reference annotations; paper writing • P. Matula, M. Maška, D. V. Sorokin, C. Ortiz-de-Solórzano, and M. Kozubek. Cell tracking accuracy measurement based on comparison of acyclic oriented graphs. PLoS ONE, 10(12):e0144959,2015. [34] Author's contribution: method design, development, and implementation; evaluation and analysis of results; paper writing 12 Figure 3.3: A comparison of the final contours obtained using both the fast level set-like (green) and graph cut (red) frameworks, with the contour intersections highlighted in yellow and the white rectangular areas zoomed in below. The white arrows identify foreground regions of low intensity levels in the close vicinity of the cell boundaries, which were undersegmented by the fast level set-like framework. The figure was reproduced from [35]. M. Anguiano, C. Castilla, M. Maska, C. Ederra, R. Pelaez, X. Morales, G. Munoz-Arrieta, M. Mujika, M. Kozubek, A. Munoz-Barrutia, A. Rouzaut, S. Arana, J. M. Garcia-Aznar, and C. Ortiz-de-Solorzano. Characterization of three-dimensional cancer cell migration in mixed collagen-Matrigel scaffolds using microfluidics and image analysis. PLoS ONE, 12(2):e0171417, 2017. [2] Author's contribution: method development and implementation; visualization of results; paper editing V. Ulman, M. Maska, K. E. G. Magnusson, O. Ronneberger, C. Haubold, N. Harder, P. Mat-ula, P. Matula, D. Svoboda, M. Radojevic, I. Smal, K. Rohr, J. Jalden, H. M. Blau, O. Dzyu-bachyk, B. Lelieveldt, P. Xiao, Y. Li, S. Cho, A. C. Dufour, J.-C. Olivo-Marin, C. C. Reyes-Aldasoro, J. A. Solis-Lemus, R. Bensch, T. Brox, J. Stegmaier, R. Mikut, S. Wolf, F. A. Ham-precht, T. Esteves, P. Quelhas, O. Demirel, L. Malmstrom, F. Jug, P. Tomancak, E. Metering, A. Munoz-Barrutia, M. Kozubek, and C. Ortiz-de-Solorzano. An objective comparison of cell-tracking algorithms. Nature Methods, 14(12):1141-1152, 2017. [58] Author's contribution: challenge co-organization and active participation; handling, evaluation, and analysis of submissions; implementation of evaluation and annotation software; supervision of annotations and creation of the consensual reference annotations; paper writing 13 OP 0.828 0.629' 0.523' 0.67611 0.636 0.546 0.938™ 0.885. 0.870. 0.757™ 0.745 . 0.659121 0.951'i 0.902 0.902 0.942' 0.940 0.901 0.688" 0.601 0.507 0.926™ 0.912 . 0.906 -P' 0.609' 0.285 , 0.219 1? 0.951 . 0.896 0.886™ 0.772 111 0.735 -P' 4?