Processing and Analysis of Label-Free Images: Comparison
View Latest Version
Please note this is a comparison between Version 4 by Conner Chen and Version 3 by Laura Antonelli.

Time-lapse microscopy imaging is a key approach for an increasing number of biological and biomedical studies to observe the dynamic behavior of cells over time which helps quantify important data, such as the number of cells and their sizes, shapes, and dynamic interactions across time. Label-free imaging is an essential strategy for such studies as it ensures that native cell behavior remains uninfluenced by the recording process. Computer vision and machine/deep learning approaches have made significant progress in this area.

  • label-free microscopy
  • cell segmentation
  • cell classification
  • cell event detection
  • cell tracking

1. Introduction

Microscopy is a fundamental research pillar enabling scientists to discover the structures and dynamics of cells and subcellular components. Most of these components are phase objects, which means they are transparent and colorless and cannot be visualized under a light microscope. To overcome this limit, a solution consists in staining the components with dyes, also known as fluorophores or fluorochromes. Such molecules absorb short-wavelength light, generally UV, and emit fluorescence light at a longer wavelength. This mechanism represents the basic premise of fluorescence microscopy techniques, and the fluorescence images show the specimen bright on a dark background. The staining techniques use different dyes to point out a specific cellular component. Then, the same specimen can appear with a red, blue, or green color and appearance, as depicted in Figure 1a,b.
As molecular genetic methodologies and tools advance, fluorescence techniques become more specific and can be applied to a larger set of different model organisms [1,2][1][2]. However, to ensure high-quality images, they require laborious and expensive sample preparation. Furthermore, the dyes reduce their luminance over time due to a light-induced degradation process called photobleaching [3]. Finally, fluorescent techniques are invasive and interfere with biological processes causing phototoxicity.
On the contrary, Label-free Imaging (LI) techniques can visualize many cellular structures simultaneously with minimal sample preparation, phototoxicity, and no photobleaching, making them particularly suitable for live-cell imaging. Thus, LI provides measurements complementary to fluorescence imaging for several biological studies. Based on optical principles, these techniques measure the light phase change (i.e., the refractive index) passing through the specimen and converting it into intensity modulations producing qualitative phase contrast images, as exemplified in Figure 1c (the phase change information contained in LI images is non-linearly coupled with its luminance intensity and cannot be retrieved quantitatively. An image produced by LI techniques is a map of path-length shifts associated with the specimen, containing information about both the thickness and refractive index of its structure; for further details and explanations, see [4]). Among the traditional techniques to phase change imaging, there are Phase Contrast (PhC) [5] and Differential Image Contrast (DIC) [6] based on the phase gradient method and differential interference contrast, respectively, to measure the refractive index. Another LI technique very similar to DIC is the Hoffman Modulation Contrast (HMC) [7]. Due to intrinsic limitations of the numerical conversion methods, phase contrast images contain artifacts [8] (i.e., bright halo surrounding cell contours) and the “shade-off effect”, which produces low contrast inside the cells with an intensity very similar to the background. Although several methods have been developed to overcome these artifacts, automatic image processing of label-free images is still challenging, especially the segmentation task for separating cells from the background.
Algorithms 15 00313 g001 550
Figure 1. Example of differences in appearance of fluorescence and phase contrast microscopy images (culture of human lymphocyte cells [9]): (a) fluorescence image of nuclear envelope; (b) fluorescence image of interior nuclei (DNA); (c) phase contrast image of whole cells.
Unlike the previous techniques, quantitative LI provides higher contrast and reduces artifacts. Among them, Quantitative Phase Imaging (QPI) techniques refer to the microscopes showing phase quantitative information. Details concerning QPI techniques can be found in [10].
A further alternative to noninvasive techniques is Bright-Field (BF) microscopy. It represents the most straightforward configuration for the light microscope, which is not only cheaper but also does not require sample preparation [11]. BF images provide information about the cellular organization, and they are preferred to visualize specimens with low contrast from the background (unlike in fluorescence) or with low resolution and magnification visualization of thin cellular components (unlike in phase contrast). Several studies describe the use of the BF channel in cell detection and automated image analysis of cell populations [12,13][12][13].
With the aim of observing the dynamic behavior of living cells over time, LI microscopes are also equipped with a real-time imaging tool named time-lapse. Broadly speaking, time-lapse is a speed-up technique to observe events changing over time. This is usually realized by taking images at regular time intervals and merging them into a video. One of the most significant applications of time-lapse microscopy is cell population monitoring to study single-cell behavior in response to physiological or external stimuli and understand the underlying mechanisms. For example, in drug discovery and cancer research [14], time-lapse microscopy is used to look at cell response to anti-mitotic drugs in terms of cell division and cell death. To achieve this goal, quantitative information on cell behavior needs to be obtained and analyzed [15]. Cell proliferation, lineage, and fate are of primary importance among various cellular events. Image analysis of these biological processes is usually performed manually with suitable protocols [16]. As manual analysis of a large volume of light microscopy images is slow, tedious, time-consuming, and subject to observer subjectivity, biological studies see an increased demand for reliable automatic imaging tools. As in many scientific disciplines, Artificial Intelligence (AI) has been changing how imaging data are processed and analyzed and how experiments are carried out. AI refers to artificial systems aiming to adapt previous knowledge to new situations and recognize meaning in data patterns. Machine Learning (ML) [17] is a subset of AI methods that extracts valuable features from large data sets to make predictions or decisions on unseen data. An ML algorithm is not designed to solve a specific problem but rather to train a computer to solve problems. The training is data-driven. Deep Learning (DL) is a set of ML algorithms using a multi-layer “neural networks” to progressively extract higher-level features from data (the number of layers is the depth of the model, hence the terminology “deep learning”; for a quick overview on DL key concepts for microscopy image data, the reader can refer to [18]).

2. Cell Segmentation

Image segmentation is the main task for producing numerical data from live-cell imaging experiments, thus providing direct insight into the living system from the quantitative cell information [23,24][19][20]. Cell segmentation is the process of splitting a microscopy image into “segments”, i.e., Regions Of Interest (ROIs), and produces an image where cells are separated from the image background by cell contours or cell labels. Accurate cell segmentations are crucial for many challenges involved in cellular analysis, including but not limited to cell tracking [25][21], cellular features quantification, proliferation, morphology, migration, interactions, and counting [26,27,28,29][22][23][24][25]. Several steps are considered in the literature for achieving cell segmentation. Some authors [8,20,30][8][26][27] consider crucial to initially perform an image reconstruction step, which produces images with higher contrast between foreground and background, increasing the success of the subsequent image processing tasks. The image formation model in phase contrast microscopy can be studied to reduce artefacts and solve the inverse problem through a regularization approach [8]. Some ML-based methods are reviewed by Vicar et al. [20][26] for PhC and DIC images, while De Haan et al. [30][27] present an overview of how DL-based frameworks solve these inverse problems in optical microscopy. Cell detection (or identification) is also frequently adopted previous to segmentation [20[26][28],22], with the aim of locating the cells in the image (e.g., via bounding boxes, as exemplified in Figure 2a).
Algorithms 15 00313 g002 550
Figure 2. Results of different image processing tasks: (a) cell detection; (b) cell semantic segmentation; (c) cell instance segmentation.
Some specific types of segmentation are frequently considered for cellular images [20,31,32,33][26][29][30][31]. Semantic segmentation identifies the object (i.e., the cell) category of each pixel for every known object within an image, as exemplified in Figure 2b. Instance segmentation, instead, identifies the object instance (i.e., the cell with specific features) of each pixel for every known object within an image [34][32]; an example is given in Figure 2c. It should be observed that the problem of cell instance segmentation is sometimes intended as joint cell detection and segmentation (e.g., see [32][30]), while other times (e.g., see [20][26]) instant segmentation is used as a synonym for single cell segmentation.

3. Event Detection and Classification

Even though segmentation remains the core of the subsequent imaging tasks, automated analysis of microscopy image sequences often bypasses the segmentation task. Usually, the detection of cell events under investigation is performed directly from heuristically generated ROIs. Detecting changes in cellular behavior plays a central role in different studies, where the focus is on identifying the changes in cellular growth, mitosis, and death. Such changes may be related to cell shape, division, and movement. They cannot be detected in a single image but require the analysis of video or time-lapse sequences. The difficulty primarily relies on the wide spatial-temporal variability of such phenomena, which requires suitable methods to handle time-varying phenomena. Nevertheless, the infinite spectrum of possible events faces an inherent shortage of labeled data. Automatic and robust approaches to detecting the time and location of cell events from image sequences often make use of the classification task. In microscopy, classification refers to identifying and distinguishing different cell types or states. Classification between other cells, types of tumors (benign or malignant), types of cell states (mitosis detection, alive-dead classification), and types of CIC (Cell-In-Cell) structures [47][33] are some typical applications.

4. Cell Tracking

Object tracking consists in locating and monitoring one or more objects of interest and their behavior over time [21][34]. The image sequence containing cells can be acquired at specific time intervals using the time-lapse technique. When discussing cell tracking, it is generally assumed that segmentation or detection and classification have been performed.

5. Software

In Table 1, we provide links to existing publicly available software.

6. Data

In Tables 2, 3, and 4, we provide details of available annotated data for cell segmentation, event detection, and tracking in traditional label-free images.

References

  1. Sebestyén, E.; Marullo, F.; Lucini, F.; Petrini, C.; Bianchi, A.; Valsoni, S.; Olivieri, I.; Antonelli, L.; Gregoretti, F.; Oliva, G.; et al. SAMMY-seq reveals early alteration of heterochromatin and deregulation of bivalent genes in Hutchinson-Gilford Progeria Syndrome. Nat. Commun. 2020, 11, 1–16.
  2. Marullo, F.; Cesarini, E.; Antonelli, L.; Gregoretti, F.; Oliva, G.; Lanzuolo, C. Nucleoplasmic Lamin A/C and Polycomb group of proteins: An evolutionarily conserved interplay. Nucleus 2016, 7, 103–111.
  3. Song, L.; Hennink, E.; Young, I.; Tanke, H. Photobleaching kinetics of fluoresce in quantitative fluorescence microscopy. Biophys J. 1995, 68, 2588–2600.
  4. Mir, M.; Bhaduri, B.; Wang, R.; Zhu, R.; Popescu, G. Quantitative Phase Imaging. Prog. Opt. 2012, 57, 133–217.
  5. Zernike, F. How I Discovered Phase Contrast. Science 1955, 121, 345–349.
  6. Nomarski, G. Differential microinterferometer with polarized waves. J. Phys. Radium Paris 1955, 16, 9S.
  7. Hoffman, R.; Gross, L. Modulation Contrast Microscope. Appl. Opt. 1975, 14, 1169–1176.
  8. Yin, Z.; Kanade, T.; Chen, M. Understanding the phase contrast optics to restore artifact-free microscopy images for segmentation. Med. Image Anal. 2012, 16, 1047–1062.
  9. Gregoretti, F.; Lucini, F.; Cesarini, E.; Oliva, G.; Lanzuolo, C.; Antonelli, L. Segmentation, 3D reconstruction and analysis of PcG proteins in fluorescence microscopy images in different cell culture conditions. In Methods in Molecular Biology; Springer: New York, NY, USA, 2022.
  10. Popescu, G. Quantitative Phase Imaging of Cells and Tissues; Mc-Graw-Hill: New York, NY, USA, 2011.
  11. Helgadottir, S.; Midtvedt, B.; Pineda, J.; Sabirsh, A.; Adiels, C.B.; Romeo, S.; Midtvedt, D.; Volpe, G. Extracting quantitative biological information from bright-field cell images using deep learning. Biophys. Rev. 2021, 2, 031401.
  12. Buggenthin, F.; Marr, C.; Schwarzfischer, M.; Hoppe, P.S.; Hilsenbeck, O.; Schroeder, T.; Theis, F.J. An automatic method for robust and fast cell detection in bright field images from high-throughput microscopy. BMC Bioinform. 2013, 14, 297.
  13. Selinummi, J.; Ruusuvuori, P.; Podolsky, I.; Ozinsky, A.; Gold, E.; Yli-Harja, O.; Aderem, A.; Shmulevich, I. Bright Field Microscopy as an Alternative to Whole Cell Fluorescence in Automated Analysis of Macrophage Images. PLoS ONE 2009, 4, 1–9.
  14. Naso, F.D.; Sterbini, V.; Crecca, E.; Asteriti, I.A.; Russo, A.D.; Giubettini, M.; Cundari, E.; Lindon, C.; Rosa, A.; Guarguaglini, G. Excess TPX2 interferes with microtubule disassembly and nuclei reformation at mitotic exit. Cells 2020, 9, 374.
  15. Jiang, Q.; Sudalagunta, P.; Meads, M.B.; Ahmed, K.T.; Rutkowski, T.; Shain, K.; Silva, A.S.; Zhang, W. An Advanced Framework for Time-lapse Microscopy Image Analysis. bioRxiv 2020.
  16. Caldon, C.E.; Burgess, A. Label free, quantitative single-cell fate tracking of time-lapse movies. MethodsX 2019, 6, 2468–2475.
  17. Janiesch, C.; Zschech, P.; Heinrich, K. Machine learning and deep learning. Electron. Mark. 2021, 31, 685–695.
  18. Gupta, A.; Harrison, P.J.; Wieslander, H.; Pielawski, N.; Kartasalo, K.; Partel, G.; Solorzano, L.; Suveer, A.; Klemm, A.H.; Spjuth, O.; et al. Deep Learning in Image Cytometry: A Review. Cytometry Part A 2019, 95, 366–380.
  19. Van Valen, D.A.; Kudo, T.; Lane, K.M.; Macklin, D.N.; Quach, N.T.; DeFelice, M.M.; Maayan, I.; Tanouchi, Y.; Ashley, E.A.; Covert, M.W. Deep Learning Automates the Quantitative Analysis of Individual Cells in Live-Cell Imaging Experiments. PLoS Comput. Biol. 2016, 12, e1005177.
  20. Lux, F.; Matula, P. Cell segmentation by combining marker-controlled watershed and deep learning. arXiv 2020, arXiv:2004.01607.
  21. Hilsenbeck, O.; Schwarzfischer, M.; Loeffler, D.; Dimopoulos, S.; Hastreiter, S.; Marr, C.; Theis, F.J.; Schroeder, T. fastER: A user-friendly tool for ultrafast and robust cell segmentation in large-scale microscopy. Bioinformatics 2017, 33, 2020–2028.
  22. Edlund, C.; Jackson, T.R.; Khalid, N.; Bevan, N.; Dale, T.; Dengel, A.; Ahmed, S.; Trygg, J.; Sjögren, R. LIVECell: A large-scale dataset for label-free live cell segmentation. Nat. Methods 2021, 18, 1038–1045.
  23. Caicedo, J.; Goodman, A.; Karhohs, K.; Cimini, B.; Ackerman, J.; Haghighi, M.; Heng, C.; Becker, T.; Doan, M.; McQuin, C.; et al. Nucleus segmentation across imaging experiments: The 2018 Data Science Bowl. Nat. Methods 2019, 16, 1247–1253.
  24. Casalino, L.; D’Ambra, P.; Guarracino, M.R.; Irpino, A.; Maddalena, L.; Maiorano, F.; Minchiotti, G.; Jorge Patriarca, E. Image Analysis and Classification for High-Throughput Screening of Embryonic Stem Cells. In Proceedings of the Mathematical Models in Biology: Bringing Mathematics to Life; Zazzu, V., Ferraro, M.B., Guarracino, M.R., Eds.; Springer International Publishing: Cham, Switzerland, 2015; pp. 17–31.
  25. Casalino, L.; Guarracino, M.R.; Maddalena, L. Imaging for High-Throughput Screening of Pluripotent Stem Cells, SIAM Conference on Imaging Science—IS18. 2018. Available online: https://www.siam-is18.dm.unibo.it/presentations/811.html (accessed on 3 August 2022).
  26. Vicar, T.; Balvan, J.; Jaros, J.; Jug, F.; Kolar, R.; Masarik, M.; Gumulec, J. Cell segmentation methods for label-free contrast microscopy: Review and comprehensive comparison. BMC Bioinform. 2019, 20, 1–25.
  27. de Haan, K.; Rivenson, Y.; Wu, Y.; Ozcan, A. Deep-Learning-Based Image Reconstruction and Enhancement in Optical Microscopy. Proc. IEEE 2020, 108, 30–50.
  28. Ulman, V.; Maška, M.; Magnusson, K.E.G.; Ronneberger, O.; Haubold, C.; Harder, N.; Matula, P.; Matula, P.; Svoboda, D.; Radojevic, M.; et al. An objective comparison of cell-tracking algorithms. Nat. Methods 2017, 14, 1141–1152.
  29. Gregoretti, F.; Cesarini, E.; Lanzuolo, C.; Oliva, G.; Antonelli, L. An Automatic Segmentation Method Combining an Active Contour Model and a Classification Technique for Detecting Polycomb-group Proteinsin High-Throughput Microscopy Images. In Polycomb Group Proteins: Methods and Protocols; Lanzuolo, C., Bodega, B., Eds.; Springer New York: New York, NY, USA, 2016; pp. 181–197.
  30. Yi, J.; Wu, P.; Jiang, M.; Huang, Q.; Hoeppner, D.J.; Metaxas, D.N. Attentive neural cell instance segmentation. Med. Image Anal. 2019, 55, 228–240.
  31. Gregoretti, F.; Cortesi, A.; Oliva, G.; Bodega, B.; Antonelli, L. An Algorithm for the Analysis of the 3D Spatial Organization of the Genome. In Capturing Chromosome Conformation: Methods and Protocols; Bodega, B., Lanzuolo, C., Eds.; Springer US: New York, NY, USA, 2021; pp. 299–320.
  32. Antonelli, L.; De Simone, V.; di Serafino, D. A view of computational models for image segmentation. In Annali dell’Universitá di Ferrara; Springer: Cham, Switzerland, 2022.
  33. Borensztejn, K.; Tyrna, P.; Gaweł, A.M.; Dziuba, I.; Wojcik, C.; Bialy, L.P.; Mlynarczuk-Bialy, I. Classification of Cell-in-Cell Structures: Different Phenomena with Similar Appearance. Cells 2021, 10, 2569.
  34. Emami, N.; Sedaei, Z.; Ferdousi, R. Computerized cell tracking: Current methods, tools and challenges. Visual Inform. 2021, 5, 1–13.
More
© Text is available under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/)
ScholarVision Creations
Feedback