The advancement of deep learning has incited a paradigm shift across a myriad of disciplines [1,2,3], notably within the medical sciences [4,5,6]. In the field of oncology, deep learning methods have showcased unparalleled capacities to extrapolate pertinent information from complex, high-dimensional data, thereby facilitating precise and timely diagnosis [7], treatment planning [8], and prognosis prediction [9]. In this context, whole slide images (WSIs) of cancerous tissues have surfaced as a crucial resource for prognosis prediction, attributing to their detailed and rich content that aptly depicts the disease’s multifaceted nature [10,11,12].

Deep learning has ascended as a potent computational paradigm by virtue of its capacity to model intricate hierarchical patterns in data [13,14]. It employs multilayered artificial neural networks to autonomously learn hierarchical representations from raw input data, thus considerably reducing the need for manual feature extraction. These representations, often termed as features, empower the model to distinguish and differentiate complex patterns in the data, rendering deep learning an apt tool for a multitude of tasks, encompassing image classification, natural language processing [15], and prognosis prediction [16], among others [17].

WSIs are digital slides derived from high-resolution scans of physical pathology slides, capturing detailed visual information about tissue structure and cellular morphology [18,19,20]. The high-resolution and multiscale nature of these images permit the representation of both the spatial context and the local texture within the tissue. This abundance of information makes WSIs a profoundly rich data source for deep learning models, enabling them to extract and learn complex patterns that may correlate with a patient’s prognosis. However, the substantial size and complexity of these images also pose unique computational and methodological challenges that necessitate skilled handling for effective utilization.

The convergence of deep learning and WSIs represents an exciting avenue in cancer prognosis [22,23,24]. The dynamic interaction between the pattern discernment capabilities of deep learning algorithms and the voluminous, multiscale information inherent in WSIs facilitates an intricate depiction of the disease, potentially paving the way towards improved prognosis predictions. Despite its substantial potential, this intersection presents an array of challenges, not least of which include the necessity for substantial volumes of labeled data, the computational demands associated with processing high-resolution images, and the interpretability of deep learning models. It is imperative to address these challenges to successfully translate this technology into clinical practice, thereby offering a pathway towards more individualized and efficacious cancer treatment.

The complexity of cancers is closely entwined with the elaborate structural variations observable at the tissue level [25], and WSIs embody a rich source of information that captures this complexity across various scales. With the onset of digital pathology and increased accessibility of whole slide scanners, there has been a considerable surge in the availability of WSIs, thus providing a propitious environment for the application of advanced deep learning methodologies. Consequently, deep learning has been progressively incorporated into the pathology workflow, enhancing the human capacity for microscopic image analysis, furnishing prognostic predictions, and thereby offering a tangible route towards personalized cancer treatment.

2. Deep Learning with Whole Slide Images in Cancer Prognosis

In the domain of cancer prognosis, significant strides have been made through the application of deep learning methodologies to WSIs. This approach has enabled researchers to develop predictive models for a wide range of cancer types. It is imperative to note that the interpretation of this rich and complex data has necessitated a myriad of sophisticated techniques, many of which have been adeptly crafted to fit the peculiarities of specific cancer types (Table 1).

Ref.	Deep Learning Methods	Expected Strengths	Expected Limitations
[83]	Multihead Attention (Attention Mechanisms)	Comprehensive WSI analysis outperforms existing approaches and contributes to prognosis prediction.	Not specified
[55]	General Deep Learning (including MLP)	Potential biomarkers discovered provide enhanced prognostic performance.	Interpretability and generalizability limitations may hinder clinical acceptance.
[69]	ResNet	Cost-effective tumor mutation burden measurement and prognostic biomarkers outperform original TMB signature.	Not specified
[78]	Deep Multimagnification Network	Highly correlated necrosis ratio estimation and outcome prediction.	Dependence on manual review of necrosis ratio from multiple slides.
[46]	Federated Learning	Privacy-preserving multicentric studies with interpretable ML model.	Biases from small-scale study and time-consuming expert annotations.
[53]	CNN	Potential for multimodal data use in clinical applications with high diagnostic accuracy.	Not specified
[72]	ResNet, Attention Mechanisms	Risk stratification facilitated in ovarian cancer through deep learning framework.	Moderate mean value of C-index; uneven prediction strength across subgroups.
[42]	Multiple-Instance Learning (MIL), GAT, Attention Mechanisms	Novel MIL fusion model enables accurate prognostic risk prediction.	Not specified, potential challenges with image segmentation and representation.
[62]	ResNet-50	MPIS integration with clinicopathological variables improves LUAD prognostic stratification.	Transferability of MPIS to all cancer types uncertain.
[50]	Weakly Supervised Deep Learning	Accurate bladder cancer diagnosis and personalized treatment decisions.	Not specified
[49]	General Deep Learning (including MLP)	The proposed model improves survival prediction in bladder cancer by assessing TILs.	Not specified
[71]	CNN, Attention Mechanisms	High-performance prognosis prediction in Epithelial ovarian cancer using AI mechanisms.	Not specified
[33]	General Deep Learning (including MLP)	High-accuracy colorectal cancer prognosis using a weakly supervised deep learning network.	Not specified
[29]	General Deep Learning (including MLP)	Deep learning-based immune index correlates strongly with colorectal cancer survival rates.	Not specified
[57]	General Deep Learning (including MLP)	Multimodality prognostic model provides high-performance survival prediction in hepatocellular carcinoma.	Not specified
[48]	General Deep Learning (including MLP)	Depiction of tumor microenvironment immunophenotypes offers insights into biological pathways in bladder cancer.	Not specified
[41]	Sparse Representation Learning	The proposed model improves risk stratification in breast cancer with integrated biomarkers.	Effectiveness tied to biomarker extraction quality; untested outside of breast cancer.
[73]	CNN with Autoencoder	Deep learning-based pathological risk score predicts cervical cancer prognosis.	Prediction performance tied to dataset quality; clinical application untested.
[65]	Autoencoder with Regularization	CMS discovery allows personalized diagnosis in lower-grade gliomas.	Limitations with validating subtypes for other cancer types and accounting for inter-tumor heterogeneity.
[76]	ResNet	The proposed model identifies morphological features associated with metastasis in cSCC.	Performance tied to data quality and diversity; untested outside of cSCC.
[38]	Deep Learning with Multiresolution	Deep learning method for breast cancer survival integrates image data, improving model performance.	Needs more validation; performance varies with data quality.
[28]	Variational Autoencoder (VAE), Generative Adversarial Network (GAN)	Improved prognostic signature for stratifying outcomes in stage III CRC.	Limited generalizability to other cancer types or stages.
[35]	Spatial Pyramid Network	Automated CRC risk stratification approach related to gland formation.	Model may require further refinement despite better discrimination.
[44]	ResNet-34	TIL infiltrates assessment in breast cancer WSIs acts as significant biomarkers.	Dependence on TIL infiltrates; performance in TIL absence unclear.
[32]	General Deep Learning (including MLP)	Prognostic utility for CRC PFS prediction based on automatic TIL quantification.	Performance tied to TIL quantification; unclear performance in TIL absence.
[64]	General Deep Learning (including MLP)	End-to-end multimodal fusion improves survival outcome prediction.	Performance tied to availability of paired WSI, genotype, and transcriptome data.
[34]	CNN	The proposed model for CLR and TIL quantification improves survival prediction in CRC.	Needs further validation on larger cohorts for generalizability and clinical deployment.
[70]	ResNet-34	The proposed model achieves high accuracy for prognosis in OCCC.	Single-institution data may limit model generalizability.
[80]	General Deep Learning (including MLP)	The proposed model reduces interoperator variation in survival prediction from WSIs.	Efficiency compromised by WSI size and pattern heterogeneity.
[30]	CNN	Stroma-immune score using deep learning improves survival prediction in CRC.	Larger validation cohorts needed for reliable assessment of model’s prognostic value.
[66]	ResNet-18	Improved prognosis and IDH mutation status prediction in lower-grade gliomas.	Small sample size may limit robustness and generalizability.
[60]	CNN	The proposed model utilizes multiscale pathology images for prognosis prediction in lung adenocarcinoma.	Not specified
[61]	EfficientUnet, ResNet	Efficient analysis of immune checkpoints and prognosis of NSCLC.	Not specified
[68]	CNN	Accurate RCC subtype diagnosis and prediction of survival outcomes.	Interrater variability and limitations in capturing all biological signals.
[58]	Weakly Supervised Deep Learning	Prognostic indicators from HCC pathological images improve risk stratification.	Efficiency and labor-saving limitations; needs further validation for patient treatment.
[51]	Ensemble Learning	Prediction of MIBC prognosis significantly higher than TNM staging system.	Further validation and clinical deployment needed.
[40]	CNN	Efficient assessment of TILs in triple negative breast cancer provides valuable prognostic information.	Optimal prognostic information yielding method unclear; lack of objective TIL assessment methods.
[75]	CNN	High accuracy in predicting metastasis risk in pancreatic neuroendocrine tumors.	Not specified
[27]	General Deep Learning (including MLP)	Accurate mucus proportion quantification in colorectal cancer suitable for clinical application.	Not specified
[47]	General Deep Learning (including MLP)	Integrative analysis of histopathological images and genomic data improves understanding of disease progression.	Might not identify all potential regulatory regions in the human genome.
[54]	General Deep Learning (including MLP)	Two deep learning algorithms aid risk stratification for hepatocellular carcinoma patients.	Not specified
[77]	Convolutional Neural Networks (CNN)	Prognostic model predicts treatment failure in nasopharyngeal carcinoma better than existing clinical models.	Not specified
[43]	General Deep Learning (including MLP)	The models developed can spatially characterize tumor heterogeneity. Showed a significant statistical link between heterogeneity and survival.	Lack of automated methods for characterizing tumor heterogeneity.
[26]	CNN, Transfer Learning	Automated deep learning method for TSR quantification in colorectal cancer reduces pathologist workload.	Not specified
[74]	CNN	CNN-based system distinguishes tissue types with high accuracy in gastric diseases.	Not specified
[82]	Transfer Learning	Deep transfer learning quantifies histopathological patterns across a range of cancer types.	Not specified
[45]	CNN	High-resolution TIL map generation on WSIs strongly associates with immune response pathways and genes.	Not specified
[63]	CNN	Exceptional accuracy in brain cancer survival rate classification based on histopathological images.	Challenges in generalizability on unseen samples and practical clinical application.
[39]	General Deep Learning (including MLP)	Deep learning classifier identifies breast cancer molecular subtypes and heterogeneity.	Potential inaccuracies due to intratumoral heterogeneity.
[59]	General Deep Learning (including MLP)	Two-step deep learning approach accurately detects lung cancer metastases.	Presence of false positives in model predictions.
[79]	General Deep Learning (including MLP)	TILAb score predicts disease-free survival in OSCC patients better than manual TIL score.	Accuracy tied to quality and clarity of WSIs.
[67]	Convolutional Neural Networks (CNN)	High accuracy distinguishing renal cell carcinoma subtypes and predicting patient survival.	Class imbalance issues in medical datasets.
[81]	Multimodal Neural Network	Model combining clinical, mRNA, microRNA data, and WSIs predicts survival for 20 cancer types.	Not specified; potential complexity in interpreting multiple data modalities.
[36]	General Deep Learning (including MLP)	Automated approach determines TSR as an independent prognosticator in rectal cancer.	Applicable only in user-provided stroma hot-spots; performance tied to input image quality.
[37]	General Deep Learning (including MLP)	Deep learning algorithm for cell identification in colon cancer images improves performance.	Patch selection for analysis may impact results.
[52]	CNN	Quantification of tumor buds in bladder cancer adds prognostic value to traditional TNM staging.	Not specified
[56]	CNN	Recurrence-related histological score allows for clinical decision making in HCC recurrence prediction.	Prediction accuracy varies; potential bias towards trained data and diseases.
[31]	CNN	Automatic evaluation of the tumor microenvironment in WSIs aids in predicting disease progression.	Varied strength of predictors; potential bias towards specific cancer types.