Five items of the RQS in which all included studies performed poorly are “prospective study”, “phantom study”, “biological correlates”, “cost-effectiveness analysis”, and “openness of data and code”. Studies are given the highest weighting in the RQS tool (7 points, accounting for around 20% of the full scale). Phantom study process ensures that only robust features are included in the following radiomics analysis. Biological correlates aim to link imaging findings with gene or molecular signatures. Previous studies have detected a 91-gene signature that highly correlates with vascular invasion in HCC [43]. Based on this finding, a contrast-enhanced CT imaging biomarker, i.e., radiogenomic venous invasion (RVI), which includes three imaging features (internal arteries, a hypo-dense halo, and a tumor-liver difference), has been shown to be an accurate predictor of MVI [44]. Future studies are required to explore and verify the correlations between radiomics features and gene expressions. A cost-effectiveness analysis can evaluate a radiomics prediction model in terms of health economics when applied in clinical routines. It assumes that a novel predictor should not be more expensive than currently available predictors when accuracy is comparable. It also compares the health effect of a radiomics predictor with a condition without a radiomics predictor, such as a quality-adjusted life year analysis. researchers think that evaluating this point seems less urgent, given that the methodological standardization and clinical/biological validation of current radiomics models are still lacking. Data and code openness aims to repeat and reproduce results and findings and to further validate and promote the prediction model in other centers. Though some initiatives have been proposed in an attempt to remove the obstacles in data sharing, other factors, such as legal/privacy issues, culture/language barriers, and insufficient staff/time, still exist [10]. None of the studies shared their codes or imaging data publicly.

Regarding the items of “imaging at multiple time points” and “multiple segmentations”, both aim to select stable imaging features for modelling considering subjective and temporal variations. However, less than half of the studies performed ICC analysis and seldom explicitly stated that imaging features from different phases/sequences were evaluated during that analysis (i.e., test–retest analysis). Furthermore, there is no generally accepted ICC threshold at which radiomics features can be considered robust. Generally, when reporting ICC, values of 0.75–0.90 are regarded as indicating good reliability, and values higher than 0.9 are regarded as excellent [45]. However, among the studies that calculated ICC, the applied threshold varied among 0.75, 0.80, and 0.9. A future study should be applied to determine the proper threshold at which robust radiomics features for modelling can be defined. Some of the studies did not rule out features with low ICC and constructed their model using only the full features extracted from their images.

When evaluating the performance and clinical utility of the radiomics model considering the items of “cut-off analysis”, “calibration statistics”, “comparison with gold-standard”, “potential clinical utility”, and “validation”, the included studies again were insufficient. The performance metrics of a model, such as the sensitivity and specificity, are often determined by a specified cut-off value, and this value can further classify a patient cohort into high and low risk groups for a certain condition. A cut-off value is also one of the prerequisites for reproducing the results of previous research. However, only five studies reported their cut-off values. Regarding calibration analysis, which evaluates the agreement between predictions and the actual events, less than half of the studies performed one. Regarding the comparison with “gold-standard”, there is currently no surrogate that can serve as a “gold-standard” for MVI prediction. As the value of semantic imaging features have been extensively explored for MVI prediction, we therefore defined conventional imaging features as the “gold-standard”. Among the 10 studies that compared prediction performance between radiomics and radiologist models, all declared that the radiomics models outperformed the radiologists’ semantic models. However, the publishing bias should be borne in mind when interpreting these results. Only two studies validated their models using independent external cohorts. However, one of them validated their model in only 18 patients, which is not a sufficiently large validation cohort according to the “10-EPV” principle (at least 10 events per variable) [16,46]. When developing a prediction model, the ratio of event and variable should be maintained at a certain level to avoid potential overfitting or underfitting. Among the 16 studies with an EPV ratio available, the median EPV (MVI positive cases/features) ratio was 4.2, indicating a potential risk of overfitting. Therefore, it is assumed that, before translating these models into a clinical routine utility, some practical issues should be well addressed, such as the reproducibility of the radiomics model, the standardization of imaging protocols, model overfitting, and the external validation of the prediction models.

Though the RQS tool aims for high-quality radiomics research, there are concerns that should be optimized in future revisions. The RQS is mainly focused on radiomics itself and ignores non-radiomics components during radiomics model/predictor development, such as blindness to outcomes and measurement, intervals between the index test and reference standard (in the case of MVI, the time between imaging and liver resection), and the influence of sample size and enrollment of study subjects. All these factors may also introduce bias. Under this context, the tool of QUADAS-2 can serve as a vital supplement to RQS when evaluating the quality of radiomics research. Most of the studies reported in this systematic search showed a low or unclear risk in the four domains of risk of bias evaluation. The missing or unclear parts observed using the RQS and QUADAS-2 tools were obvious, which implies that these tools might not be so well known or adopted. Future researchers will ideally apply the RQS or QUADAS-2 as a checklist to improve the quality of their reports. In fact, a specified checklist, i.e., CLAIM (Checklist for Artificial Intelligence in Medical Imaging) for artificial intelligence research [47], and a general guideline for diagnostic/prognostic prediction, i.e., TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) [15], have already been proposed.