It is important to first distinguish between population parameters, a sample of observed data, and statistical estimators. A parameter, which is easily understood but not observed, is a population quantity such as median survival time, response probability, or the effect of a covariate or treatment on a clinical outcome. We denote a clinical outcome by Y, which may be survival time or a response indicator, and write y for an observed value of Y, such as y = 12 months. The same convention will be used for other variables that may take on different values in some random fashion, such as X for treatment and x for a particular treatment x, for example, denoting X = 1 for pembrolizumab and X = 0 for placebo or standard of care. A statistic, which is anything that can be computed from observed data using a formula or algorithm, may be used to estimate a parameter. For example, to estimate a population median, a computed sample median may be 14.5 months with a 95% CI of 9.4 to 18.7, keeping in mind that a second sample would give different numbers because outcomes and samples are random. A Kaplan–Meier plot [21] is a statistical estimator of the survival function of the population represented by the sample used to compute the plot, and here the population survival function is the parameter. The underlying statistical principle is that, ideally, a sample of Y values should be obtained in such a way that it represents the population of interest, and statistical sampling theory provides a wide array of methods for ensuring this representation [22,23,24]. The design of experiments such as clinical trials is distinct from sampling theory and can be facilitated by the explicit representation of the causal relationships we wish to estimate.

Relationships between patient covariates, treatments, and clinical outcomes can be represented by causal diagrams, such as directed acyclic graphs (DAGs) [4,25,26]. The simplest causal relationship occurs when a variable X, known as the “exposure” in epidemiology or “treatment” in clinical medicine, directly transmits its effect on an outcome Y of interest (Figure 1A). In the RCC setting, the treatment, either adjuvant pembrolizumab (X = 1) or placebo (X = 0), acts directly on the time to disease recurrence or death, defined as Y = DFS time (Figure 1B). A prognostic variable Z may also directly affect DFS time (Figure 1C,D), and in most settings, there is a vector of several prognostic variables. A confounder C is a variable that influences both the outcome Y and treatment X (Figure 1E). For example, the presence (C = 1) or absence (C = 0) of sarcomatoid features acts as a confounding variable if it influences both DFS time and whether a physician chooses to treat a patient with adjuvant pembrolizumab (Figure 1F).

Another type of causal relationship occurs when the effect of treatment on the outcome is not entirely direct, but rather is transmitted, at least in part, through a third variable, known as a mediator, M (Figure 2A). For example, the effect of adjuvant pembrolizumab compared with placebo on DFS time is mediated by the RCC immune microenvironment (Figure 2B). The immune microenvironment of each RCC histologic subtype responds differently to immune checkpoint therapies, such as pembrolizumab [8], and the microenvironment, in turn, affects DFS time. Consequently, the effect of a given treatment on DFS time should vary with the immune microenvironment of each RCC histologic subtype because the RCC immune microenvironment acts as a mediator.

As additional variables are included, causal diagrams become more complex. For example, we can also include the influence of epithelial-mesenchymal transition (EMT) on sarcomatoid dedifferentiation, as shown in Figure 2C. A more detailed causal diagram may be obtained by including β-catenin levels that causally influence EMT (Figure 2D) [27]. As more variables are added, a causal graph may become too complex to interpret usefully. An important practical goal thus may be to reduce an overly complex causal graph to a more coarse-grained causal description that can be used to deal with the problem at hand [28,29,30,31]. For example, the causal diagram in Figure 2C may be appropriate if a drug that modifies EMT is being studied. Similarly, the more fine-grained causal diagram in Figure 2D may be used in a study evaluating the prognostic role of assays measuring β-catenin levels. Choosing the right level of granularity for a causal diagram is a subjective decision that should be guided by the goal to use the diagram as a practical tool. This can be performed by investigating how causal relationships may change when finer or coarser resolutions are chosen.

An important refinement of a causal diagram is to identify variables by which populations, or studies that are designed to represent them, may differ [5]. This is achieved by including graphical objects called selection nodes that, while they are not variables themselves, use arrows to point to variables that have different sets of possible values between populations [5,32]. If an arrow points from a selection node S to a variable Z (Figure 3A), this says that the possible values of Z differ between populations. If no arrow points to Z, then all populations of interest have the same set of possible Z values. A third scenario occurs if an arrow points from a variable Z to a selection node S (Figure 3B), which indicates the presence of selection bias, wherein there are differences that are not between the populations but instead are between the datasets due to sampling artifacts. Selection nodes with outgoing arrows are not mutually exclusive with selection nodes with incoming arrows, and both types of arrows can appear in the same selection diagram if they represent two distinct features of the data collection process [33,34]. In the present review, we will focus only on transportability scenarios wherein the differences between populations are inherent, which is denoted by selection nodes S having outgoing arrows toward variables (Figure 3A).

A selection diagram is a DAG that includes one or more selection nodes. A selection node may be used to show where treatments (Figure 3C) or mediators (Figure 3D) are different across populations. Whether or not a given variable has a selection node can be determined from either external knowledge or the available dataset itself. A selection node identifies a variable in a causal chain that may be affected by study differences. Selection nodes provide an explicit way to determine whether inferences are transportable between populations and, if not, what additional data are needed to ensure transportability [5,32,35,36]. A major practical point is that a selection node may show that the data from an RCT are not representative of a patient seen in the clinic. In this case, either the trial’s results cannot be used to choose that patient’s treatment, or possibly additional data may be obtained and combined with the RCT data to provide a basis for choosing a treatment.

A selection node can be used to show that the effects of treatments (Figure 3C) or mediators (Figure 3D) are different between populations. For example, if an RCT of two RCC treatments is performed in the United States and another RCT of the same treatments is conducted in China, since the US trial will include African Americans and the Chinese trial will not, a selection node should point to the variable “race” (Figure 3E). As another example, if one study includes a treatment given at dose levels 1, 2, and 3 and a second study includes the same treatment at dose levels 3 and 4, then a selection node should point toward treatment (Figure 3C). If, for a targeted biomarker, an “all-comers” population includes patients who were either biomarker positive or negative, while a second study population only includes biomarker-positive patients, then a selection node should point toward the biomarker (Figure 3F). A selection node can indicate that patient age differs between the population of patients being seen in the clinic, which includes patients older than 65 years, and the population of a study that restricted patients to be younger than 65 years (Figure 3G). Because Figure 3G indicates that patient age influences outcome, external data from studies that included patients older than 65 years are needed to obtain valid inferences on the outcome for such patients seen in the clinic. A selection node may also indicate that some studies included patients with sarcomatoid features whereas others did not (Figure 3H), or that a study only included patients with clear cell RCC and no other histologies (Figure 3I). The general practical point is that, if the variables denoted by selection nodes are substantively different than those in a study population, then a conclusion from that study alone cannot be transported to that patient.

In contrast, the range of a variable without a selection node does not change across populations. For example, the lack of a selection node pointing toward “treatment” in Figure 3D indicates that the treatment choices do not vary between populations. Such assumed invariance in specific causal mechanisms allow inferences to be transported from a study population to patients in a different population, e.g., a patient seen in the clinic [5,20,36].

The relationships represented by all types of DAGs, including selection diagrams, can be used to guide a set of mathematical rules known as the do-calculus [25,37], which is expressed using conditional probabilities. The central idea is to estimate causal effects by distinguishing between two types of conditional probabilities. The first type conditions on a variable, say X, that simply was observed and thus may have been confounded by other, possibly unknown variables. The second type conditions on a known value x of X that was determined by one’s action or intervention, represented by do(X = x), and thus, X is free of confounding effects. The mathematical expression do(X = x) is known as the do() operator and is used to denote a known intervention. For example, in an RCT, if a patient with covariates Z and unknown confounders C (Figure 4A) was randomly assigned adjuvant pembrolizumab (X = 1), then the do() operator may be used to replace the conditional probability P^(RCT)(Y | X, Z) with P^(RCT)(Y | do(X = 1), Z). This ensures that there are no confounding effects on X, so the causal effect of pembrolizumab versus a control treatment (X = 0) for a patient with covariates Z may be obtained from statistical estimates of P^(RCT)(Y | do(X = 1), Z) and P^(RCT)(Y | do(X = 0), Z). These estimates can be computed from the RCT data under an assumed statistical regression model including the effects of X and Z on Y. In contrast, if X is observed and not assigned, then the values of X in a dataset may have been affected by known or unknown confounding variables, which can cause severe bias and invalidate conventional statistical estimators.

In terms of causal diagrams, the expression P^(RCT)(Y | do(X = x), Z) modifies the graph in Figure 4A by removing all arrows going into X (Figure 4B). This results in a new causal model (Figure 4C) wherein the distribution P^(RCT)(Y | do(X = x), Z) is the same as P^(RCT)(Y | X = x, Z). This modification represents the physical effect of the experimental intervention that sets the value X = x, while keeping the rest of the causal model unchanged.

To illustrate how the do() operator works in practice, a common example in the RCC setting is one where a patient in an observational dataset (denoted as OBS), not obtained from an RCT, has X = 1 recorded, indicating that the patient received pembrolizumab, but it is not known how the patient’s physician chose that treatment. In statistical language, treatment selection was biased in the observational dataset. Consequently, in contrast with what is widely believed, fitting a regression model for DFS time Y as a function of treatment X and covariates Z to the observational dataset may not provide an unbiased statistical estimator of the effect of pembrolizumab versus standard of care. From a causal viewpoint, for a patient with covariates Z, the do() operator cannot be applied due to the treatment assignment bias, so while P^(OBS)(Y | X = x, Z) can be computed, P^(OBS)(Y | do(X = x), Z) cannot be computed for either X = 0 or X = 1. If, instead, the data came from an RCT of pembrolizumab (X = 1) versus surveillance (X = 0), then one can estimate P^(RCT)(Y | do(X = x), Z) from the data for each x, since randomization removes any arrows into X. This allows one to obtain an unbiased estimator of the causal effect E(Y | do(X = 1), Z) - E(Y | do(X = 0), Z), which is the difference in expected DFS times for pembrolizumab versus standard of care for a patient with covariates Z. This example illustrates the scientific difference between recording data from medical practice, where physicians choose treatments based on their knowledge and each patient’s baseline information, and data from an RCT, where each patient’s treatment is chosen by randomization in order to obtain a statistically unbiased treatment comparison for the benefit of future patients. That is, data obtained from clinical practice are inherently biased due to each physician using available information to choose the treatment that they think will be best for each patient, while data from an RCT, where each patient’s treatment was chosen by flipping a coin, are unbiased. The point is that randomization is a statistical device to obtain unbiased estimators in order to maximize the benefit for future patients when choosing their treatments.

Three inferential rules, known as the do-calculus, have been derived to transform conditional probability expressions involving the do() operator into other, more useful conditional probability expressions. The first rule considers a DAG, such as the one shown in Figure 4D, where we are interested in the probability distribution of the outcome Y following treatment X conditioned on all patient covariates, which we denote by W and C. As noted earlier, in an RCT, which sets do(X = x), all arrows into X are removed (Figure 4E), resulting in the causal model shown in Figure 4F. Rule 1 of the do-calculus guides the insertion and deletion of variables by stipulating that if, after deleting all paths into X, the set of variables C and treatment choice X = x block all the paths from W to Y regardless of the direction of the arrows on these paths (Figure 4F), then P^(RCT)(Y | do(X = x), C, W) = P^(RCT)(Y | do(X = x), C). In probability language, the do() operator ensures that Y is conditionally independent of the covariates W, given do(X = x) and the covariates C. The practical implication of Rule 1 is that it allows us to simplify our regression model by dropping the covariates W. For example, suppose that C is a prognostic score recorded in an RCT of pembrolizumab versus surveillance, W is the frequency of monitoring for treatment toxicity based on each patient’s prognosis and chosen treatment, and Y is DFS time. The path from W to Y is blocked by conditioning on C, and therefore Y is conditionally independent of W given C and do(X = x). Accordingly, in the RCT of pembrolizumab versus surveillance, we should condition on the prognostic score but not on the frequency of monitoring for treatment toxicity.

Before we move to Rule 2 of the do-calculus, we first need to define back-door path. A back-door path is a non-causal path from the treatment X to the outcome Y. As shown in Figure 4G,H, all paths from X to Y that remain after removing all arrows pointing out of X (Figure 4H) are back-door paths because they flow backward out of X into Y through C. Rule 2 of the do-calculus describes a relationship between an outcome Y, an intervention X, and a vector C of covariates by stipulating that, if the covariates in C block all back-door paths from X to Y (as shown for example in Figure 4D,G), then P(Y | do(X = x), C) = P(Y | X = x, C). This rule requires the strong, unverifiable assumption that there are no unknown external confounders that affect both Y and X and do not belong to the observed vector C. Under this assumption, the probability equality says that it is not necessary to randomize patients between X = 1 and X = 0 because knowledge of C allows the do() operator to be applied. The practical implication, however, is not that one may simply fit a regression model for P(Y | X, C) to observational data to estimate the covariate-adjusted treatment effect in an unbiased way. This is because the assumption underlying Rule 2 cannot be verified. In practice, statistical bias correction methods are applied, such as inverse probability of treatment weighting (IPTW), described in Section 2.4 below, that use the available covariates in C to provide an approximation to the strong assumption that a vector C of observed covariates blocks all back-door paths from X to Y. Thus, Rule 2 formalizes the motivation for applying statistical methods that correct for bias, or at least mitigate it, when analyzing observational data. Such methods are discussed briefly in Section 3, below.

Rule 3 of the do-calculus guides the insertion and deletion of interventions by stipulating that, if there is no path from X to Y with only forward-directed arrows (Figure 4I), then P(Y | do(X = x), Z) = P(Y | Z). In this case, observing the covariates Z implies that treatment (or exposure) X has no causal effect on outcome Y. An example from epidemiology is the once common belief that soda pop consumption caused poliomyelitis, due to the observation that higher sales of soda pop (X) were positively associated with a higher incidence of poliomyelitis (Y). However, once the season (Z) of the year was recorded, it was seen that both soda pop consumption and poliomyelitis incidence were higher during the summer months (Z = 1) and lower during the winter months (Z = 0). Thus, Y was conditionally independent of X given Z. This phenomenon is commonly described by saying that association does not imply causation, where an apparent effect of X on Y is due to the fact that Z drives both X and Y. In this case, Z is sometimes called a “lurking variable” acting as a confounder, and X is called an “innocent bystander.” For a therapeutic example, suppose that X = 1 corresponds to standard treatment combined with a completely ineffective experimental agent and X = 0 is standard treatment alone, while Z = 1 denotes good prognosis and Z = 0 denotes poor prognosis. If X = 1 is more likely to be given to good-prognosis patients (Z = 1) and less likely to be given to poor-prognosis patients (Z = 0), then the resulting data will show that X and DFS time (Y) are positively associated, with estimates Ê[Y | X = 1] > Ê[Y | X = 0] apparently implying that adding the experimental agent increases expected DFS time. However, once one accounts for prognosis (Z) the estimates are the same, that is, Ê[Y | X = 1, Z] = Ê[Y | X = 0, Z], aside from random variation in the data. This is because P(Y | do(X = x), Z) = p(Y | Z). That is, DFS time (Y) is conditionally independent of the choice of treatment (X) given prognosis (Z). While this example may seem somewhat contrived, the practice of combining an active control with a new experimental agent and cherry-picking patients with a better prognosis for evaluating the combination in a single-arm trial is not uncommon in oncology [38,39].

Causal relationships represented by DAGs and expressed by conditional probabilities involving the do() operator apply to one domain, such as the population represented by patients enrolled in an RCT. To use a DAG to determine whether causal knowledge can be transported to another domain, such as the population of patients a physician sees in the clinic [5,36,37], it is useful to distinguish between three types of causal problems. These comprise a hierarchy of increasing difficulty, called the ladder of causation (Table 2) [40]. The simplest type of causal problem, layer 1, is predicting an event in a population based on association. A conditional probability, such as P(salmonella infection | diarrhea), characterizes association and may be estimated statistically from data on salmonella and diarrhea, but it is inadequate for addressing layer 2 or layer 3 causal problems without additional experimental knowledge or assumptions. Layer 2 involves determining what happens when an intervention, such as an RCT, is performed in a cohort of patients. The do() operator can be used to denote such interventions, and the do-calculus rules can be used to transform expressions by introducing do() operators in layer 2 conditional probabilities. Layer 3 involves choosing between treatments for an individual patient, and this class of queries requires the potential outcomes framework, which is described below.

To explain potential outcomes, we return to the setting where a physician wishes to decide between adjuvant pembrolizumab (X = 1) and surveillance monitoring (X = 0) for a patient with covariates Z. The physician may consider what the patient’s future outcomes would be for each treatment, which we write as Y^{X = 1} and Y^{X = 0}. These are called potential outcomes [41,42,43] because only one treatment can be given and, thus, only one of Y^{X = 1} or Y^{X = 0} can be observed. This thought experiment corresponds to an imaginary world where one can make two identical copies of each patient, treat one with adjuvant pembrolizumab and the other with surveillance monitoring, and observe both Y^{X = 1} and Y^{X = 0}. Writing the expected value of Y for the copy treated with adjuvant pembrolizumab as E(Y^{X = 1} | Z) and for the copy treated with standard of care as E(Y^{X = 0} | Z), the difference is the causal effect of treating the patient with adjuvant pembrolizumab rather than surveillance monitoring.

Of course, in the real world one cannot make two copies of a patient. Only one of the two treatments can be given to a patient, and thus only one of each patient’s potential outcomes can be observed. If the patient is treated with pembrolizumab, then X = 1, and the potential outcome Y^{X = 0} for the treatment X = 0 is called a counterfactual. This thought experiment has a very useful practical application because it provides a basis for computing an unbiased statistical estimator of Ψ using data from either an RCT or an observational study, under some reasonable assumptions. First, thinking about Y^{X = 1} and Y^{X = 0} only makes sense if, for example, a patient actually treated with adjuvant pembrolizumab has an observed outcome equal to the potential outcome for pembrolizumab, that is, if Y = Y^{X = 1} when X = 1. Similarly, if the patient actually receives surveillance monitoring, X = 0, then Y = Y^{X = 0}. This is called consistency [43,44,45]. Defining the propensity scores as treatment assignment probabilities Pr(X = 1 | Z), both treatments must be possible for each Z, formally 0 < Pr(X = 1 | Z) < 1, called positivity. The potential outcomes Y^{X = 1} and Y^{X = 0} also must be conditionally independent of X given Z. Defining the causal estimator as