The top-quark flavour-changing neutral current (FCNC) effects can be probed directly in the production of a single top-quark, as well as in the top-quark decays. Studies of the top-quark FCNC decays are typically associated with similar sensitivities to the top-quark FCNC couplings with an up and a charm quark. Experimental sensitivities to these couplings mainly differ in terms of the performance of various reconstruction methods used for the identification of hadronic jets originating from quarks of a different flavour. At hadron colliders, the single top-quark FCNC production process is mostly sensitive to the top-quark FCNC coupling with an up quark (or an up antiquark) due to an enhanced sensitivity due to the proton distribution function of the colliding protons (or antiprotons). The importance of these two production channels depends on a specific type of the top-FCNC coupling that is probed in an experiment. Before the LHC, the top-FCNC couplings were studied in electron–positron collisions at LEP2 [34^[1][2][3][4],35,36,37], in deep inelastic scattering processes at HERA [38^{[5][6][7][8][9]},39,40,41,42], and in proton–antiproton collisions at Tevatron [43,44,45,46]^{[10][11][12][13]}. The electron–positron colliders allow for a study of the top-γ and top-Z couplings in the processes with the production of a single top quark, e+e−→tc¯(u¯). The study of the deep inelastic scattering of electrons on protons has an enhanced sensitivity to the same type of couplings in the processes of ep→et+X, as well as to the top-gluon FCNC couplings in the ep→etq(g)+X processes. The obtained experimental constraints were recently improved by almost one order of magnitude after the analysis of the LHC proton–proton collision data [47,48,49,50,51,52,53,54,55,56]^{[14][15][16][17][18][19][20][21][22][23]}. The top-Higgs FCNC transitions receive the largest suppression in the SM with respect to the other top-quark FCNC processes because of the large mass of the Higgs boson. These transitions are among the rarest processes predicted in the SM in the quark sector, and therefore, the study of these processes is associated with a generally enhanced sensitivity to potential new physics effects. The discovery of the Higgs boson at the LHC paved a way to a comprehensive study of the top-Higgs FCNC processes at the ATLAS and CMS experiments, which resulted in the first experimental constraints on these anomalous couplings [57,58,59,60,61,62,63,64,65]^{[24][25][26][27][28][29][30][31][32]}. The direct searches for the top-Higgs FCNC effects are performed in top-quark decays, as well as in the associated production of single top quarks with a Higgs boson. Many of the performed studies were targeting the top-quark FCNC decays in tt¯ events. In recent studies of the 13 TeV data, the analysis of the single top-quark-associated production with a Higgs boson was also included [62,63,64,65]^{[29][30][31][32]}.

2. h→γγ

Search channels that are relevant to the top-Higgs FCNC couplings are usually defined based on the Higgs boson decay channels. The Higgs boson decays to pairs of photons provide a clean experimental signature to look for the top-Higgs FCNC effects. In addition to the two photons, these final states consist of up to one isolated lepton with additional hadronic jets. The analysis strategy is primarily based on the reconstruction of the Higgs boson diphotonic invariant mass. The contributions from various background processes are fitted in the mass sidebands in the data, followed by its extrapolation to the signal region. In these fits, the background contributions that are associated with the SM Higgs boson production must be accounted for, representing one of the dominant resonant backgrounds in the search region. The uncertainty associated with the choice of the fit function, the statistical uncertainty in the data, as well as the background contributions from the SM processes involving the Higgs boson represent the main uncertainties in the study of these final states. The searches for top-Higgs FCNC processes in the h→γγ channel were carried out by ATLAS [59]^[26] and CMS [63]^[30] in the single-lepton and hadronic final states, including a pair of photons. The integrated luminosity of the recorded 13 TeV data corresponds to 36 and 137 fb−1, respectively. The identification of isolated photon objects and the common vertex of the photon pair is the core part of the analysis. The photon and the common vertex identification algorithms are based on the multivariate analysis (MVA) approaches. The obtained mass resolution allows the observation of a resonance structure in the diphoton invariant mass spectra in simulated signal events corresponding to the Higgs boson decay. The contributing nonresonant background processes include the diphoton production with jets, as well as the top-quark pair and the vector boson production processes with additional photons. The SM production of the Higgs boson represents the dominant resonant background. The nonresonant backgrounds are estimated directly from the data by performing a fit to the reconstructed diphoton invariant mass spectrum. The fitted function represents the sum of a double-sided Crystal Ball function that corresponds to the signal prediction, the resonant background from the SM Higgs production, and a parameterised function describing the nonresonant background obtained in a data control region. The main uncertainties include the b tagging and jet energy corrections, as well as the photon identification systematic uncertainties. The uncertainty in the limited number of events in the data also represents an important limiting factor in the final sensitivity in these searches. An additional contribution to the total systematic uncertainty is associated with theoretical uncertainties in the prediction of the resonant background processes with the SM Higgs boson production. The unbinned likelihood fit to the data using the described signal and background diphoton mass spectra is performed, and the constraints are set on the top-quark FCNC decay branching fractions.

3. h→WW/ZZ/ττ

Multilepton final states arise from the Higgs boson decays to a pair of W or Z bosons, as well as to τ leptons. The event categories in these studies are associated with the final states with two same-sign or three selected leptons. The same-sign lepton channel has dominant background contributions originating from processes with nonprompt and misidentified leptons, while the three-lepton channel is mainly affected by the presence of diboson events as well as nonprompt leptons. These backgrounds are estimated from the data. The search channels involving one hadronic τ lepton identified in the Higgs boson decay receive the dominant background contributions from the processes with misidentified τ lepton decays, as well as from events with the SM production of top quarks. In the case when the decays of both τ leptons result in hadronic final states, a significant background contribution is also associated with the Z boson decays to the pairs of τ leptons. The searches for top-Higgs FCNC couplings in the multilepton channels were performed at ATLAS [58]^[25] and CMS [61]^[28] using 36 fb−1 of 13 TeV and 20 fb−1 of 8 TeV data, respectively. The events are split into the final states with two same-sign (2lSS) and three (3l) leptons. The dominant backgrounds are associated with the nonprompt and misidentified leptons, as well as with the leptons originating from photon conversions. The prompt-lepton backgrounds correspond to events with an associated production of top-quark pairs and a W, a Z, or a Higgs boson, with additional contributions arising from the processes with diboson production. The baseline selection criteria require the presence of two or three leptons and at least two jets, with one or two b-tagged jets. The prompt lepton identification plays an important role in these studies in suppressing the dominant nonprompt lepton backgrounds. The rejection of nonprompt leptons is usually achieved through an application of an MVA approach, which exploits a number of kinematic variables that provide a separation power between the two lepton categories, such as lepton isolation and properties of the reconstructed jet in proximity of the selected lepton. The statistical and systematic uncertainties associated with the prediction of the backgrounds with nonprompt leptons are among the dominant uncertainties in these searches. Two separate boosted decision tree (BDT) discriminants involving various reconstructed kinematic variables are trained in the 2lSS and 3l channels to further suppress various backgrounds.  A dedicated study of the top-Higgs FCNC effects in the final states with one or two hadronically decaying τ leptons was recently performed by ATLAS with 139 fb−1 of 13 TeV data [65]^[32]. The analysis strategy is similar to the one used in the previous analysis [57]^[24], with an increased number of kinematic regions sensitive to the signal production, in order to account for the single-top production channel for the top-Higgs FCNC.

4. h→bb¯

The Higgs boson decays to a pair of b quarks with the largest branching fraction of ≃58% [66]^[33]. A considerable amount of background events is associated with the tt¯ production with additional hadronic jets. The analysis of this channel is systematically limited with the dominant contributions to the total uncertainty arising from the application of the heavy flavour jet identification techniques, as well as the modelling uncertainties relevant to the predictions of the top-quark production processes with additional jets. One of the important handles for suppressing background processes is the kinematic event reconstruction involving top quarks and additional jets. The assignment of the reconstructed final-state objects to the initial hard-process particles is performed using the MVA methods. The top FCNC search in the h→bb¯ channel is performed in the final states with one isolated lepton and additional jets. The total integrated luminosity used in the ATLAS analysis corresponds to 36 fb−1 of 13 TeV data [57]^[24]. The CMS results use 101 fb−1 of data [64]^[31], additionally combined with the previously published result from the analysis of 36 fb−1 data [62]^[29]. The ATLAS analysis focuses on the study of the event topology with at least four jets in the final state, mainly relevant to the top-quark FCNC decays. The corresponding CMS analysis additionally includes the signal top-quark production mode of the signal events, and therefore, the requirement on the minimum number of reconstructed jets is set to a lower value. At least three b-tagged jets are required to be present in the event. In both analyses, the selected events are classified based on the number of jets and b-tagged jets. The dominant background contributions correspond to the top-quark pair production in association with light-flavour jets in the event categories with two b-tagged jets, while the associated production of top-quark pairs with heavy-flavour jets (tt¯bb¯ and tt¯cc¯) represents the dominant background in the case of the higher number of b-tagged jets. The theoretical predictions for these processes are subject to relatively large uncertainties due to the renormalisation and the factorisation scale variations arising from the different energy scales of the top-quark mass and the jet transverse momentum involved in the generation process, as well as the inclusion of heavy quark masses in the calculations [67]^[34]. The experimental uncertainties in the measurement of the production cross sections of these processes reach ≃10–20% [68,69,70,71]^{[35][36][37][38]}. The background processes are further suppressed by using the discriminants that exploit the kinematic information of the selected reconstructed objects, defining the probability of an event to correspond to the signal process hypothesis.

5. Indirect Searches

The top-Higgs FCNC interactions can be indirectly constrained from the studies of the SM processes that can potentially include FCNC loop-level contributions involving top quarks. The relevant processes include the hadron electric dipole moments [72^[39][40],73], Z→cc¯ decays [74]^[41], and D0−D0¯¯¯¯ mixing [75]^[42]. The indirect limits are competitive with the current direct constraints obtained at the LHC [76]^[43].