The interdisciplinary reseauthorchers of this paperresearch would like to present recent advances in structural health monitoring (SHM), mainly related to Virtual Reality, Augmented Reality, and Mixed Reality for condition assessment of civil engineering structures, as the development of such novel technologies has been progressing quite considerably over the last few years. The Civil Infrastructure Technologies for Resilience and Safety (CITRS) laboratory at the University of Central Florida has a history of exploring novel technologies for civil engineering applications starting from the late 1990s ^[1]. Since then, multiple studies have been carried out including the earliest investigations of novel technologies, such as the employment of computer vision [2,3,4,5]^[2][3][4][5] in SHM applications. Additionally, the prior research of CITRS members has demonstrated the integration of digital documentation of design, inspection, and monitoring data (collected from a civil structure) coupled with calibrated numerical models (“Model Updating”) [6,7]^[6][7]. Briefly, model updating is a methodology to update a numerically built model in real/near time based on the information obtained from the monitoring and inspection data (field data). Fundamentally, this technique is nowadays known as digital twin. Over the last decade at CITRS, the investigation of using novel methods for condition assessment and SHM applications has continued extensively. One of the earliest machine-learning-based studies was conducted for SHM applications [8,9]^[8][9]. Furthermore, various other new implementations followed, such as using Mixed Reality (MR) to assist the inspectors on the field [10,11]^[10][11] and using Virtual Reality (VR) to bring the field data to the office where parties can collaborate in a single VR environment ^[12]. Another recent collaborative work discusses the latest trends in bridge health monitoring ^[13]. Generative adversarial networks (GAN) have been recently explored in the civil SHM domain ^[14]. They were investigated to address the data scarcity problem [15,16,17]^[15][16][17] and used for the first time in undamaged-to-damaged domain translation applications where the aim is to obtain the damaged response while the civil structure is intact or vice versa [18,19]^[18][19]. Members of the CITRS group are motivated to present some of the recent advances in SHM and other notable studies from the literature.

Civil engineering is arguably one of the oldest and broadest disciplines. It employs various theories, methodologies, tools, and technologies to solve different problems in the field. With the advancements in sophisticated computer graphics and hardware, a combination of artificial intelligence (AI), Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR) has been utilized more in the last decade (Extended Reality (XR) is an umbrella term for VR, AR, and MR technologies). While the first VR prototype, named “Sensorama”, was introduced in 1956 by cinematographer Morton Heilig, the first AR headset was only created by Ivan Sutherland in 1968, named “The Sword of Damocles”. These two efforts marked the beginning of the new field of XR. In the 1990s and early 2000s, there were many research and commercial efforts to advance the technology, and AEC became one of the disciplines that could best benefit from this technology. However, it was not until about ten years ago that, thanks to the advances in real-time graphics hardware and increased computation power in mobile and personal computers, XR became much more widespread and more feasible as a tool that could be utilized in different workflows. In 2022, the use of XR has extended to various types of applications in different industries such as education, military, healthcare, and architecture–engineering–construction (AEC). The recent progress in XR was possible due to the advancements in and miniaturization of graphics and computational systems and the significant reduction in cost and complexity of this technology. In addition, the introduction of the concept of “metaverse”, where users experience virtual collaborative environments online, has also accelerated research and development in the XR industry in recent years. XR technologies have various advantageous uses in the AEC industry, such as training and education, structural design, heritage preservation, construction activities, and structural condition assessment.

2. Virtual Reality, Augmented Reality, and Mixed Reality

The term Extended Reality (XR) includes VR, AR, and MR, and the areas interpolated among them, which is a set of combinations of different spectrums of the virtual world and real world as introduced by Paul Milgram [63]^[20]. In Figure 51, the reality–virtuality spectrum of XR is shown. As described in the figure, while VR is an entirely immersive experience surrounded by a digital environment which isolates the user from the real world, AR enhances the real world with a digital augmentation overlaid on it. MR is a blend of the real and virtual worlds, providing interaction between real and digital elements. In essence, MR involves the merging of real and virtual environments somewhere through the mixed reality spectrum (or virtuality–reality spectrum) (Figure 51), which connects completely real environments to completely virtual ones [64]^[21]. This is, however, quite different in VR, which isolates the user from the real world, and the user can interact with the digital elements in the virtual world. Virtual Reality (VR): As described previously, VR provides a completely digital environment where users experience full immersion. The applications that block the user’s view to provide a fully immersive digital experience are named Virtual Reality. At present, a head-mounted display (HMD) or multi-projected environments (e.g., specially designed rooms with multiple large screens: CAVE [65]^[22]) are the most common tools to immerse users in virtual spaces. The VR system allows the user to interact with virtual features or items in the virtual world. The main point of VR is the interactive, real-time nature of the medium provided to the user. Generally, while some VR systems incorporate auditory and visual feedback, some also enable different kinds of sensory and force feedback via haptic technology. Widely used HMDs are the Meta Oculus series, HTC Vive, Samsung Gear VR, and Google Cardboard. The VR applications are primarily used in education and training, gaming, marketing, travel, and AEC industries. Augmented Reality (AR): AR is a view of the real world with an overlay of digital elements where the real world is enhanced with digital objects. The applications that overlay holograms or graphics in the real world (physical world) are named AR. While the interaction with virtual superimposed content in an AR setting can also be made possible, AR applications are generally used with the aim of enhancing the real-world environment with digital objects, usually providing the user with no-to-minimal interaction with the virtual superimposed content in the AR setting. Currently, AR systems use devices such as HoloLens, Magic Leap, Google glass, Vuzix smart glasses, smartphones, tablets, cameras, etc. While some of the HMDs, e.g., HoloLens and Magic Leap, are used for AR applications, they are also used for MR applications. Similar to VR, AR has a wide range of uses, such as gaming, manufacturing, maintenance and repair, education and training, and the AEC industries. Mixed Reality (MR): The experiences that transition along the virtuality–reality spectrum (mixed reality spectrum) are named Mixed Reality. MR is also called Hybrid Reality, as it merges real and virtual worlds [66]^[23]. Presently, MR systems use HMDs for MR applications. These HMDs can be classified into two: HMD see-through glasses (e.g., HoloLens, Magic Leap, Samsung Odyssey+, and Vuzix smart glasses, which enable one to see surroundings clearly with additional digital holograms displayed in the glass that allows interaction with real surroundings) and immersive HMDs (e.g., Oculus Quest 2 and HTC Vive; although they have non-translucent displays which completely block out the real world, they use cameras for tracking the outer world). Similar to VR and AR, MR also has use cases in gaming, education and training, manufacturing, maintenance and repair, and AEC industries. Side note: Distinguishing AR and MR applications from each other can be quite challenging due to a common confusion about the fundamentals of AR and MR (as also observed in the literature). Therefore, the studies that use MR in the literature are presented together with AR in Section 4.2 (AR for Structural Condition Assessment) in this restudy. It is generally observed from the reviewed studies that the relevant works use XR to conduct field assessment remotely while simultaneously providing a collaborative work environment for engineers, inspectors, and other third parties. In addition, some other studies use XR to reduce human labor in the field and to support inspection activity by providing inspectors with digital visual aids and enabling the interaction of those visual aids with the data observed in the real world.earch. It is generally observed from the reviewed studies that the relevant works use XR to conduct field assessment remotely while simultaneously providing a collaborative work environment for engineers, inspectors, and other third parties. In addition, some other studies use XR to reduce human labor in the field and to support inspection activity by providing inspectors with digital visual aids and enabling the interaction of those visual aids with the data observed in the real world. Lastly, game engines and other programs are commonly used to develop VR, AR, and MR applications. Unity and Unreal engines are the most commonly used platforms for application development. Vuforia, Amazon Sumerian, and CRYENGINE are some of the others that are also utilized for developing VR, AR, and MR applications. The developed applications are then integrated into the standalone or PC-connected HMDs or other types of devices, mostly operating on Android, iOS, or Microsoft Windows, to be used by the end-user.