Structural health monitoring (SHM) deals with the systematic measurement of structures and their characteristics, such as factory chimneys, bridges, wind turbines or dams. These are periodically examined in order to gain possible conclusions about the structural health of the object ^[1][2][1,2].

Due to environmental impacts and permanent strain, civil structures in particular are affected by damaging influences such as cracking in the concrete, spalling, corrosion of the steel and even failure of the structure.

SHM is an efficient and reliable method to monitor the condition of structures and is necessary to detect damage to structures. The most common method is visual inspection, carried out by experienced inspectors. Often this is very time-consuming and leads to traffic obstructions or temporary closure of the structure.

Another approach is the concept of the “smart bridge”. Already during the construction phase, bridges are equipped with adaptive sensors to enable a complete evaluation of the bridge. First existing bridges are comprehensively evaluated and then expanded appropriately with sensor technology ^[3][6]. The instruments commonly used include accelerometers ^[1][4][1,7], extensometers ^[5][6][8,9], terrestrial laser scanners (TLS) ^[7][8][10,11] or tachymeters ^[9][10][12,13]. Of particular interest are measurements to determine the vibration and displacement behaviour of the structures. It is possible to analyse the monitored vibrations and calculate the natural frequencies. Changes in this natural frequency indicate possible structural damages and require further investigation by qualified experts. It is also possible to compare the parameters determined by the measured values with the calculation from a finite element model (FEM) ^[11][12][14,15]. By this way, measured values can be compared with calculated values and analysed. Another complementary measuring system to the SHM is the use of camera sensors, e.g., in form of so-called image assisted total station (IATS). IATS are characterised by the extension of a total station by using one or more cameras. The accuracy of the angle measurements of the total station is combined with photogrammetry and the associated simple target mark definition. By using telescope lenses, the distance to the structure is increased; depending on the situation, the danger zone does not have to be entered. Signalling on the object can be omitted due to the photogrammetric evaluation. However, this depends on the specific monitoring object. If there is not enough texture or contrast on the structure, signalisation may be necessary. Numerous examples of applications with IATS and signalling can be found in the literature ^{[11][13][14][15][16]}[14,16,17,18,19]. Deformations are evaluated by capturing image information from a camera and natural frequencies can be calculated. Without continuous distance measurements, a two-dimensional movement parallel to the camera sensor can be measured. The measurement field is limited by the field of view (FoV). For the measurement of frequencies, the maximum frames per second (fps) are essential. Only by an adequately frequent sampling rate the natural frequency can be determined from the measured values.

2. Image Assisted Total Station (IATS)

Since the beginning of 2000, the development of IATS has continued. For many manufacturers, the integration of one or more cameras into a total station has become standard. The integration of cameras enables to record the relevant scene and offers additional potential for the evaluation by the image-based recording. The combination of angular measurement accuracy from tacheometry in conjunction with image processing of captured image data has expanded the applicability also in the field of SHM. At IATS’s, a camera is installed in the line of sight and the visual filed is captured by the image. Limitations are the FoV of the camera used and the need for light (at least for the observed object). The automatic acquisition of images removes the need for human aiming and thus the potential impact of error. A major advantage is that there is no need for access to the structures, as prominent points on the structure can be observed directly without the use of reflectors. Measurement points can be flexibly selected over the entire structure, if a clear view of it is accessible. The focus here is on the mobile use of IATS. Due to the advantage of non-contact measurement, this measurement method can also be used spontaneously. Using image-processing techniques such as template matching or feature matching, distinctive points can be extracted and their movement detected ^[17][18][22,23]. To achieve the desired accuracy, a calibration of the different sensors is necessary. For fixed commercial systems, the calibration parameters are provided by the manufacturer. Some of these calibrations are carried out under the aspect of using the display, e.g., to rotate the instrument, and does not represent a complete photogrammetric calibration. For modular systems or external cameras, the parameters have to be determined in advance. By using a calibration, the captured images are directly georeferenced and measurements in the images become possible.

2.1. Commercially Available Systems

The technical development up to today’s state of the art reaches into the last century. Based on the technical development of tacheometers, these were successively expanded by further software, sensors and technology up to today’s IATS. According to ^[11][14], total stations can be divided into 4 different types (total station, robotic total station, image-assisted total station and image-assisted scanning total station) based on the integrated sensors and functions. Accordingly, IATS includes data registration, electrical distance measurement (EDM), reflectorless EDM, motorization, image sensor, automated aiming, tracking and imaging. By adding a scanning function, the term image-assisted scanning total station (IASTS) becomes applicable. Today’s instruments usually have the possibility to store additional information of the measured points by the captured images and videos. Images are digitally linked to surveyed points and stored externally for documentation purposes via memory card or interface. The collected georeferenced image data are available for photogrammetrical evaluation. The development of autofocusing and motorisation of the instruments allows for a point of interest to be selected in the image (with the help of the display) and the instrument aligns automatically. This way of data acquisition optimises the previous measurement process. In the event that the instrument is equipped with a scan function, the scan area can be defined in the display using the image-based selection option.

2.2. Research Prototypes

In the development of prototypes, different types of construction have emerged since the year 2000. On the one hand, there are external implementations making it possible to mount the camera on the ocular or to replace it. These are used in combination with commercial total stations or tacheometers, and can be converted and adapted to the monitoring conditions. The technical development of the automated focus in the total station made the enormously simplified combination of external cameras possible. Examples of such a modular system would be DAEDALUS ^{[13][14][19][20]}[16,17,30,31] of ETH Zurich, MoDiTa ^[21][22][23][32,33,34] of i3mainz or the applications of the University of Zagreb ^[11][24][14,35] using a GoPro5. In 2010, ^[13][16] presented DAEDALUS, a measurement system originally for automatic online astro-geodatic observations, and also for the use in SHM. The setup includes replacing the eyepiece with a CCD chip. Since no further optical components are installed, the image is no longer displayed exactly in the plane of the crosshair instead the image is now displayed in the plane of the CCD chip. This can be compensated for distances up to 13 m by changing the focus position. For longer distances, an additional lens is required on the telescope. The camera used is a monochrome Guppy F-080C from Allied Vision Technologies (AVT). The CCD sensor has a resolution of 1024 × 768 pixels with a pixel size of 4.65 µm and a frame rate of 30 Hz (full frame) up to 60 Hz (reduced field of view) ^[20][31]. At that point in time, no automated focusing was available for tacheometers. Therefore, for the necessary focusing, a special mechanism was developed enabling to autofocus by means of a small stepper motor without modification of the total station. When using newer generations of total stations or multistations with coaxial cameras already installed, the focus mechanism is already available and the special mechanics for focusing is no longer needed. The compatibility is given for models of the TCA, TPS, TS and MS series from the manufacturer Leica ^[14][17]. For applications with required software pulses, control by means of a GNSS receiver can be implemented for precise control of exposure start and exposure time. The optical system has a resolution of approximately four arcseconds/pixel (1.1 mgon/pixel). The prototype MoDiTa developed by i3mainz (University of Applied Sciences Mainz) is based on modularity. Via bayonet ring the camera replaces the standard eyepiece of the tacheometer. The camera used is mounted on the eyepiece using an adapter. By this way the crosshair plane is also captured. There are no changes at the optical beam path in the telescope. This allows a quick exchange of the camera or a quick change to the classical application of the total station. Based on the modular design both, camera and total station, are replaceable. Attached to the telescope is a counterweight to compensate for the weight of the camera. Due to the modular design of the external cameras, active crosshair tracking is required during the measurement. Because of longer pauses between measurements or after changing the telescope position, the crosshair position may change. Another prototype was presented in 2017 by combining a Leica TPS1201 and a GoPro5 Hero camera ^[24][35]. This camera offers different video recording modes. It is possible to choose between different FoV (narrow, linear, medium, wide and superview), resolutions (720 × 400, 1280 × 720, 1920 × 1080, 2560 × 1440, 2704 × 1520, 3840 × 2160 pixel) and recording speeds (30–240 Hz). The field of view and the recording mode are directly related. An adapter is required for the camera mounting on the TPS1201. This was manufactured in a 3D printing process and offers the possibility to attach the camera to the eyepiece of the telescope.

3. Structural Health Monitoring with IATS

In the field of SHM using IATS, mainly deformations and vibrations of the structure are measured. However, other applications are also being investigated. A basic distinction can be made between static and dynamic monitoring ^[14][25][26][17,49,56]. Static deformation measurements are usually carried out in the form of set measurements. For example, dynamic measurements are carried out in the SHM on bridges or wind turbines in order to map the frequency behaviour of the construction objects. Changes in the frequency response can be used to determine damage to the structure ^[27][57]. For dynamic monitoring, the maximum image frequency is important. According to Nyquist, the recording frequency must be at least twice the frequency of the oscillation. In practice, it turns out that a multiple of the frequency to be determined makes sense. Since bridges usually have a natural frequency range of 0.1 to 25 Hz, this has to be taken into account, especially for commercial IATS with much more limited recording frequencies than the prototypes ^[25][28][49,58]. In addition, to be considered is that the vibration amplitude must be higher than the measurement resolution. The amplitude would otherwise not be identifiable in the measurement noise ^[25][49]. By observing a significant point or section on the structure, movements are measured transverse to the direction of view. For a 3D measurement, the use of the EDM or several IATS is necessary. Thus, a forward section is used to calculate the change. Distance measurements are time-consuming compared to image-based and angle measurements. It is not always possible to carry out angle and distance measurements simultaneously with image acquisition. The application of commercial IATS for bridge monitoring is shown in ^{[16][29][30][25][31][32]}[19,47,48,49,59,60] using the example of the Augarten footbridge (steel construction, 74 m span width) in Graz, Austria. For the measurements, an MS50 ^[33][61] from the manufacturer Leica was used with different sensors. The IATS has a maximum frame rate of 10 fps. For video recording, a frame rate of 20 fps is possible, but this cannot be transmitted at this speed via the interface. The bridge was stimulated by pedestrians. Both natural targets and prisms were observed. To compare the results, parallel measurements were taken with an acceleration sensor. The study also investigated to what degree the capturing frequency can be increased, e.g., by omitting the distance measurement. In this case, only displacements orthogonal to the viewing direction can be detected. The image-based measurements partly achieve a higher angular resolution than the measurements on a prism. Overall, the results show the successful use of a commercial IATS for dynamic SHM. Another use of commercial IATS for monitoring bridges is shown in ^[28][58]. The stimulation of the 27 m pedestrian bridge was uncontrolled by pedestrians. The measured results could be validated by calculated results using an FEM. For the natural frequencies, only the 3.642 Hz could be verified due to the low sampling rate of the Leica Nova MS50 ^[33][61]. The further natural frequency of 13.294 Hz is not detectable due to the limited sampling frequency. ^[34][62] show in a proof-of-concept study the use of a MEMS accelerometer and an IATS. Here, the estimated model parameters from the IATS are supposed to counteract the degradation of accuracy over time for the accelerometer due to coordinated updates. This method shows significantly improved results as well as sub-millimetre accuracy for the displacement and better than 0.1 Hz for the frequencies. The combination of laser scan data with image data (both from a so-called IASTS Leica Nova MS60 ^[35][63]) for use in SHM show ^[36][37][45,64]. The scan centre and the projection centre of the camera (almost) correspond in this instrument. This means that the same perspective is captured. The acquired data have the advantage of being in the same coordinate system. For evaluation, the images are combined to form a spherical panoramic image (RGB). These are then supplemented with a channel D (depth), i.e., the interpolated distance information from the scans. By means of a congruence model, a deformation analysis becomes practicable. The successful evaluation of the different epochs is shown by the displacement vectors using the SIFT operator. For the application of façade monitoring, ^[38][65] use a IATS-prototype developed by Leica in combination with a knowledge-based systems (KBS) and later also with a terrestrial laser scanner (TLS) ^[39][40][66,67]. One of the main goals of this work was to extend the deformation analysis of single points and to use a point-set based method instead. With the help of the KBS, an automated image-based online measurement system for the rapid feature extrusion of prominent façade points is presented. Here, image acquisition, image pre-processing and point detection are automated by the KBS using image feature extraction for deformation analysis. The results show a significant reduction of the necessary decisions by the user and provide a fast performance of the feature extraction. The use of the prototype DAEDALUS for SHM with optical target recognition (OTR) is shown in ^[13][16]. A torch was mounted on the middle of a steel bridge and monitored at a distance of 40 metres. The stimulation was provided by the passage of a truck. The images were released during the day with a measuring rate of 15 Hz. Using a fast Fourier transform (FFT), the dominant natural frequency of 2.59 Hz was successfully determined. In another study ^[14][17] show the use of several synchronised QDaedalus for the determination of 3D positions at high frequencies (60 Hz). For this purpose, an innovative hybrid structure made of hardwood by the Institute of Structural Engineering of ETH Zurich was observed. For comparison, acceleration sensors with a recording rate of 1 kHz were attached to the object. The results show that the prototype has a better sensitivity than the acceleration sensor used in the measurements of displacements and accelerations up to a frequency of 3 Hz ^[14][17]. SHM measurements using IATS2 (manufactured by Leica Geosystems as a case study series) at the Fatih Sultan Mehmet Bridge in Istanbul show ^[15][41][42][18,29,44]. For this purpose, an LED target with a distance of around 128 m was observed on the object. The bridge was stimulated by normal traffic crossing the bridge. By reducing the field of view, the capturing rate was set to 25 Hz. The results were compared with calculations from a FEM and the determined frequencies can be confirmed. At three frequencies, deviations from previous investigations were found, which could indicate possible structural damage. In conducting field tests on the Kloštar railway bridge in Croatia, ^[11][14] demonstrated the use of the prototype of the Department of Applied Geodesy, Faculty of Geodesy, University of Zagreb. Since no significant natural targets can be observed on the bridge, predefined photo marks were attached to the bridge. At a distance of about 28.5 m, vertically to the longitudinal axis of the bridge, the target was observed using 60 fps. Train movements at different speeds stimulated the bridge. The comparison of the calculated natural frequencies from an FEM with the frequencies measured by means of IATS shows the successful application. Measurements of the prototype MoDiTa, also carried out on a railway bridge, show promising results ^[43][68]. Natural targets on an arched bar bridge in Lahnstein (Germany) were observed at a distance of 30 to 70 m, stimulated by passenger and cargo traffic. The capturing speed was 500 fps. At the same time, measurements were taken with an accelerometer. Since a vertically oriented view of the structure was not feasible, the measured oscillation was converted to a balanced plane as an approximation of the structure plane. The software offers this solution automatically. The calculated natural frequency could be confirmed by both measurement techniques.