2. Related Work

The utilization of cutting-edge technologies from fields, like computer vision (CV) [12,13] and AR [14] can considerably enhance the capacity of an intelligent system. Such algorithms can provide useful feedback within an industrial environment, focusing on both human-centric technologies, like emotion [15] and hand pose estimation [16], and environment mapping ones [17]. Apart from AR, there are also other extended reality (xR) technologies, like virtual reality (VR) and mixed reality (MR), that are used in maintenance within the factory of the future. VR improves the efficiency of training activities and reduces training manufacturing equipment costs [18,19]. In the MR field, the real and virtual elements are mixed, where the real arguments can be visualized in the virtual environment, added value on the limitless of location, reducing costs for international processes [20]. Especially, holograms are part of mixed reality, in which remote teams are sharing 3D holographic data to break travel barriers and improve communication, by taking data-driven decisions [21].

In the last 10 years, the number of maintenance support systems have been increased in manufacturing research, with the goal to reduce the human errors in assembly tasks and eliminate the down times of the production. In consonance with the contemporary review study [14], AR manufacturing research is focused on four main topics, viz., assembly guidance, maintenance procedures, logistics navigation and picking instruction. The majority of the systems referred to assembly operation in the context of maintenance, while 25% of the reviewed papers are AR systems within indoor logistics [22] and picking problems [23]. Werrlich et al. [24] developed an AR system to support the engine assembly line in an automotive manufacturing environment. In the aviation industry, the impact of AR is evaluated, by applying techniques in inspection, robot programming, maintenance and process guidance [25,26]. In specific, Ceruti et al. proposed a framework [27] to identify the fault parts with AR and reserve engineering methodologies, which scans and prints the fault part, using additive manufacturing techniques. In addition, Freddi et al. [28] applied AR techniques in the disassembly of a tailstock, to improve the efficiency of the maintenance process. On the other hand, there is interest in the development of remote maintenance and repair support, where Mourtzis et al. [29] presented a framework about real-time communication channels between shop floor operators and maintenance experts, using AR guidance. Aiming at real-time interaction with other users, He et al. [30] presented an AR annotation tool that maps the environment and creates notes with limited actions. Arntz et al. [31] presented an indoor navigation AR approach to support shop floor operators in heavy industries, providing handy 3D visual instructions during evacuations. Meanwhile, Fang et al. [32] developed a scalable mobile AR application to track the pick-by-order process and provide instructions, using global market-based marks on the factory floor. Regarding the observation of robots, Limeria et al. [33] presented an application in ROS, where virtual reality (VR) technologies are exploited to simulate the robot picking procedures in real environments.

Improvements have also occurred in the collaborative picking procedures, where Sarupuri et al. [34] presented a prototype AR system to improve the successful picking rate of forklift operators, by providing real-time 3D guidelines about pallet racking. Moreover, Kapinus et al. [35] designed an AR architecture, that supports the shop floor employees in the programming of industrial robotic tasks, visualizing instructions into a 3D mobile environment that eliminate the times between the screens and the work environment.

Safety retention during performance enhancements of the maintenance procedures constitutes a critical factor in the development of AR applications [36]. Kim et al. [23] examined the effectiveness of operators among AR devices, while performing tasks like order picking, maintenance and assembly. Furthermore, Sanna et al. [37] compared the required times and number of errors that occurred during assembly, maintenance and picking tasks, by using paper-based instructions and a handheld AR tool. During the maintenance procedures on food processing machines, Vignali et al. [38] proposed an AR framework to ensure the safety of the employees.

The majority of the aforementioned AR systems rely on the support of shop floor operators in different manufacturing areas, using high-cost and advanced equipment. Their value is significant but they do not offer a well-defined process, describing the creation of the 3D models, the input data acquisition, as well as the asset’s projection to the application’s interface that provides the feeling of naturalism to the users. Considering the state-of-the-art, we present an AR application that can be effectively controlled by shop-floor operators to perform fast-track repair procedures within the manufacturing plant. The proposed framework is designed to be part of a ubiquitous and interactive maintenance system that can eliminate the mean time to repair (MTR) in production’s availability, by reducing the unexpected breakdowns times, where external OEM experts may be required. For that reason, the application can be installed in operators’ mobile handhelds and support bring your own device (BYOD) policies, forming a significant part of the Industry 4.0 philosophy. As a result, MARMA is a low-cost application that replaces the paper-based maintenance instructions with AR-based ones, reducing the knowledge gap between OEMs and maintenance in-house operators.

3. Methodology

MARMA is able to estimate the position of the asset in a plant, display the 3D CAD model of the machine with the start of the maintenance procedure and let the user navigate throughout the proposed maintenance steps through the available previous and next buttons to his/her handheld. Firstly, a set of features is extracted for the machine of interest from various viewpoints and distances, which is stored to the system’s database as a 3D target model. Then, during the inference phase, MARMA receives a frame from the handheld’s camera, extracts a set of features and compares them with the features of the 3D target model stored in its database. In the case that a sufficient matching score is succeeded, the machine is successfully detected and the extracted frame’s features are assigned to the system’s tracking algorithm. The subsequent captured frames are processed only by the tracking algorithm, measuring a matching score between the frame’s features and the tracked ones. In case the tracking algorithm fails, the frame’s features are discarded, the algorithm returns a false state and the system processes the following frame again from the feature extraction and matching step. For every successful detection or tracking step, our method computes, according to the corresponding frame’s features, the position, the orientation and the distance of the machine, in order to project the corresponding CAD model from an XML file on the image plane of the device’s screen. This procedure is repeated, until our method receives the user’s choice, to change the maintenance process step. In such a case, MARMA projects the next or previous model of the XML file. The maintenance procedure is terminated when the user exits the application. An outline of our proposed method is provided in the flowchart depicted in Figure 2. Below, a detailed description regarding the individual components of MARMA is presented.

Figure 2. Flowchart of the proposed augmented reality method for fast track maintenance procedures.

4. Experimental Process

The performance of the proposed methodology is tested on a realistic use case scenario through a compressor, which is provided by a collaborating manufacturer. The current maintenance procedure requires an operator that can read the step-by-step instructions from a paper-book and perform the whole procedure. The collaborating manufacturer recognizes the capabilities of MARMA approach and decides to work on a compressor. Finally, a real-time demonstration of the maintenance process with MARMA application is presented to the operators.

4.1. Maintenance Scenario Setup

In our case study, the investigated asset is an A/C compressor, which is commonly included in the air-conditioning system of a car, as shown in Figure 4. It generates the power responsible for channeling the freon into the condenser that transforms air to liquid. During the maintenance process of the system, mechanics have to clear the region of the valve plate. The scenario setup contains the creation of the compressor’s target model through the Vuforia Object Scanner described in Section 4.2, scanning the compressor in various lighting conditions. The next step refers to the design of the 3D CAD models required for the development of the specific maintenance scenario. To support the aforementioned process with AR techniques, the compressor parts should be designed in 3D CAD models. As mentioned in Section 4.1, the Autodesk Inventor software is exploited since it provides the suitable tools for designing mechanical parts, as well as the capability of exporting them to an .obj format. Due to the low level of details, the 3D models of the compressor lasted around 35 h because of the uncomplicated structure. Both Figure 5 and Figure 6 demonstrate 3D exploded views of the compressor that illustrate its designed assembly parts.

Figure 4. The investigated compressor and its location within the topology of an air-conditioning system of a car.

Figure 5. Traditional paper-based technical drawing, presenting the correlations among the assembly parts of the compressor.

Figure 6. Exploded view diagram generated by MARS. The illustration contains the designed subparts of the compressor in the explosion option.

After the selection of the compressor, the exact maintenance procedure, which is going to be visualized via the proposed application, is defined by the manufacturer. To that end, a set of actions are specified, which shall be displayed to the operator’s mobile device. The positioning of the object in the mobile required around 25 h. In particular, the maintenance is described by the following 5 phases:

 

• Phase 1: Remove the bolts
• Phase 2: Remove the frontal phase
• Phase 3: Unscrew the sheet metal
• Phase 4: Replace the gasket
• Phase 5: Clean the metal flange

 

The traditional maintenance procedure is visualized through paper-based technical drawings, as shown in Figure 5. In addition, the compressor’s manual includes step-by-step instructions above the technical drawing. Hence, during the maintenance, significant time is spent on understanding the paper-based instructions according to the operator’s level of expertise [49]. Our system uses AR to close the time gap between highly experienced maintenance managers and not very experienced operators.

4.2. Demonstration

The developed application was executed in an Android smartphone with a 1.8GHz 4-Core CPU and 4GB RAM. For each phase, the application chooses a short description and the corresponding 3D object to guide the user during the disassembly. At the beginning, the user chooses to start the maintenance, while the application loads the XML file, including several information regarding the subsequent phases. After that, the application tries to locate the compressor’s features to the handeld’s frame, in order to visualize the first phase. To complete phase 1, the user has to unscrew and remove the five bolts from the frontal phase (Figure 7a). The next phase includes the removal of the frontal phase, in the corrected position (Figure 7b). After the disassembly of the external surface, the user is guided to unscrew the sheet metal (Figure 7c). Finally, the application visualizes the removal of the gasket and provides a warning message to clean the metal flange (Figure 7d). At the end of the replacement, the user can navigate throughout the assembly tasks by using the back button. At any time, the user can choose the explosion option to dynamically inspect the correlation and the assemblage of the different parts (Figure 6).

Figure 7. Demonstration of the maintenance steps of an A/C compressor using MARMA in a mobile device.

During the final demonstration of MARMA within Industry 4.0 (now crowdhackathon), more than 20 manufacturer experts assessed the maturity level and the usage of the application in the industry. MARMA generally received positive scores and confirmed its contribution as an application that can effectively support the maintenance engineers, since the procedure can be achieved without the need for high-skilled operators. The demonstration of MARMA application to the interested manufacturers showed their willingness and the potential of the system in simplifying complex maintenance procedures. Experts mentioned that MARMA can reduce the total repair time of compressor by 30%, compared with the paper-based procedure and digital ones. In addition, particular interest was paid in exploiting the application as a means of training new and unskilled maintenance operators, reducing both the required training, as well as the total repair time. Eventually, the integration of exponential digital technologies, like MARMA, in small–medium enterprises can strengthen the industrial competitiveness in the global landscape.

The framework of MARMA can be applied not only in manufacturing environments, but also in completely different sectors, like investigation [50], infrastructure [51], and education [52]. Generally, MARMA enhances users’ field of view with real-time AR-based digital information, reducing the required time for understanding the procedures.

In the case that we want to apply the MARMA in a production line, which is the most significant asset of a factory, it is crucial to perform a cost–benefit analysis [53]. The cost benefit analysis of an AR maintenance application correlates the break down frequency, the cost of spare parts, the repair fault time and external collaborators costs per machine. Based on the findings, the first machines to implement the framework of MARMA will be chosen, ensuring valuable recoverable. Then, a similar procedure with the one presented in Section 4.1 has to be followed. The validity of the application depends on the probability of unexpected maintenance steps. This means that if the maintenance process is standardized by the OEM, then MARMA will work in the right way. Otherwise, the success of the process is based on the experience of the operator because he/she will need to perform additional unplanned steps. In the implementation phase, the complexity of the maintenance procedure is not a time-consuming task, compared with the availability of the 3D digital models. In this case, the OEM provides the 3D models or the manufacturer owns the digital files, then the implementation time is significant lesser than designing them from scratch.

MARMA reduces the knowledge gap between the original equipment manufacturers and maintenance operators, describing in detail the pipeline development and usage. Considering the current applications, the system is designed in a user-friendly way to be effectively controlled in fast-track repair procedures by shop-floor operators and capable to be correlated with ubiquitous maintenance system that can reduce the mean time to repair (MTR) in production’s availability. To the best of our knowledge, this is the first attempt to fully and concretely present an end-to-end AR-assisted maintenance system, by describing all of its individual parts, as well as the way that those cooperate and contribute to the final system.