The integration of exponential technologies in the traditional manufacturing processes constitutes a noteworthy trend of the past two decades, aiming to reshape the industrial environment. This kind of digital transformation, which is driven by the Industry 4.0 initiative, not only affects the individual manufacturing assets, but the involved human workforce, as well. Since human operators should be placed in the centre of this revolution, they ought to be endowed with onew tools and through-engineering solutions that improve their efficiency. In addition, vivid visualization techniques must be utilized, in order to support them during their daily operations in an auxiliary and comprehensive way. Towards this end, we describe a user-centered methodology, which utilizes augmented reality (AR) and computer vision (CV) techniques, supporting low-skilled operators in the maintenance procedures. The described mobile augmented reality maintenance assistant (MARMA) makes use of the handheld’s camera and locates the asset on the shop floor and generates AR maintenance instructions.
The fourth industrial revolution has grown over the last two decades, making rapid changes day by day and removing silos within organization. The business models and manufacturing processes are transformed using digital technologies and through engineering solutions [1,2]. The factory of the future, enabled by the German initiative of Industry 4.0 (I4.0), places the human component at the center of the value chain [3]. Operators are critical elements of the smart factory, since their intelligence can not be replaced by machines. To that end, an Operator 4.0 should be flexible and adaptive, in order not only to perform tasks in collaboration with machines, but also to solve complex unplanned problems. By further exploiting the rapid configuration of smart production systems, producing huge amount of information, workforce should use innovative technology systems, providing better visualization regarding the information of the task.
The fourth industrial revolution has grown over the last two decades, making rapid changes day by day and removing silos within organization. The business models and manufacturing processes are transformed using digital technologies and through engineering solutions [1][2]. The factory of the future, enabled by the German initiative of Industry 4.0 (I4.0), places the human component at the center of the value chain [3]. Operators are critical elements of the smart factory, since their intelligence can not be replaced by machines. To that end, an Operator 4.0 should be flexible and adaptive, in order not only to perform tasks in collaboration with machines, but also to solve complex unplanned problems. By further exploiting the rapid configuration of smart production systems, producing huge amount of information, workforce should use innovative technology systems, providing better visualization regarding the information of the task.
As the cyber-physical production systems materialize, human intervention in the production process is anticipated to significantly drop off. Nevertheless, the maintenance of the assets will still be performed by the operators. Hence, in the era of I4.0, maintenance operators should be supported by connected tools and intelligent systems to efficiently complete their operations, without the necessity of knowing the accurate location of machines, as well as specific information regarding the configurable topology. Traditionally, the maintenance operators are highly experienced and they follow paper-based technical drawings to perform a maintenance activity. In the advanced cases, in which the maintenance requires knowledge and workforce from the original equipment manufacturers (OEMs), the production is usually frozen by the time an expert visits the plant [4].
One of the technological pillars of I4.0, which can solve the aforementioned traditional procedure, is augmented reality (AR) [5]. AR technologies are rapidly introduced in a wide variety of applications within the factory of the future. About 25% of their recent implementations in I4.0 covers tasks regarding remote assistance and maintenance, with the other 75% focusing either on visualization and simulation applications or on robot programming and human–robot interaction tasks [6]. Maintenance constitutes one of the most important activities of the production life-cycle, covering 60–70% of its total costs. Hence, the improvement of its efficiency and performance is among the highest priorities, leading to several implementations of AR in maintenance applications [7].
As the smart manufacturing environments become reality, handheld solutions are used to connect the operators and managers with the digital factory. Some of the applications include work order management, asset tracking, inventory management and digitized maintenance procedures. In the maintenance departments, operators, on one hand, use their mobiles to easily switch tasks and rotate jobs over a day, communicate with experts and read step-by-step digitized procedures using AR architectures [8,9], as illustrated in
As the smart manufacturing environments become reality, handheld solutions are used to connect the operators and managers with the digital factory. Some of the applications include work order management, asset tracking, inventory management and digitized maintenance procedures. In the maintenance departments, operators, on one hand, use their mobiles to easily switch tasks and rotate jobs over a day, communicate with experts and read step-by-step digitized procedures using AR architectures [8][9], as illustrated in
. On the other hand, managers can control multiple work areas and teams via one central environment.
Figure 1. In the Industry 4.0, maintenance operators are supported by augmented reality (AR)-enabled handhelds.
The utilization of cutting-edge technologies from fields, like computer vision (CV) [12,13] and AR [14] can considerably enhance the capacity of an intelligent system. Such algorithms can provide useful feedback within an industrial environment, focusing on both human-centric technologies, like emotion [15] and hand pose estimation [16], and environment mapping ones [17]. Apart from AR, there are also other extended reality (xR) technologies, like virtual reality (VR) and mixed reality (MR), that are used in maintenance within the factory of the future. VR improves the efficiency of training activities and reduces training manufacturing equipment costs [18,19]. In the MR field, the real and virtual elements are mixed, where the real arguments can be visualized in the virtual environment, added value on the limitless of location, reducing costs for international processes [20]. Especially, holograms are part of mixed reality, in which remote teams are sharing 3D holographic data to break travel barriers and improve communication, by taking data-driven decisions [21].
The utilization of cutting-edge technologies from fields, like computer vision (CV) [10][11] and AR [12] can considerably enhance the capacity of an intelligent system. Such algorithms can provide useful feedback within an industrial environment, focusing on both human-centric technologies, like emotion [13] and hand pose estimation [14], and environment mapping ones [15]. Apart from AR, there are also other extended reality (xR) technologies, like virtual reality (VR) and mixed reality (MR), that are used in maintenance within the factory of the future. VR improves the efficiency of training activities and reduces training manufacturing equipment costs [16][17]. In the MR field, the real and virtual elements are mixed, where the real arguments can be visualized in the virtual environment, added value on the limitless of location, reducing costs for international processes [18]. Especially, holograms are part of mixed reality, in which remote teams are sharing 3D holographic data to break travel barriers and improve communication, by taking data-driven decisions [19].
In the last 10 years, the number of maintenance support systems have been increased in manufacturing research, with the goal to reduce the human errors in assembly tasks and eliminate the down times of the production. In consonance with the contemporary review study [14], AR manufacturing research is focused on four main topics,
In the last 10 years, the number of maintenance support systems have been increased in manufacturing research, with the goal to reduce the human errors in assembly tasks and eliminate the down times of the production. In consonance with the contemporary review study [12], AR manufacturing research is focused on four main topics,
viz., assembly guidance, maintenance procedures, logistics navigation and picking instruction. The majority of the systems referred to assembly operation in the context of maintenance, while 25% of the reviewed papers are AR systems within indoor logistics [22] and picking problems [23]. Werrlich et al. [24] developed an AR system to support the engine assembly line in an automotive manufacturing environment. In the aviation industry, the impact of AR is evaluated, by applying techniques in inspection, robot programming, maintenance and process guidance [25,26]. In specific, Ceruti et al. proposed a framework [27] to identify the fault parts with AR and reserve engineering methodologies, which scans and prints the fault part, using additive manufacturing techniques. In addition, Freddi et al. [28] applied AR techniques in the disassembly of a tailstock, to improve the efficiency of the maintenance process. On the other hand, there is interest in the development of remote maintenance and repair support, where Mourtzis et al. [29] presented a framework about real-time communication channels between shop floor operators and maintenance experts, using AR guidance. Aiming at real-time interaction with other users, He et al. [30] presented an AR annotation tool that maps the environment and creates notes with limited actions. Arntz et al. [31] presented an indoor navigation AR approach to support shop floor operators in heavy industries, providing handy 3D visual instructions during evacuations. Meanwhile, Fang et al. [32] developed a scalable mobile AR application to track the pick-by-order process and provide instructions, using global market-based marks on the factory floor. Regarding the observation of robots, Limeria et al. [33] presented an application in ROS, where virtual reality (VR) technologies are exploited to simulate the robot picking procedures in real environments.
, assembly guidance, maintenance procedures, logistics navigation and picking instruction. The majority of the systems referred to assembly operation in the context of maintenance, while 25% of the reviewed papers are AR systems within indoor logistics [20] and picking problems [21]. Werrlich et al. [22] developed an AR system to support the engine assembly line in an automotive manufacturing environment. In the aviation industry, the impact of AR is evaluated, by applying techniques in inspection, robot programming, maintenance and process guidance [23][24]. In specific, Ceruti et al. proposed a framework [25] to identify the fault parts with AR and reserve engineering methodologies, which scans and prints the fault part, using additive manufacturing techniques. In addition, Freddi et al. [26] applied AR techniques in the disassembly of a tailstock, to improve the efficiency of the maintenance process. On the other hand, there is interest in the development of remote maintenance and repair support, where Mourtzis et al. [27] presented a framework about real-time communication channels between shop floor operators and maintenance experts, using AR guidance. Aiming at real-time interaction with other users, He et al. [28] presented an AR annotation tool that maps the environment and creates notes with limited actions. Arntz et al. [29] presented an indoor navigation AR approach to support shop floor operators in heavy industries, providing handy 3D visual instructions during evacuations. Meanwhile, Fang et al. [30] developed a scalable mobile AR application to track the pick-by-order process and provide instructions, using global market-based marks on the factory floor. Regarding the observation of robots, Limeria et al. [31] presented an application in ROS, where virtual reality (VR) technologies are exploited to simulate the robot picking procedures in real environments.
Improvements have also occurred in the collaborative picking procedures, where Sarupuri et al. [34] presented a prototype AR system to improve the successful picking rate of forklift operators, by providing real-time 3D guidelines about pallet racking. Moreover, Kapinus et al. [35] designed an AR architecture, that supports the shop floor employees in the programming of industrial robotic tasks, visualizing instructions into a 3D mobile environment that eliminate the times between the screens and the work environment.
Improvements have also occurred in the collaborative picking procedures, where Sarupuri et al. [32] presented a prototype AR system to improve the successful picking rate of forklift operators, by providing real-time 3D guidelines about pallet racking. Moreover, Kapinus et al. [33] designed an AR architecture, that supports the shop floor employees in the programming of industrial robotic tasks, visualizing instructions into a 3D mobile environment that eliminate the times between the screens and the work environment.
Safety retention during performance enhancements of the maintenance procedures constitutes a critical factor in the development of AR applications [36]. Kim et al. [23] examined the effectiveness of operators among AR devices, while performing tasks like order picking, maintenance and assembly. Furthermore, Sanna et al. [37] compared the required times and number of errors that occurred during assembly, maintenance and picking tasks, by using paper-based instructions and a handheld AR tool. During the maintenance procedures on food processing machines, Vignali et al. [38] proposed an AR framework to ensure the safety of the employees.
Safety retention during performance enhancements of the maintenance procedures constitutes a critical factor in the development of AR applications [34]. Kim et al. [21] examined the effectiveness of operators among AR devices, while performing tasks like order picking, maintenance and assembly. Furthermore, Sanna et al. [35] compared the required times and number of errors that occurred during assembly, maintenance and picking tasks, by using paper-based instructions and a handheld AR tool. During the maintenance procedures on food processing machines, Vignali et al. [36] proposed an AR framework to ensure the safety of the employees.
The majority of the aforementioned AR systems rely on the support of shop floor operators in different manufacturing areas, using high-cost and advanced equipment. Their value is significant but they do not offer a well-defined process, describing the creation of the 3D models, the input data acquisition, as well as the asset’s projection to the application’s interface that provides the feeling of naturalism to the users. Considering the state-of-the-art, we present an AR application that can be effectively controlled by shop-floor operators to perform fast-track repair procedures within the manufacturing plant. The proposed framework is designed to be part of a ubiquitous and interactive maintenance system that can eliminate the mean time to repair (MTR) in production’s availability, by reducing the unexpected breakdowns times, where external OEM experts may be required. For that reason, the application can be installed in operators’ mobile handhelds and support bring your own device (BYOD) policies, forming a significant part of the Industry 4.0 philosophy. As a result, MARMA is a low-cost application that replaces the paper-based maintenance instructions with AR-based ones, reducing the knowledge gap between OEMs and maintenance in-house operators.
MARMA is able to estimate the position of the asset in a plant, display the 3D CAD model of the machine with the start of the maintenance procedure and let the user navigate throughout the proposed maintenance steps through the available previous and next buttons to his/her handheld. Firstly, a set of features is extracted for the machine of interest from various viewpoints and distances, which is stored to the system’s database as a 3D target model. Then, during the inference phase, MARMA receives a frame from the handheld’s camera, extracts a set of features and compares them with the features of the 3D target model stored in its database. In the case that a sufficient matching score is succeeded, the machine is successfully detected and the extracted frame’s features are assigned to the system’s tracking algorithm. The subsequent captured frames are processed only by the tracking algorithm, measuring a matching score between the frame’s features and the tracked ones. In case the tracking algorithm fails, the frame’s features are discarded, the algorithm returns a false state and the system processes the following frame again from the feature extraction and matching step. For every successful detection or tracking step, our method computes, according to the corresponding frame’s features, the position, the orientation and the distance of the machine, in order to project the corresponding CAD model from an XML file on the image plane of the device’s screen. This procedure is repeated, until our method receives the user’s choice, to change the maintenance process step. In such a case, MARMA projects the next or previous model of the XML file. The maintenance procedure is terminated when the user exits the application. An outline of our proposed method is provided in the flowchart depicted in
. Below, a detailed description regarding the individual components of MARMA is presented.
Figure 2.
Flowchart of the proposed augmented reality method for fast track maintenance procedures.
The performance of the proposed methodology is tested on a realistic use case scenario through a compressor, which is provided by a collaborating manufacturer. The current maintenance procedure requires an operator that can read the step-by-step instructions from a paper-book and perform the whole procedure. The collaborating manufacturer recognizes the capabilities of MARMA approach and decides to work on a compressor. Finally, a real-time demonstration of the maintenance process with MARMA application is presented to the operators.
In our case study, the investigated asset is an A/C compressor, which is commonly included in the air-conditioning system of a car, as shown in
Figure 4. It generates the power responsible for channeling the freon into the condenser that transforms air to liquid. During the maintenance process of the system, mechanics have to clear the region of the valve plate. The scenario setup contains the creation of the compressor’s target model through the Vuforia Object Scanner described in
3. It generates the power responsible for channeling the freon into the condenser that transforms air to liquid. During the maintenance process of the system, mechanics have to clear the region of the valve plate. The scenario setup contains the creation of the compressor’s target model through the Vuforia Object Scanner described in
, scanning the compressor in various lighting conditions. The next step refers to the design of the 3D CAD models required for the development of the specific maintenance scenario. To support the aforementioned process with AR techniques, the compressor parts should be designed in 3D CAD models. As mentioned in
, the Autodesk Inventor software is exploited since it provides the suitable tools for designing mechanical parts, as well as the capability of exporting them to an
.obj
format. Due to the low level of details, the 3D models of the compressor lasted around 35 h because of the uncomplicated structure. Both
Figure 5 and
4 and
Figure 6 demonstrate 3D exploded views of the compressor that illustrate its designed assembly parts.
5 demonstrate 3D exploded views of the compressor that illustrate its designed assembly parts.
Figure 43. The investigated compressor and its location within the topology of an air-conditioning system of a car.
Figure 54. Traditional paper-based technical drawing, presenting the correlations among the assembly parts of the compressor.
Figure 65. Exploded view diagram generated by MARS. The illustration contains the designed subparts of the compressor in the explosion option.
After the selection of the compressor, the exact maintenance procedure, which is going to be visualized via the proposed application, is defined by the manufacturer. To that end, a set of actions are specified, which shall be displayed to the operator’s mobile device. The positioning of the object in the mobile required around 25 h. In particular, the maintenance is described by the following 5 phases:
Phase 1: Remove the bolts
Phase 2: Remove the frontal phase
Phase 3: Unscrew the sheet metal
Phase 4: Replace the gasket
Phase 5: Clean the metal flange
The traditional maintenance procedure is visualized through paper-based technical drawings, as shown in
The traditional maintenance procedure is visualized through paper-based technical drawings, as shown in
Figure 5. In addition, the compressor’s manual includes step-by-step instructions above the technical drawing. Hence, during the maintenance, significant time is spent on understanding the paper-based instructions according to the operator’s level of expertise [49]. Our system uses AR to close the time gap between highly experienced maintenance managers and not very experienced operators.
4. In addition, the compressor’s manual includes step-by-step instructions above the technical drawing. Hence, during the maintenance, significant time is spent on understanding the paper-based instructions according to the operator’s level of expertise [37]. Our system uses AR to close the time gap between highly experienced maintenance managers and not very experienced operators.
The developed application was executed in an Android smartphone with a 1.8GHz 4-Core CPU and 4GB RAM. For each phase, the application chooses a short description and the corresponding 3D object to guide the user during the disassembly. At the beginning, the user chooses to start the maintenance, while the application loads the XML file, including several information regarding the subsequent phases. After that, the application tries to locate the compressor’s features to the handeld’s frame, in order to visualize the first phase. To complete phase 1, the user has to unscrew and remove the five bolts from the frontal phase (
Figure 7a). The next phase includes the removal of the frontal phase, in the corrected position (
6a). The next phase includes the removal of the frontal phase, in the corrected position (
Figure 7b). After the disassembly of the external surface, the user is guided to unscrew the sheet metal (
6b). After the disassembly of the external surface, the user is guided to unscrew the sheet metal (
Figure 7c). Finally, the application visualizes the removal of the gasket and provides a warning message to clean the metal flange (
6c). Finally, the application visualizes the removal of the gasket and provides a warning message to clean the metal flange (
Figure 7d). At the end of the replacement, the user can navigate throughout the assembly tasks by using the back button. At any time, the user can choose the explosion option to dynamically inspect the correlation and the assemblage of the different parts (
6d). At the end of the replacement, the user can navigate throughout the assembly tasks by using the back button. At any time, the user can choose the explosion option to dynamically inspect the correlation and the assemblage of the different parts (
Figure 6).
5).
Figure 76.
Demonstration of the maintenance steps of an A/C compressor using MARMA in a mobile device.
During the final demonstration of MARMA within Industry 4.0 (now crowdhackathon), more than 20 manufacturer experts assessed the maturity level and the usage of the application in the industry. MARMA generally received positive scores and confirmed its contribution as an application that can effectively support the maintenance engineers, since the procedure can be achieved without the need for high-skilled operators. The demonstration of MARMA application to the interested manufacturers showed their willingness and the potential of the system in simplifying complex maintenance procedures. Experts mentioned that MARMA can reduce the total repair time of compressor by 30%, compared with the paper-based procedure and digital ones. In addition, particular interest was paid in exploiting the application as a means of training new and unskilled maintenance operators, reducing both the required training, as well as the total repair time. Eventually, the integration of exponential digital technologies, like MARMA, in small–medium enterprises can strengthen the industrial competitiveness in the global landscape.
The framework of MARMA can be applied not only in manufacturing environments, but also in completely different sectors, like investigation [50], infrastructure [51], and education [52]. Generally, MARMA enhances users’ field of view with real-time AR-based digital information, reducing the required time for understanding the procedures.
The framework of MARMA can be applied not only in manufacturing environments, but also in completely different sectors, like investigation [38], infrastructure [39], and education [40]. Generally, MARMA enhances users’ field of view with real-time AR-based digital information, reducing the required time for understanding the procedures.
In the case that we want to apply the MARMA in a production line, which is the most significant asset of a factory, it is crucial to perform a cost–benefit analysis [53]. The cost benefit analysis of an AR maintenance application correlates the break down frequency, the cost of spare parts, the repair fault time and external collaborators costs per machine. Based on the findings, the first machines to implement the framework of MARMA will be chosen, ensuring valuable recoverable. Then, a similar procedure with the one presented in
In the case that we want to apply the MARMA in a production line, which is the most significant asset of a factory, it is crucial to perform a cost–benefit analysis [41]. The cost benefit analysis of an AR maintenance application correlates the break down frequency, the cost of spare parts, the repair fault time and external collaborators costs per machine. Based on the findings, the first machines to implement the framework of MARMA will be chosen, ensuring valuable recoverable. Then, a similar procedure with the one presented in
has to be followed. The validity of the application depends on the probability of unexpected maintenance steps. This means that if the maintenance process is standardized by the OEM, then MARMA will work in the right way. Otherwise, the success of the process is based on the experience of the operator because he/she will need to perform additional unplanned steps. In the implementation phase, the complexity of the maintenance procedure is not a time-consuming task, compared with the availability of the 3D digital models. In this case, the OEM provides the 3D models or the manufacturer owns the digital files, then the implementation time is significant lesser than designing them from scratch.
MARMA reduces the knowledge gap between the original equipment manufacturers and maintenance operators, describing in detail the pipeline development and usage. Considering the current applications, the system is designed in a user-friendly way to be effectively controlled in fast-track repair procedures by shop-floor operators and capable to be correlated with ubiquitous maintenance system that can reduce the mean time to repair (MTR) in production’s availability. To the best of our knowledge, this is the first attempt to fully and concretely present an end-to-end AR-assisted maintenance system, by describing all of its individual parts, as well as the way that those cooperate and contribute to the final system.