Augmented Reality-Artificial Intelligence Tools in Manufacturing: History
View Latest Version
Please note this is an old version of this entry, which may differ significantly from the current revision.
Contributor:
Andrzej Szajna
,
Mariusz Kostrzewski

An important research area in the field of Industry 4.0 is to find a user-interface that is as convenient and intuitive to use as possible to ensure optimal human–machine interaction. Augmented Reality (AR) together with Advanced Image Recognition, powered by Artificial Intelligence (AI) seem to be a set of technologies supportive in this topic. Apart from user friendly interface, AR-AI tools are proved to provide time savings in manufacturing tasks, while simplifying the job at the same time, enabling inexperienced, unskilled, or less skilled employees to perform the work in the selected manual production processes.

  • AR glasses
  • artificial intelligence
  • augmented reality
  • industry 4.0
  • production
  • assembly
  • human-machine interaction interface
  • migration
  • refugee
  • war
  • Ukraine
  • Russia
  • Covid

1. Introduction

Industry 4.0 in a complex way is most often understood as the digitalization of the production environment. As a term, it was coined by Prof. Wolfgang Wahlster in Germany [1][2]. There has been a paradigm shift from mass production to mass customization, resulting from increasingly rapid changes in consumer tastes, rapid changes in demand, and the emergence of new competitors [3][4][5][6]. To survive and succeed in such competitive conditions, manufacturers should manage product varieties effectively [7]. Smart manufacturing aims to improve energy efficiency, productivity, and production quality while minimizing product life cycles and reducing environmental damage [8][9][10].
Regardless of the implementation of the idea of Industry 4.0, in any industry where significant automation of production or logistics processes or operations has been introduced, manual activities based on human labor are invariably important [11][12][13]. For these reasons, in addition to robotization and automation, researchers should always consider human needs in their work. As noted by researchers at the German Development Center for Artificial Intelligence (German: Deutsches Forschungszentrum für Künstliche Intelligenz, DFKI), an important research area in the field of Industry 4.0 is to find a user-interface that is as convenient and intuitive to use as possible to ensure optimal human–machine interaction [14]. That is why new concepts, techniques, and tools are emerging to support management, production, and logistics processes, as well as to support shop floor workers [15][16].
A sustainable approach to business operations requires the addition of appropriate resources, including qualified staff. However, what should be done during civilization crises that humanity periodically faces? Industry 4.0 is also a response to the ever-growing problem of a shortage of skilled employees in many countries worldwide. The Russian army’s invasion on Ukraine, and many Ukrainians unexpected movements towards and from their country, show an additional large scale of unpredictability in the labor markets. Certainly, this is not the only migration crisis of its kind in the world (e.g., Mexican migration to the United States of America, as recently discussed in [17]). From the business point of view, such people’s migrations are linked to the exodus of workers from factories, and consequently, it is a trigger for finding remedies. Not much earlier, the labor market experienced turbulence connected to staff shortages due to the spread of COVID-19 and quarantines, starting in 2020, or the also noticeable previous European migrant crisis in 2015, when employment of the coming people from Syria was one of the serious points to be solved (it was estimated that Turkey dealt with 3 million Syrian refugees [18]). Such turns of events stand in line with the challenge observed for years—namely, finding qualified employees with nonexpendable qualifications to meet the proper skills requirements in a workplace. These situations trigger the creation of innovative tools or complete workplaces, as presented in this research. An intuitive support tool could aid in such critical situations and enable new, unskilled people to work. Entering a new job in a new place is challenging for everyone, even the native inhabitants of a place. The level of challenges increases for refugees, with additional factors, such as language barrier, unfamiliarity with customs, the lack of recognition of the home country’s educational degrees and work experience, and outright discrimination—as researched in Austria [19]. Acknowledgements can be found that state that refugees in Germany are highly motivated, but they need special support because of “initial disadvantages” [20][21]. Such a support can be ensured owing to sophisticated technologies, e.g., augmented reality (AR) and artificial intelligence (AI).

2. AR-AI Tools allow the manual operations time in production to be decreased

Reaserch shows that the usage of the AR-AI tool is able to provide 10 to 40% decrease in the time needed to perform selected manual production processes [22][23].

The first example of the productin process concerned the assembly of wires in the control cabinet, with the median number of 300 wires in one complete cabinet. The research on six testers resulted in a serious improvement in the process—the assembly time saved was on average 2:15 (hh:mm) thanks to the AR-AI support tools (including AR glasses and the AR Smart Wiring 4.0 system). The average required assembly time with the AR-AI tool was reduced by 40%. That means a savings of one working shift (8 h) for every 3.5 assembled cabinets, which is a strongly significant result.

It should be noted that the speed of the testers’ adaptation to the assembly system, both with and without AR glasses, was different. A similar observation, although in the case of a different production process, was described by Sanna et al. [24]. However, correlations were observed among group of testers with similar characteristics. The shortest duration of experiments was among relatively young testers (born 1983–1989) with certain experience in the industry, including positions directly related to the assembly of control cabinets. The slowest courses of experiments were done by older testers (born 1966–1969) with no previous experience in the industry. Younger testers (born in 1989) with no experience in the industry saved the greatest amount of time in minutes. This result can be associated with the younger age, whose representatives often show natural skills in using the latest technologies (this is connected to rapid adaptation in a work environment as well). For this young, unexperienced group, the results of experiments without the AR-AI tool, associated with the need to interpret the 3D electrics schematics on the touchscreen, were clearly worse than for the group with industrial experience.

The second process was the inspection of the production line, typically lasting 45 min as a task that is traditionally done manually. The average time saved by five testers, with the use of the AR prototype was 17:05 (mm:ss)—the required inspection time with the AR-AI tool was reduced by 37%. The research into this solution continues and is yielding increasingly satisfactory results. Of the five testers, two had a background in the process and three represented the unskilled group.

The third process was the quality control of the manufactured product, performed directly in a workplace. The introduction of the AR-AI tool allowed the process to be reconfigured. With the appropriate AR-AI tool, instead of operators going to the control place for the check, the control employee headed to the operators’ workplaces. This resulted in a complete reduction of the downtime observed in the original process, within which the operators left their manufacturing places and moved to the control spot to conduct the quality control process. For a single operator working for one shift (8 h assumed), the total production downtime was simulated to equal on average of 45 min. Elimination of this time provided a savings of 10% in the case of regular operators' time.

It is worth highlighting the fact that both experienced employees and people without a particular understanding of the elementary activities and manual operations participated in the tests of the processes mentioned above. This attests to the implementation maturity and application potential of the presented solutions in enterprises willing to employ people with non-technical preparation to perform technical and engineering tasks.

3. AR-AI Tools as a Response to High Employee Turnover and Shortages

Applications based on AR and AI technologies supporting manual production processes, coping with staff shortages, and new staff training (or new skills) led the researchers of this research to develop a panoramic review as a comprehensive and thematically extensive presentation of the matter. The interest was to analyze publications connected to novel and inexperienced employee apprenticeship and support in manual operations, mostly within manufacturing processes, yet not limited to them solely. According to [25], till 2030, AR will play a significant role in the majority of industries, including military [26], medical and healthcare [27], navigation, gaming and entertainment, manufacturing, and especially—from the viewpoint of the current research—education, training, and remote assistance (providing training in a safe environment).
There are plenty of researches of the scientific contributions of AR-AI technology applications, some of them quoted below. Therefore, this research focuses on presenting the most current solutions (screening with the phrase “AR training in manufacturing research”). They are presented by year of publication.
Gonzalez-Franco et al. [28] implemented a mixed reality (MR) setup with see-through cameras attached to a head-mounted display to analyze a manufacturing procedure of aircraft door maintenance. MR training and the conventional face-to-face training were considered and compared. The differences between them were not significant. On the other hand, it was observed that MR setups can achieve high performance in the context of collaborative training.
Ferrati et al. [29] presented research on AR technology with a focus on error rates and the average assembly times during the industrial process of cherry picking. Their AR system consisted of a HoloLens with software built in Unity and Vuforia. It is worth mentioning that the testers who were working in this research experienced no discomfort or headache and it was stated as well that the training delivered through the AR tool had improved outcomes compared to the ones provided the traditional way. The results of the training process indicated that requests for help from additional trainers decreased, that learning time was improved by 22% and that time spent picking was reduced by 26%. Sorko and Brunnhofer [30] also stated that set-up times and processing times can be shortened with the application of the AR technology. On top of that, they mentioned that paths during various industrial operations can be shortened, which is reflected in the aforementioned time reduction and consequently less employee tiredness. Moreover, their research led the researchers to the statement that employees can be individually trained in a protected training environment with less training staff effort and reduced error rates.
Osborne and Mavers [31] presented an AR literature review showing both advantages (e.g., hands free to conduct tasks, full transparency of the real world, increase in task efficiency) and disadvantages (e.g., delays in obtaining information, issues when wearing corrective glasses). They noted that training and complex assemblies can be simplified, so a new employee can accomplish a process task without extensive experience.
Büttner et al. [32] investigated the use of a projection-based AR assistive system in a training scenario. Three training configurations were analyzed: the AR assistive system, a paper manual, and personal training. The personal training results were the most efficient. The AR system, however, satisfied the challenge of a trainee mislearning content, but it was not faster than the training done by a human. Despite such results, the researchers agreed that AR technology can be successfully applied as trainer-less tools for training employees in manufacturing.
Most of the research on training processes with AR applications versus the traditional approach has been finalized with a survey related to the content and effectiveness of the instructional outcomes [29][30][31]. The researchers of the mentioned research agreed that AR technologies can be applied for training both skilled employees and those requiring new skills.
It is also worth pointing out research papers with a literature review connected to AR technology. Bottani and Vignali [33] reviewed the literature connected to AR published between 2006 and early 2017 to identify the main areas and sectors of AR application. Without a doubt, that literature review should be updated (even the researchers themselves pointed out in the future research section that the review needs to be continued in greater detail to consider industrial sectors where AR systems could be successfully deployed), but the researchers specified industrial sectors, the representatives of which had pointed out the benefits of AR technologies. Among these sectors were laboratories, manufacturing, machine tools, architecture, engineering, construction and operations (AECO), and automotive. The researchers of [33] noted that at the time of publication, the AR applications were often little more than experimental prototypes. Quandt and Fraitag [34] reviewed publications connected to users’acceptance of AR technologies, released between 2011 and 2020, in the following areas of application: maintenance, training in the work environment, assembly, and order picking. In the case of training, they mentioned Fraga-Lamas et al. [35], who focused on providing step-by-step instructions on a specific piece of equipment or machine, which depended on the employees' level of knowledge, or passing knowledge from experienced employees to less experienced ones with implementation of the AR technology.
An evaluation of the AR solutions in manufacturing was also conducted by Zigart and Schlund [36]. Meanwhile, Doolani et al. [37] considered the topic from a broader perspective and reviewed research papers connected to manufacturing training with the application of extended reality (XR; including AR as well as VR—virtual reality). The researchers planned a clear progress in the application of such advance technologies and specified their domains. Kaplan et al. [38] specified the key benefits of using XR-based training, stressing the ease of implementation in difficult environments (e.g., the disaster training presented in [39]). On the other hand, it was also noted that the XR-based training poses a threat to traditional training. Werrlich et al. [40] pointed to an important result, from the perspective of the current research: the employees trained with the AR systems performed better in short- and long-term training than in the case of traditional trainings. What is also very interesting, as far as the review of [37] is concerned, is that the researchers specified which phases of manufacturing can be supported/trained with AR technology: the learning phase (with tasks such as sorting, picking, keeping, assembling, installation), the tangent phase (with tasks such as using rare tools/machinery, hand tools, power tools), and the end phase (with tasks such as an inspection and cleaning routine, including processing, shoveling, sweeping, and cleaning work areas).
At AR technology conferences, including the most respected ones, such as the International Symposium on Mixed and Augmented Reality (ISMAR) conferences, it can be noticed that in the first period of intensive AR development (review of the first decade of the ISMAR conference, 1998–2008 [41]), the biggest contribution to this field was devoted to the software and hardware parts of the AR technology itself (tracking, interaction, calibration, display), as the fundamentals of AR had a long way to go to achieve maturity [41]. In a review of AR trends in the second decade of the ISMAR conferences (2008–2018), described in [42], there was a significant increase in research on applications and evaluation (an increase from 5% to 16%), which was the second result after research on tracking techniques (19%).
It is worth presenting a summary of the above contributions to state a certain research gap. Numerous researchers considered staff training with the application of AR, MR, or XR in their publications and projects: training with AR and MR technologies in [28], training with AR glasses [29], and training with XR technologies [37][39]. Some of the researchers proved that the training could demand less training staff effort and ensure reduced error rates [30]. Other advantages of staff training with AR technologies were mentioned in [31][32], together with an investigation of the disadvantages of such potential. Almost none of the contributions considered AR technology application from the viewpoint of human migrations and consequently searching for new job opportunities by people with no or few qualifications. However, to prove that it is possible to transfer knowledge and abilities of more experienced personnel to those less so with the application of AR technologies, the publication by [35] can be mentioned. Any significant similarity to the current contribution was not found in review papers such as, e.g., [33][34][41][42].

This entry is adapted from the peer-reviewed paper 10.3390/su14116729

References

  1. Kagermann, H.; Lukas, W.D.; Wahlster, W. Industrie 4.0: Mit dem Internet der Dinge auf dem Weg zur 4. Industriellen Revolution; VDI Nachrichten, VDI Verlag GmbH: Düsseldorf, Germany, 2011; Available online: https://www.dfki.de/fileadmin/user_upload/DFKI/Medien/News_Media/Presse/Presse-Highlights/vdinach2011a13-ind4.0-Internet-Dinge.pdf (accessed on 2 December 2020).
  2. Gilchrist, A. Industry 4.0: The Industrial Internet of Things, 1st ed.; Apress: Berkeley, CA, USA, 2016; pp. 1–259.
  3. Piller, F.T. Mass Customization: Reflections on the State of the Concept. Int. J. Flex. Manuf. Syst. 2004, 16, 313–334.
  4. Tseng, H.E.; Chang, C.C.; Chang, S.H. Applying Case-based Reasoning for Product Configuration in Mass Customization Environments. Expert Syst. Appl. 2005, 29, 913–925.
  5. Fogliatto, F.S.; Da Silveira, G.J.C.; Borenstein, D. The Mass Customization Decade: An Updated Review of the Literature. Int. J. Prod. Econ. 2012, 138, 14–25.
  6. Yu, B.; Zhao, H.; Xue, D. A Multi-population Co-evolutionary Genetic Programming Approach for Optimal Mass Customisation Production. Int. J. Prod. Res. 2017, 55, 621–641.
  7. ElMaraghy, H.; Schuh, G.; ElMaraghy, W.; Piller, F.; Schönsleben, P.; Tseng, M.; Bernard, A. Product Variety Management. CIRP Ann.—Manuf. Technol. 2013, 62, 629–652.
  8. Rajput, S.; Singh, S.P. Connecting circular economy and industry 4.0. Int. J. Inf. Manag. 2019, 49, 98–113.
  9. Galati, F.; Bigliardi, B. Industry 4.0: Emerging themes and future research avenues using a text mining approach. Comput. Ind. 2019, 109, 100–113.
  10. Müller, F.; Jaeger, D.; Hanewinkel, M. Digitization in wood supply—A review on how industry 4.0 will change the forest value chain. Comput. Electron. Agric. 2019, 162, 206–218.
  11. Bonekamp, L.; Sure, M. Consequences of Industry 4.0 on Human Labour and Work Organisation. J. Bus. Media Psychol. 2015, 6, 33–40.
  12. Rodriguez, L.; Quint, F.; Gorecky, D.; Romero, D.; Siller, H.R. Developing a Mixed Reality Assistance System Based on Projection Mapping Technology for Manual Operations at Assembly Workstations. Procedia Comput. Sci. 2015, 75, 327–333.
  13. Plinta, D.; Dulina, L. Kształtowanie Systemów Produkcyjnych z Wykorzystaniem Technologii Rozszerzonej Rzeczywistości (Production Systems Design with Using Augmented Reality Technologies). In Innowacje w Zarządzaniu i Inżynierii Produkcji (Innovations in Production Management and Engineering); Knosala, R., Ed.; Polskie Towarzystwo Zarządzania Produkcją: Opole, Poland, 2018; Volume 2, pp. 682–691.
  14. Gorecky, D.; Schmitt, M.; Loskyll, M.; Zühlke, D. Human-machine-interaction in the industry 4.0 era. In Proceedings of the Industrial Informatics (INDIN) 12th IEEE International Conference, Porto Alegre, Brazil, 27–30 July 2014; pp. 289–294.
  15. Syberfeldt, A.; Holm, M.; Danielsson, O.; Wang, L.; Brewster, R.L. Support Systems on the Industrial Shop-floors of the Future: Operators’ Perspective on Augmented Reality. Procedia CIRP 2016, 44, 108–113.
  16. Kurdve, M.; Bellgran, M. Green lean operationalisation of the circular economy concept on production shop floor level. J. Clean. Prod. 2021, 278, 123223.
  17. Khan, R.B.; Bhatti, M.N.; Mustafa, G. Understanding of Mexican Migration to the United States: Rifts and Challenges. J. Arts Soc. Sci. (JASS) 2021, 8, 72–82.
  18. Kizil, C. Turkey’s Policy on Employment of Syrian Refugees and its Impact on the Turkish Labour Market. In Turkish Migration 2016 Selected Papers; Eroğlu, D., Cohen, J.H., Sirkeci, I., Eds.; Transnational Press London: London, UK, 2016; pp. 164–170.
  19. Verwiebe, R.; Kittel, B.; Dellinger, F.; Liebhart, C.; Schiestl, F.; Haindorfer, R.; Liedl, B. Finding your way into employment against all odds? Successful job search of refugees in Austria. J. Ethn. Migr. Stud. 2019, 45, 1401–1418.
  20. Worbs, S.; Baraulina, T. Female Refugees in Germany: Language, Education and Employment. (BAMFBrief Analysis, 1-2017). Nürnberg: Bundesamt für Migration und Flüchtlinge (BAMF) Forschungszentrum Migration, Integration und Asyl (FZ). Available online: https://nbn-resolving.org/urn:nbn:de:0168-ssoar-67555-1 (accessed on 20 April 2022).
  21. Fleury, A. Understanding Women and Migration: A Literature Review. Available online: http://www.atina.org.rs/sites/default/files/KNOMAD%20Understaning%20Women%20and%20Migration.pdf (accessed on 20 April 2022).
  22. Szajna, A.; Stryjski, R.; Woźniak, W.; Chamier-Gliszczyński, N.; Kostrzewski, M. Assessment of Augmented Reality in Manual Wiring Production Process with Use of Mobile AR Glasses. Sensors 2020, 20, 4755.
  23. Bauer, T.K.; Kunze, A. The Demand for High-Skilled Workers and Immigration Policy. SSRN J. 2004, 47, 77–88.
  24. Sanna, A.; Manuri, F.; Piumatti, G.; Paravati, G.; Lamberti, F.; Pezzolla, P. A Flexible AR-based Training System for Industrial Maintenance. In Proceeding of the Augmented and Virtual Reality. AVR 2015; Lecture Notes in Computer Science; De Paolis, L., Mongelli, A., Eds.; Springer: Cham, Switzerland, 2015; Volume 9254, pp. 314–331.
  25. Kaviyaraj, R.; Uma, M. A Survey on Future of Augmented Reality with AI in Education. In Proceedings of the 2021 International Conference on Artificial Intelligence and Smart Systems (ICAIS), Coimbatore, India, 25–27 March 2021; pp. 47–52.
  26. Wang, W.; Lei, S.; Liu, H.; Li, T.; Qu, J.; Qiu, A. Augmented Reality in Maintenance Training for Military Equipment. J. Phys. Conf. Ser. 2020, 1626, 012184.
  27. Harris, T.E.; DeLellis, S.F.; Heneghan, J.S.; Buckman, R.F.; Miller, G.T.; Magee, J.H.; Vasios, W.N.; Nelson, K.J.; Kane, S.F.; Choi, Y.S. Augmented Reality Forward Damage Control Procedures for Nonsurgeons: A Feasibility Demonstration. Mil. Med. 2020, 185 (Suppl. 1), 521–525.
  28. Gonzalez-Franco, M.; Pizarro, R.; Cermeron, J.; Li, K.; Thorn, J.; Hutabarat, W.; Tiwari, A.; Bermell-Garcia, P. Immersive Mixed Reality for Manufacturing Training. Front. Robot. AI 2017, 4, 3.
  29. Ferrati, F.; Erkoyuncu, J.A.; Court, S. Developing an Augmented Reality Based Training Demonstrator for Manufacturing Cherry Pickers. Procedia CIRP 2019, 81, 803–808.
  30. Sorko, S.R.; Brunnhofer, M. Potentials of Augmented Reality in Training. Procedia Manuf. 2019, 31, 85–90.
  31. Osborne, M.; Mavers, S. Integrating Augmented Reality in Training and Industrial Applications. In Proceedings of the 2019 Eighth International Conference on Educational Innovation through Technology (EITT), Biloxi, MS, USA, 27–31 October 2019; pp. 142–146.
  32. Büttner, S.; Prilla, M.; Röcker, C. Augmented Reality Training for Industrial Assembly Work—Are Projection-based AR Assistive Systems an Appropriate Tool for Assembly Training? In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA, 25–30 April 2020.
  33. Bottani, E.; Vignali, G. Augmented reality technology in the manufacturing industry: A review of the last decade. IISE Trans. 2019, 51, 284–310.
  34. Quandt, M.; Freitag, M. A Systematic Review of User Acceptance in Industrial Augmented Reality. Front. Educ. 2021, 6, 498.
  35. Fraga-Lamas, P.; Fernandez-Carames, T.M.; Blanco-Novoa, O.; VilarMontesinos, M.A. A Review on Industrial Augmented Reality Systems for the Industry 4.0 Shipyard. IEEE Access 2018, 6, 13358–13375.
  36. Zigart, T.; Schlund, S. Evaluation of Augmented Reality Technologies in Manufacturing—A Literature Review. In Advances in Human Factors and Systems Interaction. AHFE 2020. Advances in Intelligent Systems and Computing; Nunes, I., Ed.; Springer: Cham, Switzerland, 2020; Volume 1207, pp. 75–82.
  37. Doolani, S.; Wessels, C.; Kanal, V.; Sevastopoulos, C.; Jaiswal, A.; Nambiappan, H.; Makedon, F. A Review of Extended Reality (XR) Technologies for Manufacturing Training. Technologies 2020, 8, 77.
  38. Kaplan, A.D.; Cruit, J.; Endsley, M.; Beers, S.M.; Sawyer, B.D.; Hancock, P. The Effects of Virtual Reality, Augmented Reality, and Mixed Reality as Training Enhancement Methods: A Meta-Analysis. Hum. Factors 2021, 63, 706–726.
  39. Farra, S.; Miller, E.; Timm, N.; Schafer, J. Improved training for disasters using 3-D virtual reality simulation. West. J. Nurs. Res. 2013, 35, 655–671.
  40. Werrlich, S.; Eichstetter, E.; Nitsche, K.; Notni, G. An overview of evaluations using augmented reality for assembly training tasks. Int. J. Comput. Inf. Eng. 2017, 11, 1068–1074.
  41. Zhou, F.; Duh, H.B.L.; Billinghurst, M. Trends in augmented reality tracking, interaction and display: A review of ten years of ISMAR. In Proceedings of the 7th IEEE/ACM International Symposium on Mixed and Augmented Reality, ISMAR ‘08, Cambridge, UK, 15–18 September 2008; pp. 193–202.
  42. Kim, K.; Billinghurst, M.; Bruder, G.; Duh, H.B.L.; Welch, G.F. Revisiting trends in augmented reality research: A review of the 2nd decade of ISMAR (2008–2017). IEEE Trans. Vis. Comput. Graph. 2018, 24, 2947–2962.
More
© Text is available under the terms and conditions of the Creative Commons Attribution (CC BY) license; additional terms may apply. By using this site, you agree to the Terms and Conditions and Privacy Policy.
This entry is offline, you can click here to edit this entry!
Video Production Service
Feedback