Defect Synthesis for Automated Visual Inspection: History
View Latest Version
Please note this is an old version of this entry, which may differ significantly from the current revision.
Contributor:
Seunghwan Song
,
Kyuchang Chang
,
Kio Yun
,
Changdong Jun
,
Jun-Geol Baek

Defect inspection, which detects defects in real-time and classifies defect types, is one of the key technologies required for smart factory implementation. Defect detection on steel surfaces is an important task to ensure the quality of industrial production. To build an automated visual inspection (AVI) and achieve smartization of steel manufacturing, detecting defects in products in real-time and accurately diagnosing the quality of products are essential elements. 

  • automated visual inspection
  • internet of things
  • generative adversarial networks

1. Introduction

In Industry 4.0, internet of things (IoT) applications are becoming important. IoT connects the physical and digital worlds, enabling smart factory development through faster communication and better analytics [1]. In general, a smart factory is one in which all internal elements are organically connected and operated intelligently based on advanced information and communication technology (ICT). Product quality must be measured in real-time to manufacture products at minimal cost and time. There is an increasing demand for steel products with better surface and shape qualities [2]. The end product of the manufacturing process is directly related to economic factors as it affects productivity. IoT applications in the steel industry can make a variety of industries more efficient and flexible, thereby increasing their productivity and yield [3][4].
Defects are physical and chemical failures caused by problems in the manufacturing process, facility, or manufacturing environment. Steel is manufactured through various processes such as rolling and forging. During this process, defects such as crazing, inclusions, pitted surfaces, rolled-in scales, and scratches occur, as shown in Figure 1 [5].
Electronics 11 02763 g001 550
Figure 1. Types of defects in the steel manufacturing process [6].
Defect inspection, which detects defects in real-time and classifies defect types, is one of the key technologies required for smart factory implementation [7][8]. Defect detection on steel surfaces is an important task to ensure the quality of industrial production. Defect detection on a steel surface involves three preliminary steps, as shown in Figure 2. The first step is inspection, in which defects on the steel surface are detected by inspection tools [9]. The second step is review, in which images of the detected defects are captured by a specific tool. The third step is the detection and classification of defect types based on the captured images. Steel-surface defect detection processes allow engineers to perform cause analysis and defect control. However, visual inspection relies heavily on the experience and abilities of individual engineers. Additionally, this process is usually performed manually in the industry, making it unreliable and time-consuming. Therefore, automated visual inspection (AVI) targeting the surface quality has emerged as a standard configuration for steel manufacturing mills to improve product quality and promote production efficiency [10]. AVI, which performs classification through image-based algorithms, is not only widely applied to the steel manufacturing process but also to glass, fiber, and semiconductor production processes [11].
Electronics 11 02763 g002 550
Figure 2. Steel manufacturing process and defect detection steps [6].
Although a convolutional neural network (CNN)-based AVI model exhibits excellent classification performance for numerous defect types, it has two practical problems in the steel manufacturing process. First, the frequency of defect data occurrence is extremely low, and very little data can be used for the development of a deep learning model [12]. In general, sufficient training data for both defect and normal classes are required to improve the classification performance of deep learning models [13]. However, in the actual industry, the quantity of defective data is minimal compared to that of normal data. When performing AVI with only data collected from the industry, data imbalance issues can result in lower learning rates for defect types and poor performance. Therefore, it is necessary to balance the normal and defective classes. Class imbalance refers to a substantial proportional difference between the classes in the total dataset. When the class distribution is unbalanced, the model is trained with a bias toward the majority class, classifying the class with a large amount of data; however, the opposite is true for the minority class. Furthermore, an imbalanced class distribution can lead to serious type II errors. Therefore, preprocessing for class imbalance is essential for improving the overall classification performance in defect detection.
Second, the steel defect data consisted of defects of various sizes. Large defects can be easily generated using a simple generator when a generative model is used to solve the imbalance problem. However, the generation of small-sized defects is significantly influenced by the type of generative model used [12]. In particular, circumstances such as the cold rolling process, where the end product is 2 m wide and the size of the defect is approximately 0.2 mm, require a sophisticated classifier [14].

2. Defect Synthesis for Automated Visual Inspection

In the Introduction, two problems that need to be solved in this research are discussed. This section addresses related work on the class imbalance problem. In particular, previous studies conducted on steel manufacturing are explored. Figure 3 shows the data imbalance of the Severstal dataset in a visual context [15]. The defect data consisted of 53.04% (5680 EA) of the total data, and the proportions of each class were as follows:12.64% (718 EA) for class 1: crazing, 3.48% (198 EA) for class 2: rolled-in scale, 72.59% (4123 EA) for class 3: pitted surface and scratch, and 11.29% (641 EA) for class 4: inclusion. In this research, the Severstal dataset was sampled and used to create a class imbalance problem. Sampling was performed only in the areas where the defect was present. The sampling method used can be found in original text.
Electronics 11 02763 g003 550
Figure 3. Imbalance ratio of steel defect images in the Severstal dataset.
The numerous solutions proposed to solve class imbalance problems in AVI can be divided into two methods: correcting the model itself and directly processing data [16]. In the former, the data instances of different classes are treated differently in a manner similar to active learning or kernel-based methods. In the latter, the direct processing of data utilizes methods such as sampling or data generation to directly control the number of instances.
Sampling is a method used to correct the bias between classes in data with an overwhelmingly small proportion of abnormal data compared with normal data. Representative methods for dealing with class imbalance include oversampling and undersampling. Oversampling is a method for creating new data of the minority class to even the class ratio, whereas undersampling is a method for removing existing data of the majority class to match the ratio. Because undersampling reduces the amount of sample data from the majority class, it has the advantage of reducing the model training time. However, it can also distort data features by removing crucial information. Regarding oversampling, the risk of data distortion is relatively small because it creates new data while preserving original data. The oversampling methods mainly used for AVI include random oversampling, synthetic minority oversampling technique (SMOTE) [17], and adaptive synthetic sampling approach (ADASYN) [18]. Random oversampling increases the amount of minority class data by randomly selecting and replicating a sample from a minority class. SMOTE synthesizes data by selecting random data belonging to a minority class and randomly selecting the closest top k number of data. ADASYN is a method of adaptively synthesizing k data from marginal minority data according to the number of majority classes after calculating the ratio of the data of a majority class. However, this oversampling method for image data generates low-resolution images.
To simplify this problem, researchers use a technique to handle the raw image. This method is called data augmentation and has been commonly used as a model regularization technique in recent studies [19]. Some common augmentation methods include flipping an image vertically or horizontally, shifting the image vertically or horizontally, and slightly rotating or zooming it. This method helps the training model be robust to small changes in the image. However, simple geometric transformations do not significantly change the image characteristics, making it impossible to identify additional features.
Among data generation methods, GAN is an algorithm of great interest [20]. The GAN generates data based on the distribution and exhibits excellent performance in image generation. Various GANs have been studied previously. A deep convolutional GAN (DCGAN) was used for ball-bearing failure detection [21]. In addition, a wafer defect image was adaptively generated using conditional GAN (CGAN) [22]. The progressive growing GAN (PGGAN) increased the model training speed by gradually increasing the generator and discriminator and producing a high-quality image [23]. In addition, to address shortcomings such as the vanishing gradient or mode collapse of a GAN, Wasserstein GAN (WGAN) was proposed [24].
Liu et al. [25] proposed a GAN-based one-class classification method for detecting strip steel surface defects. Their model achieved 94% good test results on images provided by the Handan Iron and Steel Plant. Lai et al. [26] proposed a new detection method using a GAN and statistical-based representation learning mechanism. This method achieved an accuracy of 93.75% on the solar panel dataset. Akhyar et al. [27] proposed a method for generating more detailed contours in the original steel image. The method achieves better performance and effectiveness in terms of processing time compared to the original method.
The AVI is critical for effective and efficient maintenance, repair, and operation in advanced manufacturing. However, AVI is often constrained by the lack of defect samples [28].

This entry is adapted from the peer-reviewed paper 10.3390/electronics11172763

References

  1. Brozzi, R.; Forti, D.; Rauch, E.; Matt, D.T. The Advantages of Industry 4.0 Applications for Sustainability: Results from a Sample of Manufacturing Companies. Sustainability 2020, 12, 3647.
  2. Akhyar, F.; Lin, C.Y.; Muchtar, K.; Wu, T.Y.; Ng, H.F. High efficient single-stage steel surface defect detection. In Proceedings of the 2019 International Conference on Advanced Video and Signal Based Surveillance (AVSS), Taipei, Taiwan, 18–21 September 2019; pp. 1–4.
  3. Tortorella, G.L.; Giglio, R.; van Dun, D. Industry 4.0 adoption as a moderator of the impact of lean production practices on operational performance improvement. Int. J. Oper. Prod. Manag. 2019, 39, 860–886.
  4. Beliatis, M.J.; Jensen, K.; Ellegaard, L.; Aagaard, A.; Presser, M. Next generation industrial IoT digitalization for traceability in metal manufacturing industry: A case study of industry 4.0. Electronics 2021, 10, 628.
  5. Neogi, N.; Mohanta, D.K.; Dutta, P.K. Review of vision-based steel surface inspection systems. EURASIP J. Image Video Process 2014, 1, 50.
  6. Song, S.; Baek, J.G. Defect Information Synthesis via Latent Mapping Adversarial Networks. In Proceedings of the IEEE International Conference on Artificial Intelligence in Information and Communication (ICAIIC2022), Jeju Island, Korea, 21–24 February 2022; pp. 17–22.
  7. Yun, J.P.; Shin, W.C.; Koo, G.; Kim, M.S.; Lee, C.; Lee, S.J. Automated defect inspection system for metal surfaces based on deep learning and data augmentation. J. Manuf. Syst. 2020, 55, 317–324.
  8. Li, S.; Wu, C.; Xiong, N. Hybrid Architecture Based on CNN and Transformer for Strip Steel Surface Defect Classification. Electronics 2022, 11, 1200.
  9. Wang, J.; Yang, Z.; Zhang, J.; Zhang, Q.; Chien, W.T.K. AdaBalGAN: An improved generative adversarial network with imbalanced learning for wafer defective pattern recognition. IEEE Trans. Semicond. Manuf. 2019, 32, 310–319.
  10. Luo, Q.; Fang, X.; Liu, L.; Yang, C.; Sun, Y. Automated visual defect detection for flat steel surface: A survey. IEEE Trans. Instrum. Meas. 2020, 69, 626–644.
  11. Cheon, S.; Lee, H.; Kim, C.O.; Lee, S.H. Convolutional neural network for wafer surface defect classification and the detection of unknown defect class. IEEE Trans. Semicond. Manuf. 2019, 32, 163–170.
  12. Xu, Z.J.; Zheng, Z.; Gao, X.Q. Operation optimization of the steel manufacturing process: A brief review. Int. J. Miner. Metall. Mater. 2021, 28, 1274–1287.
  13. Zhang, E.; Li, B.; Li, P.; Chen, Y. A deep learning based printing defect classification method with imbalanced samples. Symmetry 2019, 11, 1440.
  14. Kang, D.; Jang, Y.J.; Won, S. Development of an inspection system for planar steel surface using multispectral photometric stereo. Opt. Eng. 2013, 52, 039701.
  15. Severstal: Steel Defect Detection. In Kaggle. Available online: https://www.kaggle.com/c/severstal-steel-defect-detection (accessed on 20 July 2022).
  16. Kondo, N.; Harada, M.; Takagi, Y. Efficient training for automatic defect classification by image augmentation. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Tahoe, NV, USA, 12–15 March 2018; pp. 226–233.
  17. Chawla, N.V.; Bowyer, K.W.; Hall, L.O.; Kegelmeyer, W.P. SMOTE: Synthetic minority over-sampling technique. J. Artif. Intell. Res. 2002, 16, 321–357.
  18. He, H.; Bai, Y.; Garcia, E.A.; Li, S. ADASYN: Adaptive synthetic sampling approach for imbalanced learning. In Proceedings of the IEEE international joint conference on neural networks, Hong Kong, China, 1–8 June 2008; pp. 1322–1328.
  19. Cubuk, E.D.; Zoph, B.; Mane, D.; Vasudevan, V.; Le, Q.V. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 15–20 June 2019; pp. 113–123.
  20. Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; Bengio, Y. Generative adversarial nets. Adv. Neural. Inf. Process. Syst. 2014, 27, 2672–2680.
  21. Viola, J.; Chen, Y.; Wang, J. FaultFace: Deep convolutional generative adversarial network (DCGAN) based ball-bearing failure detection method. Inf. Sci. 2021, 542, 195–211.
  22. Mirza, M.; Osindero, S. Conditional generative adversarial nets. arXiv 2014, arXiv:1411.1784.
  23. Karras, T.; Aila, T.; Laine, S.; Lehtinen, J. Progressive growing of gans for improved quality, stability, and variation. arXiv 2017, arXiv:1710.10196.
  24. Arjovsky, M.; Chintala, S.; Bottou, L. Wasserstein generative adversarial networks. In Proceedings of the 2017 International conference on machine learning (PMLR), Sydney, Australia, 6–11 August 2017; pp. 214–223.
  25. Liu, K.; Li, A.; Wen, X.; Chen, H.; Yang, P. Steel Surface Defect Detection Using GAN and One-Class Classifier. In Proceedings of the 2019 25th International Conference on Automation and Computing (ICAC), Lancaster, UK, 5–7 September 2019; pp. 1–6.
  26. Lai, Y.T.K.; Hu, J.S. A Texture Generation Approach for Detection of Novel Surface Defects. In Proceedings of the 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Miyazaki, Japan, 7–10 October 2018; pp. 4357–4362.
  27. Akhyar, F.; Furqon, E.N.; Lin, C.-Y. Enhancing Precision with an Ensemble Generative Adversarial Network for Steel Surface Defect Detectors (EnsGAN-SDD). Sensors 2022, 22, 4257.
  28. Zhang, G.; Cui, K.; Hung, T.Y.; Lu, S. Defect-GAN: High-fidelity defect synthesis for automated defect inspection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA, 4–8 January 2021; pp. 2524–2534.
More
© Text is available under the terms and conditions of the Creative Commons Attribution (CC BY) license; additional terms may apply. By using this site, you agree to the Terms and Conditions and Privacy Policy.
This entry is offline, you can click here to edit this entry!
Video Production Service
Feedback