A Review on Surface Defect Detection Based on Deep Intelligent Vision

doi:10.3969/j.issn.1007-7375.230233

Abstract

Abstract: The exploration on surface defect detection based on deep intelligent vision plays an increasingly important role in the manufacturing industry. The importance of surface defect detection based on deep intelligent vision in modern industrial quality inspection is explained and the existing research progress is summarized in this paper. Deep intelligent vision provides high-precision and high-efficiency surface defect detection algorithms for different industrial scenarios based on the technologies of machine vision and deep learning. Surface defect detection can be divided into three categories: surface defect classification, localization, and segmentation from the perspective of detection fineness. The classification, localization, and segmentation methods are systematically reviewed, respectively, to sort out the problematic points and lines of the existing surface defect detection methods. Surface defect classification focuses on the problem of data and defective graphical features, which shows decentralized development due to its basic and easily expandable nature for application in different industrial scenarios. Surface defect localization takes the model framework, rectangular box detection mechanism, and annotation cost as the main problems, showing a research trend of pursuing lightweight and feature fusion mechanisms. Surface defect segmentation pays more attention to detailed features of an image. A multi-task framework for classification, localization, and segmentation, is studied to explore the complementarity between classification and segmentation detection. Finally, the current issues of existing surface defect detection studies are concluded and an outlook on the development trend is given.

Key words: surface defect detection, defect classification, defect localization, defect segmentation

CLC Number:

F406.3
TP183

GAO Yiping, WANG Hao, LI Xinyu, GAO Liang. A Review on Surface Defect Detection Based on Deep Intelligent Vision[J]. Industrial Engineering Journal, 2024, 27(2): 27-36,66.

References

[1] 周东华, 魏慕恒, 司小胜. 工业过程异常检测、寿命预测与维修决策的研究进展[J]. 自动化学报, 2013, 39(6): 711-722
ZHOU Donghua, WEI Muheng, SI Xiaosheng. A survey on anomaly detection, life prediction and maintenance decision for industrial processes[J]. Acta Automatica Sinica, 2013, 39(6): 711-722
[2] 黄聪, 农英雄, 张毅. 基于高斯受限玻尔兹曼机的工业产品质量智能异常检测[J]. 工业工程, 2022, 25(6): 152-159
HUANG Cong, NONG Yingxiong, ZHANG Yi. Intelligent Anomaly detection for industrial product quality inspection based on Gaussian restricted Boltzmann machine[J]. Industrial Engineering Journal, 2022, 25(6): 152-159
[3] 姚池, 潘尔顺. 基于两视图半监督学习的产品质量问题识别方法[J]. 工业工程, 2023, 26(3): 86-94
YAO Chi, PAN Ershun. Identification method of product quality problems based on two-view semi-supervised learning[J]. Industrial Engineering Journal, 2023, 26(3): 86-94
[4] 刘飏, 高文科, 张志胜, 等. 基于多尺度估计理论的晶圆减薄工艺方差变化检测方法[J]. 工业工程, 2018, 21(3): 75-81
LIU Yang, GAO Wenke, ZHANG Zhisheng, et al. A study of the standard deviation change in the wafer thinning process based on the multiscale estimation theory[J]. Industrial Engineering Journal, 2018, 21(3): 75-81
[5] 叶祎旎, 李艳婷. 基于CNN-集成学习的多风电机组故障诊断[J]. 工业工程, 2022, 25(1): 136-143
YE Yini, LI Yanting. Fault diagnosis of multi wind turbine based on CNN-ensemble learning[J]. Industrial Engineering Journal, 2022, 25(1): 136-143
[6] GAO Y, GAO L, LI X, et al. A multilevel information fusion-based deep learning method for vision-based defect recognition[J]. IEEE Transactions on Instrumentation and Measurement, 2020, 69(7): 3980-3991
[7] GAO Y, LI X, WANG X V, et al. A review on recent advances in vision-based defect recognition towards industrial intelligence[J]. Journal of Manufacturing Systems, 2022, 62: 753-766
[8] HE Y, SONG K, MENG Q, et al. An end-to-end steel surface defect detection approach via fusing multiple hierarchical features[J]. IEEE Transactions on Instrumentation and Measurement, 2020, 69(4): 1493-1504
[9] GAO Y, GAO L, LI X. A hierarchical training-convolutional neural network with feature alignment for steel surface defect recognition[J]. Robotics and Computer-Integrated Manufacturing, 2023, 81: 102507
[10] KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet classification with deep convolutional neural networks[J]. Communications of the ACM, 2017, 60(6): 84-90
[11] DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16x16 words: transformers for image recognition at scale[DB/OL]. (2020-10-22). https://doi.org/10.48550/arXiv.2010.119290.
[12] HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition[C/OL]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE (2016-12-12) . https://ieeexplore.ieee.org/document/7780459.
[13] IANDOLA F N, HAN S, MOSKEWICZ M W, et al. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5 MB model size[DB/OL]. (2016-02-24) . https://doi.org/10.48550/arXiv.1602.07360.
[14] LIU Z, LIN Y, CAO Y, et al. Swin transformer: hierarchical vision transformer using shifted windows[DB/OL]. (2021-03-25) . https://doi.org/10.48550/arXiv.2103.14030.
[15] CHEON S, LEE H, KIM C O, et al. Convolutional neural network for wafer surface defect classification and the detection of unknown defect class[J]. IEEE Transactions on Semiconductor Manufacturing, 2019, 32(2): 163-170
[16] WEI Y, WANG H. Mixed-type wafer defect recognition with multi-scale information fusion transformer[J]. IEEE Transactions on Semiconductor Manufacturing, 2022, 35(2): 341-352
[17] GOODFELLOW I J, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial nets[DB/OL]. (2014-06-0) . https://arxiv.org/abs/1406.2661.
[18] HE Y, SONG K, DONG H, et al. Semi-supervised defect classification of steel surface based on multi-training and generative adversarial network[J]. Optics and Lasers in Engineering, 2019(122): 294-302
[19] HU G, HUANG J, HUANG X, et al. Unsupervised fabric defect detection based on a deep convolutional generative adversarial network[J]. Textile Research Journal, 2020, 90(3-4): 247-270
[20] RADFORD A, METZ L, CHINTALA S. Unsupervised representation learning with deep convolutional generative adversarial networks[DB/OL]. (2016-01-07). https://doi.org/10.48550/arXiv.1511.06434.
[21] NIU S, LI B, WANG X, et al. Defect image sample generation with GAN for improving defect Recognition[J]. IEEE Transactions on Automation Science and Engineering, 2020, 17(3): 1611-1622
[22] YANG J, FU G, ZHU W, et al. A Deep learning-based surface defect inspection system using multiscale and channel-compressed features[J]. IEEE Transactions on Instrumentation and Measurement, 2020, 69(10): 8032-8042
[23] LIU Z, YANG B, DUAN G, et al. Visual defect inspection of metal part surface via deformable convolution and concatenate feature pyramid neural networks[J]. IEEE Transactions on Instrumentation and Measurement, 2020, 69(12): 9681-9694
[24] ZHANG D, SONG K, WANG Q, et al. Two deep learning networks for rail surface defect inspection of limited samples with line-level label[J]. IEEE Transactions on Industrial Informatics, 2021, 17(10): 6731-6741
[25] LI Y, YANG M, HUA J, et al. A channel attention-based method for micro-motor armature surface defect detection[J]. IEEE Sensors Journal, 2022, 22(9): 8672-8684
[26] ZHU Y, XIE L, YIN M, et al. Convolution with rotation invariance for online detection of tiny defects on Magnetic tile surface[J/OL]. IEEE Transactions on Instrumentation and Measurement, 2023, 72: 5021412 (2023-07-14). https://ieeexplore.ieee.org/document/10184057.
[27] WANG Y, HUANG J, WANG Y, et al. A CNN-based adaptive surface monitoring system for fused deposition modeling[J]. IEEE/ASME Transactions on Mechatronics, 2020, 25(5): 2287-2296
[28] HOWARD A G, ZHU M, CHEN B, et al. MobileNets: efficient convolutional neural networks for mobile vision applications[DB/OL]. (2017-04-17). https://doi.org/10.48550/arXiv.1704.04861.
[29] SANDLER M, HOWARD A, ZHU M, et al. MobileNetV2: inverted residuals and linear bottlenecks[C/OL]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE (2018-12-16) . https://ieeexplore.ieee.org/abstract/document/8578572.
[30] QIAN S, NING C, HU Y. MobileNetV3 for image classification[C/OL]//2021 IEEE 2nd International Conference on Big Data, Artificial Intelligence and Internet of Things Engineering (ICBAIE) . New York: IEEE (2021-04-02) . https://ieeexplore.ieee.org/document/9389905.
[31] MEHTA S, RASTEGARI M. MobileViT: light-weight, general-purpose, and mobile-friendly vision transformer[DB/OL]. (2021-10-05) . https://doi.org/10.48550/arXiv.2110.02178.
[32] ZHANG X, ZHOU X, LIN M, et al. ShuffleNet: an extremely efficient convolutional neural network for mobile devices[C/OL]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE (2018-12-16). https://ieeexplore.ieee.org/abstract/document/8578814.
[33] HINTON G, VINYALS O, DEAN J. Distilling the knowledge in a neural network[DB/OL]. (2015-03-09) . https://doi.org/10.48550/arXiv.1503.02531.
[34] LIU W, ANGUELOV D, ERHAN D, et al. SSD: single shot multibox detector[DB/OL]. (2015-12-08). https://doi.org/10.1007/978-3-319-46448-0_2.
[35] REDMON J, DIVVALA S, GIRSHICK R, et al. You only look once: unified, real-time object detection[C/OL]//Proceedings of the IEEE conference on computer vision and pattern recognition. New York: IEEE (2016-12-12). https://ieeexplore.ieee.org/document/7780460.
[36] REDMON J, FARHADI A. YOLO9000: better, faster, stronger[C/OL]//Proceedings of the IEEE conference on computer vision and pattern recognition. New York: IEEE (2017-11-09) . https://ieeexplore.ieee.org/document/8100173.
[37] REDMON J, FARHADI A. YOLOv3: an incremental improvement[DB/OL]. (2018-04-08). https://doi.org/10.48550/arXiv.1804.027678.
[38] CARION N, MASSA F, SYNNAEVE G, et al. End-to-end object detection with transformers[C/OL]//European Conference on Computer Vision. Cham: Springer International Publishing (2020-11-03). https://doi.org/10.48550/arXiv.2005.12872.
[39] LIN T Y, GOYAL P, GIRSHICK R, et al. Focal loss for dense object detection[C/OL]//Proceedings of the IEEE International Conference on Computer Vision. New York: IEEE (2017-11-25). https://ieeexplore.ieee.org/document/8237586.
[40] TIAN Z, SHEN C, CHEN H, et al. FCOS: fully convolutional one-stage object detection[C/OL]//Proceedings of the IEEE/CVF International Conference on Computer Vision. New York: IEEE (2020-02-27). https://ieeexplore.ieee.org/document/9010746.
[41] REN S, HE K, GIRSHICK R, et al. Faster R-CNN: towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(6): 1137-1149
[42] LIU Y, GAO H, GUO L, et al. A data-flow oriented deep ensemble learning method for real-time surface defect inspection[J]. IEEE Transactions on Instrumentation and Measurement, 2020, 69(7): 4681-4691
[43] BLOCK S B, DA SILVA R D, DORINI L B, et al. Inspection of imprint defects in stamped metal surfaces using deep learning and tracking[J]. IEEE Transactions on Industrial Electronics, 2021, 68(5): 4498-4507
[44] WU Y, CHEN P, QIN Y, et al. Automatic railroad track components inspection using hybrid deep learning framework[J/OL]. IEEE Transactions on Instrumentation and Measurement, 2023, 72: 5011415 (2023-04-10) . https://ieeexplore.ieee.org/document/10098120.
[45] LIU Z, MAO H, WU C Y, et al. A ConvNet for the 2020s[C/OL]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE (2022-12-27) . https://ieeexplore.ieee.org/document/9879745.
[46] CHENG X, YU J. RetinaNet with difference channel attention and adaptively spatial feature fusion for steel surface defect detection[J/OL]. IEEE Transactions on Instrumentation and Measurement, 2021, 70: 2503911 (2020-11-25). https://ieeexplore.ieee.org/document/9270024.
[47] ZENG N, WU P, WANG Z, et al. A small-sized object detection oriented multi-scale feature fusion approach with application to defect detection[J/OL]. IEEE Transactions on Instrumentation and Measurement, 2022, 71: 3507014 (2022-02-24). https://ieeexplore.ieee.org/abstract/document/9720996.
[48] LIN T Y, DOLLÁR P, GIRSHICK R, et al. Feature pyramid networks for object detection[C/OL]//Proceedings of the IEEE conference on computer vision and pattern recognition. New York: IEEE (2017-11-09). https://ieeexplore.ieee.org/document/8099589.
[49] YEUNG C C, LAM K M. Efficient fused-attention model for steel surface defect detection[J/OL]. IEEE Transactions on Instrumentation and Measurement, 2022, 71: 2510011 (2022-05-18). https://ieeexplore.ieee.org/document/9777969.
[50] NI X, MA Z, LIU J, et al. Attention network for rail surface defect detection via consistency of intersection-over-union (IoU) -guided center-point estimation[J]. IEEE Transactions on Industrial Informatics, 2022, 18(3): 1694-1705
[51] LUO J, YANG Z, LI S, et al. FPCB surface defect detection: a decoupled two-stage object detection framework[J/OL]. IEEE Transactions on Instrumentation and Measurement, 2021, 70: 5012311 (2021-06-25). https://ieeexplore.ieee.org/document/9465824.
[52] GAO Y, GAO L, LI X. A two-stage focal transformer for human-robot collaboration-based surface defect inspection[J]. ASME Journal of Manufacturing Science and Engineering, 2023: 1-32.
[53] GAO L, ZHANG J, YANG C, et al. Cas-VSwin transformer: a variant swin transformer for surface-defect detection[J]. Computers in Industry, 2022, 140: 103689 (2022-11).
[54] ZHOU H, YANG R, HU R, et al. ETDNet: Efficient Transformer-based detection network for surface defect detection[J/OL]. IEEE Transactions on Instrumentation and Measurement, 2023, 72: 2525014 (2023-08-23). https://ieeexplore.ieee.org/abstract/document/10227321 .
[55] LU H, FANG M, QIU Y, et al. An anchor-free defect detector for complex background based on pixelwise adaptive multiscale feature fusion[J/OL]. IEEE Transactions on Instrumentation and Measurement, 2023, 72: 5002312 (2022-12-19). https://ieeexplore.ieee.org/document/9992183.
[56] ZHANG K, LOU W, WANG J, et al. PA-DETR: End-to-end visually indistinguishable bolt defects detection method based on transmission line knowledge reasoning[J/OL]. IEEE Transactions on Instrumentation and Measurement, 2023, 72: 5016914 (2023-06-09). https://ieeexplore.ieee.org/document/10147381.
[57] WANG C, XIE H. MeDERT: a metal surface defect detection model[J]. IEEE Access, 2023, 11: 35469-35478
[58] YUKSEL V, TETIK Y E, BASTURK M O, et al. A novel cascaded deep learning model for the detection and quantification of defects in pipelines via magnetic flux leakage signals[J/OL]. IEEE Transactions on Instrumentation and Measurement, 2023, 72: 2512709 (2023-05-02). https://doi.org/10.1109/TIM.2023.3272377.
[59] SILVÉN O, NISKANEN M, KAUPPINEN H. Wood inspection with non-supervised clustering[J]. Machine Vision and Applications, 2003, 13(5-6): 275-285
[60] TABERNIK D, ŠELA S, SKVARČ J, et al. Segmentation-based deep-learning approach for surface-defect detection[J]. Journal of Intelligent Manufacturing, 2020, 31(3): 759-776
[61] LONG J, SHELHAMER E, DARRELL T. Fully Convolutional Networks for Semantic Segmentation[C/OL]//Proceedings of the IEEE conference on computer vision and pattern recognition. New York: IEEE (2016-05-24) . https://ieeexplore.ieee.org/document/7478072.
[62] CHEN L C, PAPANDREOU G, KOKKINOS I, et al. Semantic image segmentation with deep convolutional nets and fully connected CRFs[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(4): 834-848
[63] CHEN L C, PAPANDREOU G, KOKKINOS I, et al. DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 40(4): 834-848
[64] CHEN L C, PAPANDREOU G, SCHROFF F, et al. Rethinking atrous convolution for semantic image segmentation[DB/OL]. (2017-06-17) . https://doi.org/10.48550/arXiv.1706.05587.
[65] RONNEBERGER O, FISCHER P, BROX T. U-Net: convolutional networks for biomedical image segmentation[DB/OL]. (2015-05-18) . https://doi.org/10.48550/arXiv.1505.04597.
[66] DONG H, SONG K, HE Y, et al. PGA-Net: pyramid feature fusion and global context attention network for automated surface defect detection[J]. IEEE Transactions on Industrial Informatics, 2020, 16(12): 7448-7458
[67] LUO Q, SU J, YANG C, et al. CAT-EDNet: cross-attention transformer-based encoder-decoder network for salient defect detection of strip steel surface[J/OL]. IEEE Transactions on Instrumentation and Measurement, 2022, 71: 5009813(2022-04-14). https://ieeexplore.ieee.org/abstract/document/9757932.
[68] ZHANG Q, LAI J, ZHU J, et al. Wavelet-guided promotion-suppression transformer for surface-defect detection[J]. IEEE Transactions on Image Processing, 2023, 32: 4517-4528
[69] WANG Y, NIU M, SONG K, et al. Normal-knowledge-based pavement defect segmentation using relevance-aware and cross-reasoning mechanisms[J]. IEEE Transactions on Intelligent Transportation Systems, 2023, 24(4): 4413-4427
[70] BOŽIČ J, TABERNIK D, SKOČAJ D. Mixed supervision for surface-defect detection: from weakly to fully supervised learning[DB/OL]. (2021-04-13). https://doi.org/10.1016/j.compind.2021.103459.
[71] SAMPATH V, MAURTUA I, MARTÍN J J A, et al. Attention-guided multitask learning for surface defect identification[J]. IEEE Transactions on Industrial Informatics, 2023, 19(9): 9713-9721
[72] DU Z, GAO L, LI X. A new contrastive GAN with data augmentation for surface defect recognition under limited data[J/OL]. IEEE Transactions on Instrumentation and Measurement, 2023, 72: 3502713 (2022-12-27). https://ieeexplore.ieee.org/document/9999697.
[73] GAO Y, GAO L, LI X. A generative adversarial network based deep learning method for low-quality defect image reconstruction and recognition[J]. IEEE Transactions on Industrial Informatics, 2021, 17(5): 3231-3240
[74] JAIN S, SETH G, PARUTHI A, et al. Synthetic data augmentation for surface defect detection and classification using deep learning[J]. Journal of Intelligent Manufacturing, 2022, 33(4): 1007-1020
[75] MIRZA M, OSINDERO S. Conditional generative adversarial nets[DB/OL]. (2014-11-06) . https://doi.org/10.48550/arXiv.1411.1784.
[76] CHEN X, DUAN Y, HOUTHOOFT R, et al. InfoGAN: interpretable representation learning by information maximizing generative adversarial nets[DB/OL]. (2016-06-12). https://doi.org/10.48550/arXiv.1606.03657.
[77] NIU S, LI B, WANG X, et al. Region- and strength-controllable GAN for defect generation and segmentation in industrial images[J]. IEEE Transactions on Industrial Informatics, 2022, 18(7): 4531-4541