基于深度智能视觉的表面缺陷检测研究进展

doi:10.3969/j.issn.1007-7375.230233

摘要/Abstract

摘要： 基于深度智能视觉的表面缺陷检测研究在制造业中起着越发重要的作用，本文阐述深度智能视觉的表面缺陷检测在现代工业质检中的重要性，对现有研究进展进行梳理总结。深度智能视觉以机器视觉和深度学习为技术基础，为不同工业场景提供高精高效的表面缺陷检测算法。本文从检测细粒度的角度将表面缺陷检测分为表面缺陷分类、定位、分割检测3个部分，并分别对分类、定位、分割方法进行系统综述，梳理现有表面缺陷检测研究的问题和思路。分类检测针对数据和缺陷图形特征问题进行研究，因其基础性和易拓展性于不同工业场景的应用呈现分散发展；定位检测以模型框架、矩形框检测和标注成本为主要问题，表现出追求轻量化和特征融合机制的研究趋势；分割检测更关注图像细节特征。通过研究分类、定位、分割的多任务模型框架以探索分类、分割检测之间的互补性。最后总结目前表面缺陷检测研究存在的问题，并对发展趋势进行展望。

关键词: 表面缺陷检测, 缺陷分类, 缺陷定位, 缺陷分割

Abstract: The exploration on surface defect detection based on deep intelligent vision plays an increasingly important role in the manufacturing industry. The importance of surface defect detection based on deep intelligent vision in modern industrial quality inspection is explained and the existing research progress is summarized in this paper. Deep intelligent vision provides high-precision and high-efficiency surface defect detection algorithms for different industrial scenarios based on the technologies of machine vision and deep learning. Surface defect detection can be divided into three categories: surface defect classification, localization, and segmentation from the perspective of detection fineness. The classification, localization, and segmentation methods are systematically reviewed, respectively, to sort out the problematic points and lines of the existing surface defect detection methods. Surface defect classification focuses on the problem of data and defective graphical features, which shows decentralized development due to its basic and easily expandable nature for application in different industrial scenarios. Surface defect localization takes the model framework, rectangular box detection mechanism, and annotation cost as the main problems, showing a research trend of pursuing lightweight and feature fusion mechanisms. Surface defect segmentation pays more attention to detailed features of an image. A multi-task framework for classification, localization, and segmentation, is studied to explore the complementarity between classification and segmentation detection. Finally, the current issues of existing surface defect detection studies are concluded and an outlook on the development trend is given.

Key words: surface defect detection, defect classification, defect localization, defect segmentation

中图分类号:

F406.3
TP183

高艺平, 王浩, 李新宇, 高亮. 基于深度智能视觉的表面缺陷检测研究进展[J]. 工业工程, 2024, 27(2): 27-36,66.

GAO Yiping, WANG Hao, LI Xinyu, GAO Liang. A Review on Surface Defect Detection Based on Deep Intelligent Vision[J]. Industrial Engineering Journal, 2024, 27(2): 27-36,66.

参考文献

[1] 周东华, 魏慕恒, 司小胜. 工业过程异常检测、寿命预测与维修决策的研究进展[J]. 自动化学报, 2013, 39(6): 711-722
ZHOU Donghua, WEI Muheng, SI Xiaosheng. A survey on anomaly detection, life prediction and maintenance decision for industrial processes[J]. Acta Automatica Sinica, 2013, 39(6): 711-722
[2] 黄聪, 农英雄, 张毅. 基于高斯受限玻尔兹曼机的工业产品质量智能异常检测[J]. 工业工程, 2022, 25(6): 152-159
HUANG Cong, NONG Yingxiong, ZHANG Yi. Intelligent Anomaly detection for industrial product quality inspection based on Gaussian restricted Boltzmann machine[J]. Industrial Engineering Journal, 2022, 25(6): 152-159
[3] 姚池, 潘尔顺. 基于两视图半监督学习的产品质量问题识别方法[J]. 工业工程, 2023, 26(3): 86-94
YAO Chi, PAN Ershun. Identification method of product quality problems based on two-view semi-supervised learning[J]. Industrial Engineering Journal, 2023, 26(3): 86-94
[4] 刘飏, 高文科, 张志胜, 等. 基于多尺度估计理论的晶圆减薄工艺方差变化检测方法[J]. 工业工程, 2018, 21(3): 75-81
LIU Yang, GAO Wenke, ZHANG Zhisheng, et al. A study of the standard deviation change in the wafer thinning process based on the multiscale estimation theory[J]. Industrial Engineering Journal, 2018, 21(3): 75-81
[5] 叶祎旎, 李艳婷. 基于CNN-集成学习的多风电机组故障诊断[J]. 工业工程, 2022, 25(1): 136-143
YE Yini, LI Yanting. Fault diagnosis of multi wind turbine based on CNN-ensemble learning[J]. Industrial Engineering Journal, 2022, 25(1): 136-143
[6] GAO Y, GAO L, LI X, et al. A multilevel information fusion-based deep learning method for vision-based defect recognition[J]. IEEE Transactions on Instrumentation and Measurement, 2020, 69(7): 3980-3991
[7] GAO Y, LI X, WANG X V, et al. A review on recent advances in vision-based defect recognition towards industrial intelligence[J]. Journal of Manufacturing Systems, 2022, 62: 753-766
[8] HE Y, SONG K, MENG Q, et al. An end-to-end steel surface defect detection approach via fusing multiple hierarchical features[J]. IEEE Transactions on Instrumentation and Measurement, 2020, 69(4): 1493-1504
[9] GAO Y, GAO L, LI X. A hierarchical training-convolutional neural network with feature alignment for steel surface defect recognition[J]. Robotics and Computer-Integrated Manufacturing, 2023, 81: 102507
[10] KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet classification with deep convolutional neural networks[J]. Communications of the ACM, 2017, 60(6): 84-90
[11] DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16x16 words: transformers for image recognition at scale[DB/OL]. (2020-10-22). https://doi.org/10.48550/arXiv.2010.119290.
[12] HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition[C/OL]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE (2016-12-12) . https://ieeexplore.ieee.org/document/7780459.
[13] IANDOLA F N, HAN S, MOSKEWICZ M W, et al. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5 MB model size[DB/OL]. (2016-02-24) . https://doi.org/10.48550/arXiv.1602.07360.
[14] LIU Z, LIN Y, CAO Y, et al. Swin transformer: hierarchical vision transformer using shifted windows[DB/OL]. (2021-03-25) . https://doi.org/10.48550/arXiv.2103.14030.
[15] CHEON S, LEE H, KIM C O, et al. Convolutional neural network for wafer surface defect classification and the detection of unknown defect class[J]. IEEE Transactions on Semiconductor Manufacturing, 2019, 32(2): 163-170
[16] WEI Y, WANG H. Mixed-type wafer defect recognition with multi-scale information fusion transformer[J]. IEEE Transactions on Semiconductor Manufacturing, 2022, 35(2): 341-352
[17] GOODFELLOW I J, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial nets[DB/OL]. (2014-06-0) . https://arxiv.org/abs/1406.2661.
[18] HE Y, SONG K, DONG H, et al. Semi-supervised defect classification of steel surface based on multi-training and generative adversarial network[J]. Optics and Lasers in Engineering, 2019(122): 294-302
[19] HU G, HUANG J, HUANG X, et al. Unsupervised fabric defect detection based on a deep convolutional generative adversarial network[J]. Textile Research Journal, 2020, 90(3-4): 247-270
[20] RADFORD A, METZ L, CHINTALA S. Unsupervised representation learning with deep convolutional generative adversarial networks[DB/OL]. (2016-01-07). https://doi.org/10.48550/arXiv.1511.06434.
[21] NIU S, LI B, WANG X, et al. Defect image sample generation with GAN for improving defect Recognition[J]. IEEE Transactions on Automation Science and Engineering, 2020, 17(3): 1611-1622
[22] YANG J, FU G, ZHU W, et al. A Deep learning-based surface defect inspection system using multiscale and channel-compressed features[J]. IEEE Transactions on Instrumentation and Measurement, 2020, 69(10): 8032-8042
[23] LIU Z, YANG B, DUAN G, et al. Visual defect inspection of metal part surface via deformable convolution and concatenate feature pyramid neural networks[J]. IEEE Transactions on Instrumentation and Measurement, 2020, 69(12): 9681-9694
[24] ZHANG D, SONG K, WANG Q, et al. Two deep learning networks for rail surface defect inspection of limited samples with line-level label[J]. IEEE Transactions on Industrial Informatics, 2021, 17(10): 6731-6741
[25] LI Y, YANG M, HUA J, et al. A channel attention-based method for micro-motor armature surface defect detection[J]. IEEE Sensors Journal, 2022, 22(9): 8672-8684
[26] ZHU Y, XIE L, YIN M, et al. Convolution with rotation invariance for online detection of tiny defects on Magnetic tile surface[J/OL]. IEEE Transactions on Instrumentation and Measurement, 2023, 72: 5021412 (2023-07-14). https://ieeexplore.ieee.org/document/10184057.
[27] WANG Y, HUANG J, WANG Y, et al. A CNN-based adaptive surface monitoring system for fused deposition modeling[J]. IEEE/ASME Transactions on Mechatronics, 2020, 25(5): 2287-2296
[28] HOWARD A G, ZHU M, CHEN B, et al. MobileNets: efficient convolutional neural networks for mobile vision applications[DB/OL]. (2017-04-17). https://doi.org/10.48550/arXiv.1704.04861.
[29] SANDLER M, HOWARD A, ZHU M, et al. MobileNetV2: inverted residuals and linear bottlenecks[C/OL]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE (2018-12-16) . https://ieeexplore.ieee.org/abstract/document/8578572.
[30] QIAN S, NING C, HU Y. MobileNetV3 for image classification[C/OL]//2021 IEEE 2nd International Conference on Big Data, Artificial Intelligence and Internet of Things Engineering (ICBAIE) . New York: IEEE (2021-04-02) . https://ieeexplore.ieee.org/document/9389905.
[31] MEHTA S, RASTEGARI M. MobileViT: light-weight, general-purpose, and mobile-friendly vision transformer[DB/OL]. (2021-10-05) . https://doi.org/10.48550/arXiv.2110.02178.
[32] ZHANG X, ZHOU X, LIN M, et al. ShuffleNet: an extremely efficient convolutional neural network for mobile devices[C/OL]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE (2018-12-16). https://ieeexplore.ieee.org/abstract/document/8578814.
[33] HINTON G, VINYALS O, DEAN J. Distilling the knowledge in a neural network[DB/OL]. (2015-03-09) . https://doi.org/10.48550/arXiv.1503.02531.
[34] LIU W, ANGUELOV D, ERHAN D, et al. SSD: single shot multibox detector[DB/OL]. (2015-12-08). https://doi.org/10.1007/978-3-319-46448-0_2.
[35] REDMON J, DIVVALA S, GIRSHICK R, et al. You only look once: unified, real-time object detection[C/OL]//Proceedings of the IEEE conference on computer vision and pattern recognition. New York: IEEE (2016-12-12). https://ieeexplore.ieee.org/document/7780460.
[36] REDMON J, FARHADI A. YOLO9000: better, faster, stronger[C/OL]//Proceedings of the IEEE conference on computer vision and pattern recognition. New York: IEEE (2017-11-09) . https://ieeexplore.ieee.org/document/8100173.
[37] REDMON J, FARHADI A. YOLOv3: an incremental improvement[DB/OL]. (2018-04-08). https://doi.org/10.48550/arXiv.1804.027678.
[38] CARION N, MASSA F, SYNNAEVE G, et al. End-to-end object detection with transformers[C/OL]//European Conference on Computer Vision. Cham: Springer International Publishing (2020-11-03). https://doi.org/10.48550/arXiv.2005.12872.
[39] LIN T Y, GOYAL P, GIRSHICK R, et al. Focal loss for dense object detection[C/OL]//Proceedings of the IEEE International Conference on Computer Vision. New York: IEEE (2017-11-25). https://ieeexplore.ieee.org/document/8237586.
[40] TIAN Z, SHEN C, CHEN H, et al. FCOS: fully convolutional one-stage object detection[C/OL]//Proceedings of the IEEE/CVF International Conference on Computer Vision. New York: IEEE (2020-02-27). https://ieeexplore.ieee.org/document/9010746.
[41] REN S, HE K, GIRSHICK R, et al. Faster R-CNN: towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(6): 1137-1149
[42] LIU Y, GAO H, GUO L, et al. A data-flow oriented deep ensemble learning method for real-time surface defect inspection[J]. IEEE Transactions on Instrumentation and Measurement, 2020, 69(7): 4681-4691
[43] BLOCK S B, DA SILVA R D, DORINI L B, et al. Inspection of imprint defects in stamped metal surfaces using deep learning and tracking[J]. IEEE Transactions on Industrial Electronics, 2021, 68(5): 4498-4507
[44] WU Y, CHEN P, QIN Y, et al. Automatic railroad track components inspection using hybrid deep learning framework[J/OL]. IEEE Transactions on Instrumentation and Measurement, 2023, 72: 5011415 (2023-04-10) . https://ieeexplore.ieee.org/document/10098120.
[45] LIU Z, MAO H, WU C Y, et al. A ConvNet for the 2020s[C/OL]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE (2022-12-27) . https://ieeexplore.ieee.org/document/9879745.
[46] CHENG X, YU J. RetinaNet with difference channel attention and adaptively spatial feature fusion for steel surface defect detection[J/OL]. IEEE Transactions on Instrumentation and Measurement, 2021, 70: 2503911 (2020-11-25). https://ieeexplore.ieee.org/document/9270024.
[47] ZENG N, WU P, WANG Z, et al. A small-sized object detection oriented multi-scale feature fusion approach with application to defect detection[J/OL]. IEEE Transactions on Instrumentation and Measurement, 2022, 71: 3507014 (2022-02-24). https://ieeexplore.ieee.org/abstract/document/9720996.
[48] LIN T Y, DOLLÁR P, GIRSHICK R, et al. Feature pyramid networks for object detection[C/OL]//Proceedings of the IEEE conference on computer vision and pattern recognition. New York: IEEE (2017-11-09). https://ieeexplore.ieee.org/document/8099589.
[49] YEUNG C C, LAM K M. Efficient fused-attention model for steel surface defect detection[J/OL]. IEEE Transactions on Instrumentation and Measurement, 2022, 71: 2510011 (2022-05-18). https://ieeexplore.ieee.org/document/9777969.
[50] NI X, MA Z, LIU J, et al. Attention network for rail surface defect detection via consistency of intersection-over-union (IoU) -guided center-point estimation[J]. IEEE Transactions on Industrial Informatics, 2022, 18(3): 1694-1705
[51] LUO J, YANG Z, LI S, et al. FPCB surface defect detection: a decoupled two-stage object detection framework[J/OL]. IEEE Transactions on Instrumentation and Measurement, 2021, 70: 5012311 (2021-06-25). https://ieeexplore.ieee.org/document/9465824.
[52] GAO Y, GAO L, LI X. A two-stage focal transformer for human-robot collaboration-based surface defect inspection[J]. ASME Journal of Manufacturing Science and Engineering, 2023: 1-32.
[53] GAO L, ZHANG J, YANG C, et al. Cas-VSwin transformer: a variant swin transformer for surface-defect detection[J]. Computers in Industry, 2022, 140: 103689 (2022-11).
[54] ZHOU H, YANG R, HU R, et al. ETDNet: Efficient Transformer-based detection network for surface defect detection[J/OL]. IEEE Transactions on Instrumentation and Measurement, 2023, 72: 2525014 (2023-08-23). https://ieeexplore.ieee.org/abstract/document/10227321 .
[55] LU H, FANG M, QIU Y, et al. An anchor-free defect detector for complex background based on pixelwise adaptive multiscale feature fusion[J/OL]. IEEE Transactions on Instrumentation and Measurement, 2023, 72: 5002312 (2022-12-19). https://ieeexplore.ieee.org/document/9992183.
[56] ZHANG K, LOU W, WANG J, et al. PA-DETR: End-to-end visually indistinguishable bolt defects detection method based on transmission line knowledge reasoning[J/OL]. IEEE Transactions on Instrumentation and Measurement, 2023, 72: 5016914 (2023-06-09). https://ieeexplore.ieee.org/document/10147381.
[57] WANG C, XIE H. MeDERT: a metal surface defect detection model[J]. IEEE Access, 2023, 11: 35469-35478
[58] YUKSEL V, TETIK Y E, BASTURK M O, et al. A novel cascaded deep learning model for the detection and quantification of defects in pipelines via magnetic flux leakage signals[J/OL]. IEEE Transactions on Instrumentation and Measurement, 2023, 72: 2512709 (2023-05-02). https://doi.org/10.1109/TIM.2023.3272377.
[59] SILVÉN O, NISKANEN M, KAUPPINEN H. Wood inspection with non-supervised clustering[J]. Machine Vision and Applications, 2003, 13(5-6): 275-285
[60] TABERNIK D, ŠELA S, SKVARČ J, et al. Segmentation-based deep-learning approach for surface-defect detection[J]. Journal of Intelligent Manufacturing, 2020, 31(3): 759-776
[61] LONG J, SHELHAMER E, DARRELL T. Fully Convolutional Networks for Semantic Segmentation[C/OL]//Proceedings of the IEEE conference on computer vision and pattern recognition. New York: IEEE (2016-05-24) . https://ieeexplore.ieee.org/document/7478072.
[62] CHEN L C, PAPANDREOU G, KOKKINOS I, et al. Semantic image segmentation with deep convolutional nets and fully connected CRFs[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(4): 834-848
[63] CHEN L C, PAPANDREOU G, KOKKINOS I, et al. DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 40(4): 834-848
[64] CHEN L C, PAPANDREOU G, SCHROFF F, et al. Rethinking atrous convolution for semantic image segmentation[DB/OL]. (2017-06-17) . https://doi.org/10.48550/arXiv.1706.05587.
[65] RONNEBERGER O, FISCHER P, BROX T. U-Net: convolutional networks for biomedical image segmentation[DB/OL]. (2015-05-18) . https://doi.org/10.48550/arXiv.1505.04597.
[66] DONG H, SONG K, HE Y, et al. PGA-Net: pyramid feature fusion and global context attention network for automated surface defect detection[J]. IEEE Transactions on Industrial Informatics, 2020, 16(12): 7448-7458
[67] LUO Q, SU J, YANG C, et al. CAT-EDNet: cross-attention transformer-based encoder-decoder network for salient defect detection of strip steel surface[J/OL]. IEEE Transactions on Instrumentation and Measurement, 2022, 71: 5009813(2022-04-14). https://ieeexplore.ieee.org/abstract/document/9757932.
[68] ZHANG Q, LAI J, ZHU J, et al. Wavelet-guided promotion-suppression transformer for surface-defect detection[J]. IEEE Transactions on Image Processing, 2023, 32: 4517-4528
[69] WANG Y, NIU M, SONG K, et al. Normal-knowledge-based pavement defect segmentation using relevance-aware and cross-reasoning mechanisms[J]. IEEE Transactions on Intelligent Transportation Systems, 2023, 24(4): 4413-4427
[70] BOŽIČ J, TABERNIK D, SKOČAJ D. Mixed supervision for surface-defect detection: from weakly to fully supervised learning[DB/OL]. (2021-04-13). https://doi.org/10.1016/j.compind.2021.103459.
[71] SAMPATH V, MAURTUA I, MARTÍN J J A, et al. Attention-guided multitask learning for surface defect identification[J]. IEEE Transactions on Industrial Informatics, 2023, 19(9): 9713-9721
[72] DU Z, GAO L, LI X. A new contrastive GAN with data augmentation for surface defect recognition under limited data[J/OL]. IEEE Transactions on Instrumentation and Measurement, 2023, 72: 3502713 (2022-12-27). https://ieeexplore.ieee.org/document/9999697.
[73] GAO Y, GAO L, LI X. A generative adversarial network based deep learning method for low-quality defect image reconstruction and recognition[J]. IEEE Transactions on Industrial Informatics, 2021, 17(5): 3231-3240
[74] JAIN S, SETH G, PARUTHI A, et al. Synthetic data augmentation for surface defect detection and classification using deep learning[J]. Journal of Intelligent Manufacturing, 2022, 33(4): 1007-1020
[75] MIRZA M, OSINDERO S. Conditional generative adversarial nets[DB/OL]. (2014-11-06) . https://doi.org/10.48550/arXiv.1411.1784.
[76] CHEN X, DUAN Y, HOUTHOOFT R, et al. InfoGAN: interpretable representation learning by information maximizing generative adversarial nets[DB/OL]. (2016-06-12). https://doi.org/10.48550/arXiv.1606.03657.
[77] NIU S, LI B, WANG X, et al. Region- and strength-controllable GAN for defect generation and segmentation in industrial images[J]. IEEE Transactions on Industrial Informatics, 2022, 18(7): 4531-4541