Secure Hardware Assurance Using Visual AI and Deep Learning on AOI Imaging of Electronic Assemblies

Surface-mounted technology (SMT) welding defect detection plays a key role in the printed circuit board assembly (PCBA) production process, which affects the use of electronic products and cost. Previous works tend to realize defect detection with only defect samples and they assume that there are sufficient defect samples. However, defect samples are usually difficult to collect in real-life scenarios while enough template samples can be easily obtained. In addition, most existing works carry out defect detection based on benchmarks with simple backgrounds of PCBA, which is not suitable for PCBA with complex structures in modern electronic product manufacturing. To address the above issues, we propose a template-guidance Siamese network based on YOLO for SMT welding defect detection (TGSYOLO), which is deployed on a real SMT automatic optical inspection (AOI) system. First, the two-stream structure is introduced to extract deep features in defect images and template images, in which template features serve as guidance knowledge. Then, a template fusion Transformer (TFT) is proposed to model global features between detect and template features in the low-level stage, which could acquire long-range correlations to force the network to focus on potential defect regions. Next, to avoid the disappearance of tiny defect features during deep feature fusion, a multiscale attention feature pyramid network (MAFPN) is proposed to directly fuse defect semantic information from low-level features, which retains detailed expressions of defects through skip connection and obtains compact fusion features. Furthermore, we collect limited welding defect samples based on more complex PCBA backgrounds than previous works through a real SMT AOI system. Experiments on the limited dataset show that TGSYOLO could reach 0.985 of mAP@0.5, 0.885 of mAP@0.75, and 0.984 of F1, which is 0.008, 0.054, and 0.025 higher than other SOTA methods. Also, generalization experiments on the public DeepPCB show that TGSYOLO could still reach the best with 0.991 of mAP@0.5 and 0.89 of mAP@0.75, which proves that TGSYOLO has good generalization performance.

Automatic Optical Inspection (AOI) is any method of detecting defects during a Printed Circuit Board (PCB) manufacturing process. Early AOI methods were based on classic image processing algorithms using a reference PCB. The traditional methods require very complex and inflexible preprocessing stages. With recent advances in the field of deep learning, especially Convolutional Neural Networks (CNN), automating various computer vision tasks has been established. Limited research has been carried out in the past on using CNN for AOI. The present systems are inflexible and require a lot of preprocessing steps or a complex illumination system to improve the accuracy. This paper studies the effectiveness of using CNN to detect soldering bridge faults in a PCB assembly. The paper presents a method for designing an optimized CNN architecture to detect soldering faults in a PCBA. The proposed CNN architecture is compared with the state-of-the-art object detection architecture, namely YOLO, with respect to detection accuracy, processing time, and memory requirement. The results of our experiments show that the proposed CNN architecture has a 3.0% better average precision, has 50% less number of parameters and infers in half the time as YOLO. The experimental results prove the effectiveness of using CNN in AOI by using images of a PCB assembly without any reference image, any complex preprocessing stage, or a complex illumination system. Doi: 10.28991/HIJ-2022-03-01-01 Full Text: PDF

With a shortage of highly-skilled employees for printed circuit board assembly, the electronics industry faces a severe challenge, which is being complicated with a high turnover rate as employees jump between companies or simply become misaligned with the organization. While leading industry organizations like IPC and SMTA are tackling the issue head-on with education and training programs, a need exists for equipment suppliers to cooperate with other industry leaders and organizations to adopt Machine-to-Machine (M2M) communication standards. Initiatives like the IPC Connected Factory Exchange (CFX) and the IPC-Hermes-9852 standard underpin the efforts within the industry, and many companies are working together to develop standards to create seamless production. Guided in part by Industry 4.0, these M2M communication standards are quickly altering the manufacturing process by improving metrics like first pass yield and throughput by applying autonomous process adjustments. Far beyond an automatic line changeover, this bi-directional communication allows equipment to automatically adjust production parameters to increase board quality and lower costs by eliminating rework and scrap. Building on our expertise and testing, this presentation will explore Industry 4.0, and then explore how Advanced Process Control (APC) can increase production yields and reduce defects. Specifically, the presentation will discuss how APC improves process repeatability by automatically adjusting component placement to the paste, rather than to the pad location. Moreover, it will show how APC will identify a shift trend and implement further position correction by using true 3D measurement data from the Automatic Optical Inspection (AOI) system. While a smart factory will help resolve the skilled employee challenge, it will also revolutionize process optimization.

This paper examines the process real-world SMT manufacturers are employing to evaluate, select and implement automated optical inspection (AOI) systems for post-placement/pre-reflow inspection during the PCB assembly process. A case study from a prominent European SMT manufacturer describes the process used to evaluate a post-placement/pre-reflow AOI system. Traditional, algorithm-based AOI techniques are briefly described and evaluated, along with their strengths and limitations. A new AOI methodology, called statistical appearance modeling, and a stereo-vision inspection machine are briefly described and compared to traditional AOI systems. Examples of payback and ROI calculations and analysis are highlighted. The system requirements are specified and the actual results of the AOI performance trial are summarized. Other relevant topics include analyzing GR&R, interpreting SPC data, and implementing post/placement/pre-reflow AOI.

Automated Optical Inspection (AOI) is among the most common and effective quality checks employed in production lines. This paper details the design of a Deep Learning solution that was developed for addressing a specific quality control in a Printed Circuit Board Assembly (PCBA) manufacturing process. The developed Deep Neural Network exploits transfer learning and a synthetic data generation process to be trained even if the quantity of the data samples available is low. The overall AOI system was designed to be deployed on low-cost hardware with limited computing capabilities to ease its deployment in industrial settings.

Research on image classification sparked the latest deep-learning boom. Many downstream tasks, including semantic segmentation, benefit from it. The state-of-the-art semantic segmentation models are all based on deep learning, and they sometimes make some semantic mistakes. In a semantic segmentation dataset with a small number of categories, images are often collected from a single scene, and there is a close semantic connection between any two categories. However, in the semantic segmentation dataset collected from multiple scenes, two categories may be irrelevant. The probability of objects in one category appearing next to objects in other categories is different, which is the basis of the paper. Semantic segmentation methods need to solve two problems of positioning and classification. This paper is dedicated to correcting those clearly wrong classifications that are contrary to reality. Specifically, we first calculate the relevancy between different class pairs. Then, based on this knowledge, we infer the category of a connected component according to the relationships of this connected component with its surrounding connected components and correct the obviously wrong classifications made by a deep learning semantic segmentation model. Several well-performing deep learning models are experimented on two challenging public datasets in the field of semantic image segmentation. Our proposed method improves the performance of UPerNet, OCRNet and SETR from 40.7%, 43% and 48.64% to 42.07%, 44.09% and 49.09% mean IoU on the ADE20K validation set, and the performance of PSPNet, DeepLabV3 and OCRNet from 37.26%, 37.3% and 39.5% to 38.93%, 38.95% and 40.63% mean IoU on the COCO-Stuff dataset, which shows the effectiveness of the method.

In computer vision, shape recovery is a classic problem. The shape from shading problem aims to reconstruct a 3D shape from one or more intensity images. Inspection of solder joint is a critical step in the assembly of PCB (Printed Circuit Board). So, non-destructive inspection technology has been widely used in a production of PCB. In automated optical inspection (AOI) field 2D inspection appears to be more prevalent industry-wide than 3D inspection, but the limit of 2D inspection in some defects is apparent. This paper deals with the reconstruction of the solder joint's surface in PCB based on shape form shading technology. The advantages of SFS method are analyzed firstly. Then, theory and survey of SFS method is introduced. To overcome the disadvantages existing in conventional SFS methods, some improved algorithms are provided and proved by experiments. Experimental results reveal that the proposed method shows practical value in solder joint inspection.

Chapter 12 - Semantic scene segmentation for robotics

Printed circuit boards (PCBs) are primarily used to connect electronic components to each other. It is one of the most important stages in the manufacturing of electronic products. A small defect in the PCB can make the final product inoperable. Therefore, careful and meticulous defect detection steps are necessary and indispensable in the PCB manufacturing process. The detection methods can generally be divided into manual inspection and automatic optical inspection (AOI). The main disadvantage of manual detection is that the detection speed is too slow, resulting in a waste of human resources and costs. Thus, in order to speed up the production speed, AOI techniques have been adopted by many PCB manufacturers. Most current AOI mechanisms use traditional optical algorithms. These algorithms can easily lead to misjudgments due to different light and shadow changes caused by slight differences in PCB placement or solder amount so that qualified PCBs are judged as defective products, which is also the main reason for the high misjudgment rate of AOI detection. In order to effectively solve the problem of AOI misjudgment, manual re-judgment is currently the reinspection method adopted by most PCB manufacturers for defective products judged by AOI. Undoubtedly, the need for inspectors is another kind of labor cost. To reduce the labor cost. of manual re-judgement, an accurate and efficient PCB defect reinspection mechanism based on deep learning algorithm is proposed. This mechanism mainly establishes two detection models, which can classify the defects of the product. When both models have basic recognition capabilities, the two models are then combined into a main model to improve the accuracy of defect detection. In the study, the data provided by Lite-On Technology Co., Ltd. were implemented. To achieve the practical application value in the industry, this research not only considers the problem of detection accuracy, but also considers the problem of detection execution speed. Therefore, fewer parameters are used in the construction of the model. The research results show that the accuracy rate of defect detection is about 95%, and the recall rate is 94%. Compared with other detection modules, the execution speed is greatly improved. The detection time of each image is only 0.027 s, which fully meets the purpose of industrial practical application.

The wood-based furniture manufacturing industries prioritize quality of production to meet higher market demands. Identifying various types of edge-glued wooden panel defects are a challenge for a human worker or a camera. Several studies have shown that the detection of edge-glued defects with low, high, normal, overlong, short is identified but detection of residue and bluntness is highly challenging. Thus, the present model identifies defects of low, high, normal, overlong, short by computer vision and/or deep learning, whereas defects of residue and bluntness by deep learning based decide by pass for having better performance. The goal of this paper is to provide an improved defect detection solution for wood-based furniture manufacturing industries by process automation. Therefore, a system was designed that takes defect input images from a camera as raw image and laser-aligned image for defect detection of the edge-glued wooden panel. The process automation then performs computer vision-based image features extraction with deep learning for defect detection. The aim of this paper is to solve edge-glued defect detection problems by using design and implementation of edge-glued wooden defect detection, that can be stated as edge-glued wooden panel defect detection using deep learning (WDD-DL) for process automation by artificial intelligence and Automated Optical Inspection (AOI) consolidation. Possibly there exist several types of defects on the edges while edge-banding on the wooden panel in furniture manufacturing. Therefore, the scope is to achieve higher accuracy by raw image and laser-aligned image feature extraction using deep learning algorithms for final result defect classification in WDD-DL by AOI. The WDD-DL system uses Gabor, Harris corner, morphology, structured light detection and curvature calculation for pre-processing and InceptionResnetV2 Convolutional Neural Network algorithm to attain the best results. The applications of this work can be found in quality control of the furniture manufacturing industry for an edge, corner, joint defect detection of the wooden panels. The WDD-DL achieves best results as the precision, recall and F1 score are 0.97, 0.90 and 0.92, respectively. The experiments demonstrate higher accuracy achievement as compared to other methods with overkill and escape rate analysis. Ultimately, the discussion section provides an interesting experience sharing about the necessary factors for implementing the WDD-DL in real-time industrial operations.

Automated Optical Inspection (AOI) Systems are commonly used on Printed Circuit Boards (PCB) manufacturing. The use of this technology has been proven as highly efficient for process improvements and quality achievements. The correct extraction of the component for posterior analysis is a critical step of the AOI process. Nowadays, the Pattern Matching Algorithm is commonly used, although this algorithm requires extensive calculations and is time consuming. This paper will present an improved algorithm for the component localization process, with the capability of implementation in a parallel execution system.

Automated optical inspection (AOI) system is popularly used in the manufacturing industry. It can be used for accurate and precise inspection and measurement beyond human inspector. However, the development of an AOI system is unlike the general system. There are some specific steps or considerations simply only for the development of AOI system. In order to design the development process of AOI system systematically, a series of guidelines are proposed in this paper based on the practical experience. Many important issues are listed and discussed. Some practical examples are used to illustrate the importance of guidelines. The guidelines are also mapping to the traditional waterfall model for better understanding. They are expected to be helpful to increase the efficiency of AOI system development.

Semantic segmentation is an important task in self-driving cars. The aims of semantic segmentation are to recognize pre-defined objects and its pixel-by-pixel location. The most popular method in semantic segmentation is Deep learning which has considerably improved semantic image segmentation. This work does an overview for semantic segmentation using Deep learning. This works also implement comparisons in term of precision, mean IOU and processing time. Three popular algorithms are PSPNet, FCN and SegNet that are examined carefully. In detail, the aim of this work points out a trade-off between processing time and mean IOU, and also between processing time and precision. Moreover, this paper concentrates on road segmentation for embedded devices, so processing time is significantly important. This work also figures out which method is suitable for embedded devices on road segmentation.

Semantic segmentation based on deep learning has very broad application scenarios in the field of computer vision. DeepLab v3+ is currently a widely used semantic segmentation framework. This paper uses the DeepLab v3+ model in TensorFlow to semantically segment foreign objects on the airport runway. Experimental results show that the semantic segmentation model based on DeepLab v3+ can accurately classify foreign objects at the pixel level, which effectively improves the accuracy of foreign object detection and reduces the risk of accidental aircraft take-off and landing.

An Automatic Optical Inspection (AOI) system for optical inspection of imaging devices used in automotive industry using an inspecting optics of lower spatial resolution than the device under inspection is described. This system is robust and with no moving parts. The cycle time is small. Its main advantage is that it is capable of detecting and quantifying defects in regular patterns, working below the Shannon-Nyquist criterion for optical resolution, using a single low resolution image sensor. It is easily scalable, which is an important advantage in industrial applications, since the same inspecting sensor can be reused for increasingly higher spatial resolutions of the devices to be inspected. The optical inspection is implemented with a notch multi-band Fourier filter, making the procedure especially fitted for regular patterns, like the ones that can be produced in image displays and Head Up Displays (HUDs). The regular patterns are used in production line only, for inspection purposes. For image displays, functional defects are detected at the level of a sub-image display grid element unit. Functional defects are the ones impairing the function of the display, and are preferred in AOI to the direct geometric imaging, since those are the ones directly related with the end-user experience. The shift in emphasis from geometric imaging to functional imaging is critical, since it is this that allows quantitative inspection, below Shannon-Nyquist. For HUDs, the functional detect detection addresses defects resulting from the combined effect of the image display and the image forming optics.