Intelligent Defect Inspection Powered by Computer Vision and Deep Learning

However, quality control remains an issue. And it’s a costly one: many companies experience quality-related costs as high as 15%-20% of sales revenue. In some cases, those can even go as high as 40% of the total number of operations. The European Commission also estimated that in some industries 50% of production can be scrapped due to defects; the defect rate can reach a staggering 90% in complex manufacturing lines.

The biggest problem for manufacturers is that even a slight variance in production processes or materials (invisible to the human eye) can make the entire production run defective. Surely, those parts will not reach the end users due to extensive post-production quality checks. However, relying on current (mostly manual) defect inspection practices means that thousands of items can be manufactured at significant expense before the defect is discovered.

Moving from Manual to Automated Optical Inspection

As mentioned already, manual inspection of products, parts, and components can be a cumbersome and expensive process. Firstly, it involves significant training for human experts to perform such inspections. Secondly, such inspections can cause bottlenecks in the production/time to market timelines. Thirdly, manual inspections do not scale as products do – further training is required to have enough specialists for performing timely inspections.

As a replacement for more cumbersome and error-prone manual inspection emerged the automated optical inspection (AOI), powered by machine vision. The new types of AOI systems are equipped with multi-cameras ranging from simple XGA (Extended Graphics Array) units to high-resolution, multiple-megapixel video sensors. Depending on the camera type, an AOI system can either provide monochrome or color images of the inspected items, and the captured images can span a wide range, from mere thousands of data points to millions of data points.

Source

The benefits of automated optical inspection are multifold:

• Such systems enable early error detection in the manufacturing processes and help ensure high quality of the item before it’s moved to the next manufacturing step.
• AOI helps gather historical and production statistics used to improve manufacturing lines.
• As a result, it will help reduce material waste, repair and rework costs, as well as added manufacturing labor time and expenses.

AOI systems can be programmed to use a different technique for quality assurance and defect inspection such as:

• Template matching: the system is programmed to compare the obtained item image with the image of perfectly made, non-defective items. The system first learns about all the correct attributes of a certain part of the item and then assesses the quality of a produced item according to the estimated standards.
• Pattern matching: the system stores information of both good and bad assemblies, comparing and contrasting the actual product versus available patterns.
• Statistical pattern matching: In this case, the system stores the results of several products and several types of defects, so that it becomes capable of greenlighting acceptable minor deviations without flagging errors.

Automated optical inspection systems have been a major breakthrough in quality control, providing more accurate and rapid inspections throughout the production process. However, it is still not the pinnacle of automated defect inspection.

By incorporating AI and deep learning, not only can the optical inspection process images of already produced products, but also identify the defects and over time, learn more about different types of defects (without explicit programming). Ultimately, it will be targeted at conducting predictive analyses and thus, reaching error-free production.

How AI and Deep Learning Can Further Improve the Visual Inspection Process

While automated optical systems are a powerful method of inspecting defects, they are still relatively slow, inaccurate and expensive in maintenance. With the rapid transition towards Industry 4.0., most companies can no longer lose time and resources on long-term setups. For instance, final assembly verification is extremely difficult to program due to numerous variables that can be hard for an AOI to isolate (e.g., lighting, changes in curvature, color, etc.). Though statistical pattern matching can help tolerate some variability in items’ appearance, complex surface textures and image quality issues can pose some serious inspection challenges.

In addition, machine vision systems typically cannot cope with the following issues:

• They cannot properly distinguish the variability and deviation between visually similar parts.
• Also, they struggle to properly differentiate between “functional” defects – that are almost always a cause for rejection; and “cosmetic” defects – some mild issues with the part’s overall look that are not viewed as critical by the manufacturer.

Computer vision and deep learning-based systems have emerged as a powerful alternative to AOI systems, addressing the aforementioned shortcomings.

What is computer vision?

Computer vision is a subfield of artificial intelligence and machine learning based on specific algorithms and other methods allowing computers to understand the content of digital images. In short, computer vision software attempts to reproduce the capability of human vision.

There are two specific tasks most computer vision tools are aimed at solving:

• Object classification: a model is trained on a dataset of specific objects (e.g., images of defects) and then it classifies new as belonging to one or more of your training categories.
• Object identification: a model is trained to recognize a specific instance of an object. For instance, when it determines two components in an image it can tag one as a circuit board and another as a microcontroller.

Here’s how computer vision works in simple terms:

Source

The interpreting device (computers + software) is the element that performs most of the work in this case. Models, trained with the help of machine learning and deep learning techniques, deconstruct the incoming visual data into pixels, then assess them according to various parameters and compare to other images in the dataset to find the best “match” and make a valid prediction of what it is.

At this point, it’s important to recap what is deep learning.

“Deep learning is a subdivision of machine learning with a strong emphasis on teaching computers to learn like humans: by being presented with an example.”

Unlike machine learning, deep learning models do not need to be constantly programmed with explicit instructions for analyzing data. Typically, such models are only presented with a large dataset of relevant information and some initial parameters for operationalizing the data. They can churn such data and self-learn to predict which output (e.g., a classification) is accurate or not.

If you want to learn more about deep learning, refer to our Executive Guide to Deep Learning and Neural Networks for Businesses.

Neural networks and deep learning models can help overcome the current limitations within AOI systems.

Source

Deep learning-based defect inspections are particularly effective when it comes to assessing complex surfaces and detecting cosmetic defects such as scratches or dents. As well, such systems have the ability to inspect more precisely or classify features of certain items based on their defining characteristics – even if those characteristics vary in subtle but acceptable ways.

Additionally, they are more adaptable when it comes to analyzing hard-to-capture visual elements. A good example is one of the recent projects Infopulse team developed together with a German manufacturer. They managed to come with an IoT device capturing and recognizing digits on old gauges despite image defects from 7-segment LCD display, such as glares, white spots, reflections of objects or people, etc. For this, our team had to build a convolutional neural network (CNN) that could capture and process the image in less than 2 seconds, saving the client a lot of operational expenses on manual data collection. Similar computer vision solutions can also be implemented to analyze different types of visual defects.

Computer vision algorithms and applications can be powered both by deep learning or machine learning. The latter is often a more suitable choice for performing assessments related to gauging, measuring, and performing a precision alignment. However, deep learning systems can greatly complement ML-based ones as it allows computerizing visual and sound inspections that previously required specific human expertise. This technology broadens the limits of what a computer and camera/sensors can accurately inspect.

Source

Use Cases of Intelligent Defect Inspection, Powered by Deep Learning

Consider the importance of industries involved in high-value manufacturing – aerospace, automotive, construction, medical devices. Defects in parts or components can quite literally prove fatal.

At the same time, these industries (just like many others) face pressure to “get to market” faster, if they are to remain competitive and meet production deadlines of customers/clients. Computer vision and deep learning can prove to be the answer to these pressures while still ensuring that the parts/components are without defects.

Below are several examples of deployments and technical proofs of concepts executed by manufacturers across different industries:

Source

1. The Airline Industry

It’s not just a question of meeting government requirements and specifications for the production of all parts and components of airplanes, but also a matter of costly delays when defects are found after the fact.

According to Boeing, 70% of a $2.6 trillion aerospace services market is now dedicated to quality and maintenance. This should come as no surprise considering that mechanical issues in the U.S. alone, in 2016, were estimated to cost more than half a billion dollars. Also, delays (many with customers left sitting on board) were about one-third maintenance-related.

Visual inspection technologies, coupled with DL, can compare components to specs and find flaws in the production process. Beyond that, manufacturers should also meet scheduled maintenance requirements to ensure passenger safety and follow government guidelines. These technologies can identify such flaws and issues as:

• Corrosion – computer vision systems can perform inspections of the various parts in less time and return more thorough inspection results.
• Internal engine components wear and tear – conduct a remote visual inspection of internal engine components to ensure efficient and secure operation.
• Aluminum panel thickness (via 3D measurements) – thickness gauges can provide high precision, non-destructive readings on large and complex aluminum panels; and upload the collected data to your systems for further analysis.
• Automated wheel inspection – receive instant data about fatigue cracks on aircraft wheels. By using systems powered by computer vision, the operators don’t have to waste time on lengthy inspections and can focus on other work at hand.
• Blade inspection during manufacturing or post-repair – computed radiography and digital radiography can be used to inspect blades for crucial flaws. As a result, images can be obtained quickly without using chemicals in a digital format. The latter can be further put on failure analysis and bring in precise 3D measurements.

2. The Automotive Industry

Safety issues are the things that keep OEMs and Tier 1 suppliers up at night. It’s a matter of passenger risk/death, as well as the company’s reputation. Recently, Toyota has agreed to a $1.3 billion settlement because of a defect that caused cars to accelerate, even when drivers attempted to slow down. Six deaths in the U.S. resulted from this defect.

Using the cognitive capabilities of visual inspections, AI, and deep learning, OEMs can analyze and identify quality issues much more accurately and even resolve them before they occur. Over time, these visual insights and analyses can determine the severity of defects, as they become acquainted with more types of defects.

One leading automotive manufacturer employed an intelligent optical inspection solution to detect scratch and dent defects on metal surfaces. It was extremely difficult for human inspectors to notice such defects on shiny surfaces. Traditional machine inspection was of little help as well, as the system could not capture unpredictable dent or scratch defects. Another issue was that the system had to be trained by human staff by supplying images of defective parts to teach it to distinguish what’s acceptable or not. However, the company had a rather low defect rate in the first place, it took them a lot of time and effort to gather even a small dataset of visuals representing enough defects variations. Eventually, they opted for a more “intelligent” system and quickly realized the following benefits post-adoption:

• 66% reduction in inspection cost
• 33% reduction in the False Positive rate
• False Negative rate maintained at 1% or less

3. Computer Equipment Manufacturing

AOI (automated optical inspection) used to be the natural choice for the computer equipment in the manufacturing industry, especially since the demand for smaller circuit board designs is growing and because, in some instances, defects can prove highly injurious.

Among other things, AOI can detect:

• Area defects
• Component offsets
• Solder issues
• Foreign materials
• Height and volume defects
• Short circuits
• Mounting errors

When specifications and margins for variance are set for AOI inspections, they are far more accurate than human inspections. However, AOI systems still fail to detect “cosmetic” issues of the equipment.

That’s why, Fujitsu Laboratories has been spearheading the development of AI-enabled recognition systems for the electronics industry and reports a great deal of progress in areas of quality, cost, and delivery. By relying on genetic programming, the company’s image recognition system actually generated inspection code and was able to produce almost 100% rate of recognition. Applying this technology to its production sites, the company reports about 80% reduction in development time and a recognition rate of 97%.

However, it is not just high-value manufacturing that can use automated visual inspection and AI. Consider these two examples from consumer-based industries.

4. The Textile Industry

At some point, all of us have purchased clothing and found those little stickers inside saying “Inspected by (some number)”. This means that a human employee manually went over the product looking for defects in stitching, weaving, etc. and determined that it met the company’s specs for quality. Obviously, this is a costly and fallible process. Customers who receive flawed products have to go through the return/exchange process, something that does not endear them to the company.

The implementation of automated visual inspection, along with DL functionality, can now detect issues of texture, weaving, stitching, and even color matching. When flaws are detected, not only can a textile or product be removed from the line, but also the root cause of a defect can be found, fixed and prevented in the future.

At the same time, such systems can be fine-tuned to accept a larger range of fault tolerance. For instance, Datacolor’s AI system can take into account historical data of past visual inspections conducted by human operators to create custom tolerances that match more closely to the samples of visual inspections.

One of the newer AI use cases in the textile industry is creating a system for detecting and measuring wrinkles on the fabric. The current process of measuring fabric wrinkle performance is largely manual and thus, tedious. Computer Vision-powered wrinkle measurement may help manufacturers reduce costs and time required for this process.

5. The Glass Industry

One of the issues often faced by this industry is flaws in the glass during the production – chips, scratches, etc. Oftentimes, the current inspection methods produce “false positives,” such as water drops being identified as chips.

With the implementation of custom computer vision and AI models, these types of errors can be eliminated. In one case, correct defect identification saved a large glass manufacturer an average of $36,000 in savings on each production line, totaling annual savings of close to $1 million.

Conclusions

Quality control remains a major factor in a company’s ability to maintain the manufacturing standards, customer satisfaction, and, ultimately, its reputation and profits. Computer vision and deep learning hold tremendous potential for quality control in virtually every industrial sector. So much so that most executives plan to double their use of cognitive computing for operational optimization in the next 3 years.

Where are you in the process of digitizing your operations? Infopulse team has already assisted several major OEMs with the transition towards Industry 4.0. and development of cutting-edge solutions for optimizing low value, human-centered processes. Let’s have a discussion about jump-starting or enhancing your current quality control initiatives!