Introduction to Data Engineering in Manufacturing

Introduction to Data Engineering in Manufacturing

Scope of the Module

• Introduction to data engineering and its importance.

• Data workflow within an organization.

• Comparison of data engineer vs. data scientist roles.

• Big data in manufacturing.

• Main data engineering techniques.

• Use case examples of big data, data engineering, and AI driving manufacturing.

• Resources utilized in this module are provided at the end of the slides.

Learning Outcomes

1. Understand the importance of big data in manufacturing.

2. Compare the role of data engineer versus data scientist.

3. Identify the workflow in data engineering.

4. Explain data engineering techniques.

5. Describe examples of how big data + data engineering + AI drives manufacturing.

What is Data Engineering and Why is it Important?

• Data is growing rapidly (big data).

• Data growth is shifting towards zettabytes.

• Big data comes from personal, social, business, and manufacturing sources.

• Rapid growth of big data creates more jobs that require engineers to properly manage these large amounts of data.

• Data engineering involves designing and building infrastructure to transform and transport data into useful forms to extract value for various applications.

Data Flow Within an Organization

1. Collect and Store Data: Raw format.

2. Prepare Data: Organize the raw data.

3. Exploit Data: Visualize and explore organized data to find patterns and insights.

4. Experiment and Build Models: Use AI to forecast or predict targets (e.g., predictive maintenance, product defects in manufacturing).

Job Roles: Data Engineer vs. Data Scientist

• Characters: Thor (Data Engineer) and Scarlet (Data Scientist).

Thor (Data Engineer)

• Focuses on the first two parts of the workflow.

• Collects and stores data for easy accessibility.

• Prepares data by cleansing it for analysis by data scientists.

Scarlet (Data Scientist)

• Takes care of the last two parts of the workflow.

• Explores data and builds insightful visualizations.

• Runs experiments or builds predictive models for different applications.

Enabling Data Scientists

• Data engineers enable data scientists.

Job Responsibilities

• Data Engineer (Thor):

  • Develops scalable database architecture.

  • Streamlines data acquisition.

  • Cleans up data.

• Data Scientist (Scarlet):

  • Mines data for patterns.

  • Applies statistical modeling.

  • Builds predictive models using machine learning.

Data Engineering Techniques

• Database Architecture Development: Design and develop scalable database models.

• Streamlined Data Acquisition: Perform scheduling to move data.

• Data Cleaning: Process data, including filtering, merging, and cleansing.

Big Data Sources in a Manufacturing Plant

1. Design and engineering sources.

2. Production manufacturing sources.

3. User, service, and post-production tracking sources.

Data Engineering Techniques Applied

• Database Design:

  • Expert use of database systems.

  • Database is a computer system holding large amounts of data.

  • Decide between open source or closed database systems.

Database Management Systems

• Examples include various types available in the market (non-exhaustive list).

Database Platform Types

• Virtual Desktop Infrastructure (VDI):

  • Company owns data center and servers.

  • Requires in-house IT team for maintenance.

  • Expensive to scale up.

  • Ensures data confidentiality.

• Database as a Service (DBaaS):

  • Company does not own data center and servers.

  • Cloud-based storage.

  • Pays subscription fees to third-party vendors.

  • No hardware or maintenance costs.

  • Fast scalability.

  • Lean IT department.

  • Data confidentiality relies on the third-party data center provider.

Scheduling

• Move data as scheduled.

• Trigger can be time-based or event-based.

• Tasks: database maintenance (backup), data collection, monitoring production lines.

Data Processing

• Filtering: Narrowing a dataset to a specific group of records.

  • Example: Filtering product dataset to show only phones with two cameras.

• Joining: Combining relevant parts of datasets from multiple files using a common attribute.

  • Example: Joining datasets using Product ID as the common attribute.

• Cleansing: Ensuring consistency of datasets within columns.

  • Example: Autofilling missing company data or standardizing country names.

Big Data and AI in Manufacturing

• Data engineers lay the groundwork for data scientists to analyze big data.

• Big data can be rapidly converted into useful information.

• AI technologies can drive industry outcomes such as improved productivity and profit.

Use Case Examples in Manufacturing

• Quality Checks for Production:

  • Spotting flaws too small for the naked eye using machine vision data.

  • Improving quality check efficiency.

• Production Process Control:

  • Detecting factors influencing production process output using sensors.

  • Improving production process yield and profit.

• Predictive Maintenance:

  • Predicting when machines need maintenance with high accuracy.

  • Preventing unplanned machine downtime using sensors and alerts.

  • Enables preventive maintenance.

Machine Learning Based Visual Quality Inspection

• Visual quality inspection based on machine learning techniques is used for the defect and mismatch assessment.

• It can be used for such industries as retail, manufacturing, airport baggage screening, food industry, medicine, and etcetera.

How it works

• Images captured by a camera are processed by a neural network, which is trained to detect and localize the defect.

• Once the visual inspection system is confident about where the problem is, it takes predefined actions, like sending a notification or executing other operations.

Visual Inspection for Business

• Sunflower oil production: Visual inspection detects mismatch and notifies us in real time so that defective object can be excluded before they reach the final packaging stage.

• Different working areas on a plant require workers to have certain equipment to be worn, like helmets, gloves, or boots. The neural network is able to analyze analyze stream from monitoring cameras and notify if anyone working in the areas violates the requirements for this certain zone.

Implementation

• A correctly trained neural network ensures the high accuracy of the quality management system.

• The network architecture depends on the task: image classification, object detection, or semantic segmentation.

Challenges and Solutions in Manufacturing

• Customers expect competitive pricing, exceptional quality and immediate availability.

• Manufacturers focus on optimization and flexibility in production lines.

• Heavy automation.

• Quality control relies on up to 80% human resource for manual inspections.

• Machine vision technology has not been able to achieve the detection accuracy.

• Artificial intelligence now represents opportunity to overcome these challenges.

Fujitsu Advanced Image Recognition Technology (FARE)

• Designed to work with current manufacturing equipment, FARE can be implemented without any change to current image capture systems.

• Fair learns how to identify defects quickly from images of products, analyzing defects with a human like intelligence to achieve accuracy levels of over 99% and reducing inspection time by 80%.

Digital Transformation in Manufacturing

• Manufacturers today can fulfill their customers' individual needs through a collaborative and agile technology ecosystem.

• The biggest win for this digital transformation so far has been predictive maintenance.

• Atos delivers those benefits through a predictive maintenance framework that works like this.

Predictive Maintenance Framework

• Data from machine sensors is ingested and organized by an open, secure, cloud based IoT operating system such as Siemens MindSphere, which reports on the overall health of your assets.

• The data is stored in a cloud based database such as SAP HANA or another scalable pay as you go service such as Amazon Web Services or Microsoft Azure.

• Then the Atos Codex MindSphere application uses predictive modeling to give you the truest, most complete picture of current and future operational performance.

• Improve yields and quality.

Conclusion

• Overview of how data engineering can help prepare data for AI to benefit various applications.