Fundamentals Of Data Engineering Masterclass

Concepts and notes taking from the learnings of the following video: https://youtu.be/hf2go3E2m8g?si=SR_wCm6Blra8IO1R

Data Engineering Pipeline

The process from data generation to data delivery that defines the core function of a Data Engineer (DE). DEs regularly build these for various business usecases.

Data Integration

The process of taking data from multiple data generation sources and combining them into a single record prior to data delivery.

Data Engineering Life Cycle

The overall process Data Engineers follow when building out data pipelines. Involves taking data from data generation, ingesting it into a pipeline, transforming it, storing it, and finally delivering it to users.

Data Generation

The first stage of the Data Engineering Life Cycle. Often the origin of the data DEs often utilize. Can come from the following non-exhaustive methods:

• RDBMS

• IoT Device

• API or 3rd party data

• Machine Log

Data Storage

Not to be confused with the storage layer from the Software Engineering perspective, this is the underlying layer of the ingesting, transforming, and delivery stages of the DE life cycle. Can take the following non-exhaustive forms:

• RDBMS

• NoSQL DB

• Data Warehouse(s)

• Object Storage / Data Lake (e.g. Amazon E3)

Data Modeling

A visual representation or blueprint of how how a database is organized. Often indicates how various tables are tied together via their relationships.