Data Engineering Fundamentals Study Guide

Term (Front)	Definition (Back)
Data Engineering

Building and maintaining systems that transform raw data into reliable, usable information.

Data Pipeline

A series of processes that move and transform data between systems.

Data Engineering Lifecycle

The stages data passes through, from generation to serving.

Data Generation

The process of creating data at its original source.

Data Ingestion

Bringing data from external sources into a system.

Data Transformation

Cleaning, modifying, or restructuring data into a usable format.

Data Serving

Making processed data accessible to people or downstream systems.

ETL

Extract, Transform, Load — transforming data before loading it into its destination.

ELT

Extract, Load, Transform — loading data before transforming it in the destination system.

Reverse ETL

Moving processed data from analytical storage into operational business systems.

Data Architecture

The overall design of how an organization collects, stores, manages, and uses data.

Data Integration

Combining information from multiple sources.

Data Lake

A centralized repository that stores structured and unstructured data, often in its original format.

Data Warehouse

A structured data storage system optimized for analytics and reporting.

Data Mart

A subset of a Data Warehouse designed for a particular department or business function.

Modern Data Stack

A collection of technologies used together to ingest, store, transform, and analyze data.

Cloud Storage Bucket

A cloud-based container for storing files and other objects.

Object Storage

A storage approach that stores information as objects, often used for files and raw data.

Data Quality

The accuracy, completeness, consistency, and reliability of information.

Scalability

A system's ability to handle increasing workloads.

FinOps

Managing and optimizing the financial costs of cloud infrastructure.

Fail Loudly

Ensure system failures generate visible alerts instead of going unnoticed.

Plan for Failure

Design systems expecting that problems will eventually occur.

Loosely Coupled Systems

Systems designed to minimize unnecessary dependencies on one another.

One-Way Door

An architectural decision that is difficult to reverse.

Two-Way Door

A decision that can be reversed relatively easily.

Data Orchestration

Coordinating, scheduling, and managing tasks within data workflows.

Dependency Management

Ensuring tasks execute only when their required dependencies are satisfied.

Apache Airflow

An open-source platform used to orchestrate complex workflows and data pipelines.

DAG

Directed Acyclic Graph — a workflow defining tasks and their dependencies without cycles.

Task

An individual unit of work within a workflow.

Airflow Scheduler

Determines when task instances should run based on scheduling and dependencies.

Airflow Executor

The component responsible for executing scheduled tasks through its configured execution mechanism.

Relational Database

A database that organizes information into related tables of rows and columns.

Database Schema

A logical grouping of database objects, such as tables and views.

Primary Key (PK)

A column or set of columns that uniquely identifies each record in a table.

Foreign Key (FK)

A column or set of columns used to reference a key in another table.

OLTP

Online Transaction Processing — optimized for everyday operational transactions.

OLAP

Online Analytical Processing — optimized for complex analytical queries.

ACID

Four properties that support reliable database transactions.

Atomicity

All operations in a transaction succeed, or none do.

Consistency

Transactions maintain the database's rules and constraints.

Isolation

Concurrent transactions should not improperly interfere with each other.

Durability

Committed transactions remain saved even after system failures.

Database Transaction

A sequence of database operations treated as one logical unit of work.

Document Database

Stores information in documents, commonly using JSON-like structures.

Key-Value Store

Stores information using unique keys associated with values.

Columnar Database

Organizes data by columns, making it useful for analytical queries.

Graph Database

Stores relationships using nodes, edges, and properties.

Node

An entity or object represented in a graph database.

Edge

A relationship connecting nodes in a graph database.

Horizontal Scaling

Increasing capacity by adding more machines.

Vertical Scaling

Increasing capacity by making an existing machine more powerful.

Big Data

Large or complex datasets that can be difficult to process using traditional approaches.

Python

A programming language commonly used for automation, data processing, and developing pipelines.

SQL

Structured Query Language — used to interact with relational databases.

API

Application Programming Interface — allows different software systems to communicate.

REST API

An API that uses REST principles to exchange information between systems.

SFTP

Secure File Transfer Protocol — transfers files securely between systems.

Cron

A time-based scheduler that automatically executes commands or scripts.

Git

A version control system that tracks changes to code.

Unit Testing

Testing individual components or functions in isolation.

Integration Testing

Testing whether multiple components work correctly together.

Docker

A platform used to build, package, and run applications in containers.

Containerization

Packaging applications and dependencies into portable containers.

Infrastructure as Code (IaC)

Managing infrastructure using configuration files and code instead of manual setup.

Hadoop

A framework associated with distributed storage and large-scale data processing.

Apache Spark

A distributed engine used for processing large amounts of data.

Apache Kafka

A distributed platform used for event streaming and real-time messaging.

HDFS

Hadoop Distributed File System — stores large datasets across multiple machines.

Data Modeling

Planning how information will be structured, organized, and connected in a database.

Conceptual Data Model

A high-level representation of a business's information needs.

Logical Data Model

Defines entities, attributes, and relationships without focusing on database-specific implementation.

Physical Data Model

Specifies how a data model will actually be implemented in a database.

ERD

Entity Relationship Diagram — visually represents database entities and their relationships.

Entity

A person, object, event, or concept represented in a database.

Attribute

A characteristic or property of an entity.

One-to-One Relationship

One record is associated with one record in another table.

One-to-Many Relationship

One record can be associated with multiple records in another table.

Many-to-Many Relationship

Multiple records can relate to multiple records in another table.

Data Normalization

Organizing database tables to reduce redundancy and improve data organization.

First Normal Form (1NF)

Eliminates repeating groups and ensures fields contain individual values.

Second Normal Form (2NF)

Requires 1NF and ensures non-key attributes depend on the entire candidate key.

Third Normal Form (3NF)

Requires 2NF and eliminates transitive dependencies of non-key attributes on keys.

Normalization memory trick:

1NF = No repeating groups.

2NF = Depend on the whole key.

3NF = No non-key dependencies.

Inmon Approach

A top-down approach emphasizing a centralized, integrated enterprise Data Warehouse.

Kimball Approach

A bottom-up approach using dimensional models built around business processes.

Fact Table

Stores measurable business events and associated keys.

Dimension Table

Stores descriptive information used to provide context to facts.

Normalized Data

Data organized into related tables to reduce unnecessary redundancy.

Denormalized Data

Data structured with some intentional redundancy, often to simplify and improve analytical queries.

PII

Personally Identifiable Information — information that can identify an individual.

Direct PII

Information that can directly identify someone, such as a name or identity number.

Indirect PII

Information that may identify someone when combined with additional information.

GDPR

European Union regulation governing personal data protection.

CCPA

California privacy law establishing rights concerning personal information.

Right to Be Forgotten

The right to request erasure of qualifying personal information under applicable law.

Data Minimization

Collecting only the information necessary for a specific purpose.

Principle of Least Privilege

Giving individuals and systems only the minimum access required to perform their duties.

RBAC

Role-Based Access Control — assigns permissions based on roles.