Azure Data Engineering Overview

Data engineering

Working with multiple types of data to perform data operations using various tools and scripting languages

Streaming data

Continuous flow of data records

Data pipeline

Orchestrated activities for data transfer and transformation, often used for ETL or ELT operations

Operational data

Transactional data used by applications

Analytical data

Data optimized for analysis and reporting

Apache Spark

Open-source engine for distributed data processing

Data Warehouse

Analytical data stored in a relational database, typically modeled as a star schema

Data Lake

Analytical data stored in files, distributed storage for massive scalability

Azure Stream Analytics

Azure service for running data pipelines and managing analytical data in a data lake or data warehouse

Azure Synapse Analytics

Cloud platform for data analytics, large-scale data warehousing, and advanced analytics

Azure Data Lake Storage Gen2

Distributed cloud storage for data lakes with HDFS-compatibility and flexible security

Hierarchical Namespace

Feature to enable in a blob container to use Azure Data Lake Storage Gen2

Azure Blob Storage

Storage service in Azure for unstructured data

Structured data

Data in a relational database table

Unstructured data

Data stored in a data lake

Global replication

Option to enable for using Azure Data Lake Storage Gen2

Azure Data Factory

Native pipeline functionality in Azure Synapse Analytics for ingesting and transforming data

SQL Server based pools

In Azure Synapse Analytics, used for scalable relational data processing

Data Explorer

Tool for high-performance real-time data analytics with Kusto query language

Serverless SQL pool

Feature in Azure Synapse Analytics for data exploration and analysis of files in the data lake

Dedicated SQL pool

Pool in Azure Synapse Analytics for hosting large-scale relational data warehouses

Apache Spark pool

Pool in Azure Synapse Analytics for processing and analyzing data with Apache Spark

Pipelines

Feature in Azure Synapse Analytics for transferring data between stores and applying transformations