Delta Lake Medallion Architecture

Flashcards covering the characteristics, functions, and processes of the three layers in a Delta Lake medallion architecture.

Medallion architecture

A design pattern used to logically organize data in a lake house where data quality is improved as it flows through bronze, silver, and gold layers.

Bronze layer

The ingestion layer where raw data such as API dumps, database exports, and logs are stored without changes, usually partitioned hourly or daily.

Silver layer

The processing and cleansing layer that transforms raw data into a usable tabular form by removing duplicates, fixing formatting, and handling missing or corrupted values.

Gold layer

The consumption layer where curated and aggregated data is served to business users, executives, and BI tools like Power BI and Tableau.

Data enrichment

A process in the silver layer where data is improved by adding calculated columns like profit or discount rate, or by converting codes to specific names.

Business view

The result of joining multiple data sources in the gold layer, such as marketing and sales records, to create a shareable comprehensive outlook for the board.

Predefined logic

The method used in the silver layer to handle missing or corrupted values, such as converting a blank country entry into "unknown" in the dataset.

Node checks

One of several ways missing or corrupted values are handled during the cleansing process in the silver layer.