Data_Engineering_Part5

Livy API

A REST API that allows users to submit and manage Apache Spark jobs remotely.

Lakehouse

A data architecture platform in Microsoft Fabric for storing, managing, and analyzing structured and unstructured data.

Delta Lake

A unified table format designed to provide reliable data storage and enable features like ACID transactions.

Microsoft Entra

A comprehensive identity and access management service for secure application access.

Jupyter Notebooks

A web-based interactive computing environment that enables the creation of notebook documents containing live code, equations, visualizations, and narrative text.

Spark Batch Jobs

Jobs submitted to Apache Spark for processing large datasets in batch mode.

ABFS

Azure Blob File System, a service that enables access to Blob storage over the Hadoop Distributed File System (HDFS) protocol.

Azure Data Lake Storage (ADLS) Gen2

A scalable and secure data lake service built on Azure Blob Storage for big data analytics.

V-Order

A write-time optimization for Parquet files that enhances read performance in Apache Spark.

Optimize Write

A Delta Lake feature that reduces the number of small files written during data ingestion to improve performance.

Apache Livy REST API documentation

Official documentation for the Livy API, providing details on how to submit and manage Spark jobs.

MLflow

An open-source platform for managing the machine learning lifecycle.

Data Wrangler

A low-code tool for data integration and transformation within Microsoft Fabric.

NotebookUtils

A built-in package to help you perform common tasks in Fabric notebooks, including file management and running other notebooks.

Secret Redaction

The process of automatically hiding secret values from notebook outputs for security purposes.

Monitoring Hub

A feature in Microsoft Fabric that allows users to view and track the status of activities and jobs across various workspaces.

Execution Context

Information regarding the current state and environment in which a notebook is running.

Job Status

The current state of a submitted job in a computing environment, such as 'Succeeded' or 'Failed'.

Batch Job Connection String

A configured string used to establish a connection between a client and the Livy API for batch job submissions.

Touchpoint

A point of interaction or communication within Microsoft Fabric used by users to engage with tools like notebooks and APIs.

Apache Spark configuration

Settings that determine how Spark applications run, including memory allocation and the level of parallelism.

SQL Analytics Endpoint

An endpoint that allows for SQL querying of data stored in Microsoft Fabric's Lakehouse.