Data_Engineering_Part4

Spark Job Definition Inline Monitoring

A feature that allows real-time monitoring of Spark job submissions and execution status.

Pipeline Spark Activity Inline Monitoring

Enables viewing of Spark application execution details within pipelines, including snapshots and logs.

Apache Spark Advisor

Provides real-time advice and recommendations for optimizing Spark notebook runs.

Run Series Analysis

Automatically categorizes Spark applications based on recurring activities and detects anomalies.

Autotune Analysis

Compares outcomes from autotune to enhance Spark application performance.

Outlier Detection

Identifies and analyzes outlier runs in the Spark run series to troubleshoot performance.

Extended History Server

A tool used for debugging and diagnosing completed Spark applications.

Spark Capacity Consumption

Reports on the resource usage of Spark jobs, providing insights for administrators.

Livy API

An API helpful for submitting Spark code within Microsoft Fabric without creating Notebooks.

Intelligent Cache

A feature that caches data to reduce latency and improve the execution speed of Spark jobs.

Data Skew

An issue where data distribution is uneven across tasks, leading to performance bottlenecks.

Kusto Queries

Used to retrieve logs and metrics from Azure Log Analytics and inspect Spark application data.

Session Jobs

Livy jobs that maintain an active Spark session for state-sharing between jobs.

Batch Jobs

Livy jobs that do not maintain an ongoing session, useful for single job execution.

Output of Autotune

Performance metrics provided after autotuning Spark SQL queries for optimization.

Focused Mode

A feature that allows zooming in on specific visual details for better monitoring.

Spark SQL Analytics Endpoint

A way to access SQL-based queries from Spark applications for data management.