Data_Engineering_Part3

Page 1: Introduction to Fabric Runtime

Overview

• Microsoft Fabric Runtime: An Azure-integrated platform based on Apache Spark.

• Combines internal and open-source components to enable data engineering and data science experiences.

• Often referred to simply as Fabric Runtime.

Major Components

• Apache Spark: An open-source distributed computing library designed for large-scale data processing and analytics.

• Delta Lake: An open-source storage layer that adds ACID transactions and data reliability features to Apache Spark.

• Native Execution Engine: Enhances performance by executing Spark queries directly on lakehouse infrastructure, boasting up to 4x faster query speeds compared to traditional OSS Spark.

  • Compatible with Parquet and Delta formats supported in Runtime 1.3.

  • Built on Meta's Velox and Intel's Apache Gluten.

• Default-level Packages: Includes Java/Scala, Python, and R packages that are pre-installed for ease of use.

Page 2: Runtime Versions and Optimizations

Current Production Version

• Always use the most recent General Availability (GA) runtime version.

Runtime Comparison

• Check links for detailed features and migration scenarios for each runtime.

Fabric Optimizations

• Incorporates optimizations specifically for Spark and Delta Lake, designed for native integration within Fabric.

• Nearly 100 built-in query performance enhancements.

Page 3: Writer Capabilities and Runtime Changes

Writer Capabilities in Fabric

• Optimized writing processes for better performance.

• Default V-Order optimization for Delta Parquet files enhances read performance.

Support for Multiple Runtimes

• Users can switch between multiple runtimes without risk of disruption.

• Changing the runtime version affects all system-created items in the workspace, with specific guidance on doing so.

Page 4: Consequences of Runtime Changes on Settings

Migration of Spark Settings

• Aim to migrate Spark settings; warnings issued for incompatible settings.

• Configuration settings differ between mutable and immutable settings.

Page 5: Library Management During Runtime Changes

Handling Libraries

• Python and R libraries generally operate without issues when versions are unchanged.

• Jars may face compatibility issues due to dependency changes—users must address conflicts with their libraries.

Page 6: Delta Lake Protocol Management

Protocol Upgrade

• Delta Lake features are backward compatible, but forward compatibility may be compromised when certain features are enabled.

• Use method delta.upgradeTableProtocol to upgrade Delta table protocols with caution.

Page 7: Default Table Format Changes

Table Format Transition

• Runtime 1.3 changes default table format from Parquet to Delta in various Spark commands.

• Scripts assuming Parquet should be revised as Delta is now the default.

Page 8: Feedback Request

Feedback

• Provide product feedback and engage with community for further questions.

Page 9: Release Cadence of Apache Spark Runtimes

General Release Information

• Minor versions of Apache Spark are released every 6 to 9 months.

• Microsoft Fabric Spark team rapidly delivers new runtime versions.

Page 10: Lifecycle and Support Date for Runtimes

Runtime Lifecycle

• Each runtime has distinct support phases including Experimental, Public Preview, GA, LTS, and End of Support.

Page 11: Lifecycle Continued

End-of-Support Phase

• Once a runtime's end-of-support date arrives, the runtime is removed and will not receive any updates.

Page 12: Version Numbering

Runtime Versioning

• Runtime major versions correspond to Apache Spark's major version.

Page 13: Fabric Runtime 1.3 Overview

Runtime 1.3 Features

• Latest GA version introduces Apache Spark 3.5 with numerous performance enhancements.

Page 14: Apache Spark 3.5 Improvements

Notable Enhancements

• Compatibility upgrades, new features for structured streaming, and expanded functionality in PySpark.

Page 15: Delta Lake 3.2 Improvements

Enhanced Interoperability

• Improvements for Delta Lake 3.2 focused on performance and ease of use.

Page 16: Fabric Runtime 1.2 Overview

Runtime 1.2 Features

• Maintains a range of updates for improved performance and patches.

Page 17: Apache Spark 3.4.1 Features

New Enhancements

• Various fixes and enhancements including improvements in stability.

Page 18: Limitations and Best Practices

Limitations for Concurrent Writes

• New algorithms minimize data loss issues common with parallel insert operations.

Page 19: Delta Lake Advantages

ACID Transactions and Reliability

• Overview of Delta Lake capabilities reinforcing its use in lakehouse architecture.

Page 20: Feedback Reminder

Community Engagement

• Encourage users to provide feedback and engage with the community.

Pages 21-100: Continued Documentation

Further Documentation and Topics

• Covers Advanced settings, REST API usage, Git integration, Spark job definitions, and administration for enhanced user experience in Microsoft Fabric.