Data Science Part2

Page 1

No content notes provided.

Page 2 - Overview of Microsoft Fabric Notebooks

Introduction

  • Microsoft Fabric Notebook: Primary code item for developing Apache Spark jobs and machine learning experiments.

  • Web-based Interactive Surface: Used by data scientists and engineers for code, visualizations, and Markdown text.

Uses of Fabric Notebooks

  • Data ingestion, preparation, and transformation by data engineers.

  • Building machine learning solutions by data scientists, including experiments, models, tracking, and deployment.

Key Features

  • Zero setup effort to get started.

  • Low-code experience for intuitive data exploration and processing.

  • Built-in enterprise security features for data protection.

  • Capability to analyze raw formats (CSV, JSON) and processed formats (Parquet, Delta Lake).

  • Enhanced authoring capabilities with built-in visualization options.

Security Context

  • Execution Triggers: Interactive runs, pipeline activities, and scheduler-triggered executions with varied security contexts—important for design considerations to avoid unexpected behavior or security issues.

Page 3 - Security Best Practices

Workflow during Notebook Execution

  • Warning Message: Displayed when a notebook is created to remind about code execution risks.

Best Practices to Avoid Security Issues

  1. Review modification updates in the Notebook settings before running.

  2. Check updates before adding notebooks to pipelines or updating the scheduler plan.

  3. Maintain separate environments (dev, test, prod) with controlled access, especially to production.

Page 4 - Notebook Creation and Importing

Creation Options

  • Create a new notebook through Fabric Data Engineering homepage, New option, or Create Hub.

  • Import Notebooks: You can import existing Jupyter Notebooks (.ipynb) or source files (.py, .scala, .sql).

Exporting Notebooks

  • Export in formats: .ipynb, .html, .py, .tex.

Page 5 - Saving Notebooks

Automatic and Manual Saving

  • Default auto-save after opening/editing a notebook.

  • Option to manually save or switch to a local branch.

  • Manual Save Option: Allows saving on command, beneficial for project organization.

Connecting to Lakehouses

  • Fabric supports easy interactions between notebooks and lakehouses, setting one as default.

  • Requires session restart after pinning a new lakehouse as default.

Page 6 - Adding/Removing Lakehouses

Lakehouse Management

  • Remove a lakehouse without deleting it from workspace.

  • Add or create more lakehouses easily.

  • Explore and view the contents of lakehouses.

File and Folder Operations

  • Load data using Spark/Pandas API by right-clicking on files in the lakehouse explorer.

Page 7 - Notebook Resource Management

Notebook Resource Explorer

  • Offers a Unix-like file system to manage folders/files, acting like local files.

  • Operations include upload, delete, and organize resources efficiently.

Page 8 - Built-in Resources and Storage Limits

Resources Storage

  • Maximum storage for built-in and environment folders is 500 MB with single file size up to 100 MB.

Accessing Resources

  • Use notebookutils.notebook.run() to access target notebook resources, path defined relative to the built-in folder.

Page 9 - Sharing and Collaboration

Collaborated Features

  • The collaborative environment allows multiple users to edit the same notebook simultaneously.

  • Visibility of colleagues' edits, cursor positions, and runs within shared notebooks enhances teamwork.

Task Collaboration

  • Features enable pair programming and remote debugging, highlighting real-time updates in a shared workspace.

Page 10 - Implementing Python in Notebooks

Code Implementation

  • Python scripts can be executed directly in the notebook, leveraging the integration for interactive data analysis.

  • Output logs and execution status are visible to enhance user experience.

Pages 11-12 - Commenting and Collaboration within Notebooks

Collaborative Commenting

  • Cell-level comments support collaborative discussion and context-sharing within notebooks.

  • Process: Select code in a cell and add comments for discussion, tagging relevant teammates enhances communication.

Page 13-15 - Notebook Version History and Collaborative Features

Version Control

  • Users can access version history to track and revert changes.

  • Manual checkpoints and system checkpoints ensure that past versions can be restored as needed.

Sharing & Permissions

  • Notebooks can be shared with customizable permission settings—view, edit, or run permissions for collaborators.

Pages 16-20 - Modes of Operation

Notebook Modes

  • Different modes allow users to switch between Development, Run-Only, Edit, and View modes, each with unique permissions.

Remaining Pages - Advanced Features and Visualizations

Advanced Features

  • The notebook supports integrations for visualizations with Python libraries, facilitating exploratory data analysis.

  • Rich code snippets and markdown formatting can be applied throughout notebooks to enhance readability and usability.

Visualizations

  • Built-in functions to turn outputs into interactive visualizations provide deeper insights into datasets.

Data Science Capabilities

  • Comprehensive data science features include support for libraries like Scikit-learn, interactive graphs, and machine learning tracking with MLFlow.