Enhancing Workflows with Python - Notes

  • Introduction

    • Ryan Dowling: Data Engineer at Johns Hopkins University
    • Will Preedhar: Prospect Data Analyst at Johns Hopkins University

  • Objective of the Session

    • Demonstrate how Python can automate a data processing workflow.
    • Focus on preparing data for acknowledging donor gifts.

  • Workflow Scenario

    • Regularly received dataset of new gifts that need processing into acknowledgement lists.
    • Different outreach channels: phone calls, emails, postcards, personal contact.
    • Segmentation based on predefined criteria (e.g., gift amount, assigned gift officer).

  • Automation Process Overview

    • Three Fundamental Steps:
    1. Read: Import data into Python.
    2. Calculate: Transform the data through specified calculations.
    3. Output: Export the transformed data into a deliverable format.

  • Data Structure

    • Starting with tabular data stored in Google Sheets.
    • End product is also a spreadsheet in Google Sheets, reflecting the segmented datasets.

  • Why Automate the Gift Acknowledgment Process?

    • Regular task (weekly or monthly).
    • Standardized input data format, allowing for easier automations.
    • Predefined output requirements: necessity to deliver files segmented for each outreach channel.

  • Benefits of Automation

    • Saves time compared to manual methods (e.g., Excel).
    • Frees up resources for more analytical or creative tasks.

  • Technical Actions for Python Automation

    • Read Step:
    • Import data from Google Sheets into Python.
    • Data cleaning: ensuring correct data types and formats prior to processing.
    • Calculate Step:
    • Specify transformations and calculations needed to segment the dataset.
    • Output Step:
    • Create separate acknowledgement lists and export results into the designated file format.

  • Session Goals

    • Comfortably think about automating similar workflows.
    • Learn to work efficiently in Python, utilizing cloud-based tools.
    • Break down complex data problems into manageable steps.
    • Transform tabular data using Python efficiently.
    • Utilize search engines and AI tools for troubleshooting and code generation.

  • Key Takeaways

    • The rubric of Read, Calculate, Output can be applied universally to various data processes.
    • Tools like Google Colab allow you to execute Python code without local installation.
    • Jupyter Notebook format: Code can be executed in chunks (cells), facilitating iterative testing.
    • Google Colab comes pre-installed with useful data libraries, enhancing productivity.