Data Engineering Principles and Lifecycle Study Guide

Core Data Roles and Responsibilities

  • Software Engineer     * Primary Function: Building applications, websites, and services that users interact with.     * Data Collection Site: They create the initial interface (e.g., the Netflix website or mobile app) that serves as the gathering point for user data.

  • Data Engineer     * The Metaphorical "Plumber": Responsible for laying the metaphorical pipes that allow clean water (data) to flow through a business.     * ETL (Extract, Transform, Load): Their primary responsibility involves scheduling and managing data pipelines. This includes moving order data into a warehouse so it can be fulfilled.     * Maintenance and Reliability: Their focus is on building trustworthy, reliable systems rather than complex mathematical algorithms. They ensure data reaches the correct destination.

  • Data Scientist     * Pattern Identification: Uses statistics and modeling to find patterns within the data gathered by engineers.     * Predictive Analysis: Handles advanced tasks such as machine learning and customer churn prediction.     * Example Analysis: Analyzing sales data to find that specific product sales or Christmas-themed movie viewership increases during the holiday season.

  • Data Analyst     * Reporting and Visualization: Uses the clean data produced by engineers to create dashboards, visuals, and reports for sales and other business metrics.     * Operational Tasks: Typically responsible for identifying top-selling products and writing SQL queries, such as identifying the top 1010 customers based on total spend.

Practical Task Allocation

  • Building Pipelines: Move orders into the warehouse — Data Engineer.
  • SQL Queries: Finding top 1010 customers by spend — Data Analyst (primary) or Data Scientist (secondary); sometimes handled by Data Engineers.
  • Identifying Best-Selling Products: Creating visual reports of sales data — Data Analyst.
  • Customer Churn Prediction: Utilizing machine learning to predict if a customer will leave — Data Scientist.
  • Bug/Pipeline Failures: Repairing broken data flow — Data Engineer.

The Daily Workflow of a Data Engineer

  • Morning Routine (9:00 AM9:00\,AM): The first task is to check the health of the pipelines. Since ETL processes often run overnight (e.g., around 2:00 AM2:00\,AM), the engineer must verify all processes were successful.
  • Failure Investigation: If a pipeline fails, the engineer must investigate the specific records that failed validation and prevented data from passing through.
  • Stand-up Meeting: Providing updates to the team regarding which systems passed and which failed, along with identified issues.
  • Testing and Deployment: Implementing fixes, testing changes in a safe environment, and then deploying them to the live system.
  • Health Review: Monitoring dashboards to ensure overall system reliability.

Data Preparation and Quality Control

  • Cleaning Scenario: If a company receives 10,00010,000 customer records where 50%50\% have missing email addresses and many have duplicate names, the Data Engineer is responsible for the cleanup.
  • Core Tasks in Cleaning:     * Writing cleaning scripts.     * Defining quality rules (identifying what data is "good enough").     * Removing duplicates and filling (or handling) missing data.     * Transforming data into a usable format for other departments (e.g., Marketing or Finance).
  • Downstream Protection: Identifying and catching errors before they travel downstream to Analysts or Scientists. This includes feeding requirements back to Software Engineers (e.g., making certain form fields "required" in the app).

The Journey of Data through a Business (ShopSmart Example)

  • Order Placement Lifecycle:     1. Creation: A customer visits the website, browses, adds items to the basket, enters payment details, and clicks "place order."     2. Data Recorded: Customer ID, Product ID, quantity, cost, payment details, delivery address, and a timestamp.     3. Transfer: Data flows from the app interface to the database.     4. Processing: Delivery details are passed to fulfillment for shipping.     5. Storage/Reporting: The transaction eventually appears on business reports for Finance or Marketing.

Real-World Data Flow Examples

  • Food Delivery (Deliveroo, Just Eat, Uber Eats):     * Data flows from the app to the specific restaurant.     * The restaurant provides an "Accept" or "Reject" response.     * A driver is assigned, and GPS data is tracked in real-time.     * Confirms: Delivery confirmation, processed payment, and rating request data flow.

  • Streaming Services (Netflix, YouTube, TikTok):     * Netflix: Captures login, play, pause, rewind, and rating data to feed the recommendation engine overnight.     * YouTube/TikTok: The primary metric is Watch Time. High watch time for a specific genre or video type (even if it's a short) triggers recommendations for similar content.

  • Ride-Hailing (Uber):     * Dynamic Pricing: Prices are set by demand. For the same route, a user might see different prices at 1:00 AM1:00\,AM versus 9:00 AM9:00\,AM (rush hour).     * Flow: Request -> Location sync -> Driver match -> Price calculation -> Payment -> Rating.

Departmental Impact of Pipeline Failures (ETL Failure Discussion)

  • Warehouse (High Impact): Without updated ETL data, they face outdated stock levels. This leads to "overstocking" (wasted capital in moving assets) or "understocking" (lost revenue and customer complaints due to delays).
  • Finance (Moderate Impact): Meetings are delayed due to missing sales figures. While they have manual workarounds (spreadsheets, calculators), this increases the risk of human error and calculation mistakes.
  • Marketing (Lower Priority): Impacted by lost revenue potential (e.g., cannot send automated abandoned basket reminders), but deemed less critical than stock management.

Data Storage Architecture

  • Operational Database (MySQL, Postgres): Used for "live" data. Optimized for fast reading and writing of real-time transactions while a user is on the page.
  • Data Warehouse (Snowflake, BigQuery): Stores historical data. Optimized for querying large volumes of data for analysis and reporting (e.g., looking at order history).
  • Data Lake (AWS S3, Azure Data Lake): A repository for raw, unprocessed data (structured and unstructured). Used for long-term storage and massive datasets.     * Example: Spotify Wrapped uses a Data Lake to store every song a user has listened to and for how long throughout the an entire year to generate the end-of-year summary.

Miscellaneous Observations

  • Information Collected by Amazon:     * Payment: Method used (e.g., Visa, Mastercard), prices paid, and any discounts/coupons/affiliate codes applied.     * Customer Logistics: Address, postcode, and age (for restricted items like alcohol).     * Technical/Behavioral: Browser type, device used, cookies, products viewed but not bought, and time spent on specific pages.     * Delivery: Logistics data regarding whether an item is "Fulfilled by Amazon" or a third party, which determines delivery speed (e.g., same-day before 10 PM10\,PM).
  • Color Theory and Fitness (Transition Points):     * Color Pairing: "Truffle" pairs well with brown shades such as Umber, Espresso, and Cocoa, as well as neutrals like Lunar Rock.     * Fitness Routine: Exercises mentioned include RDLs (Romanian Deadlifts), single-leg hip thrusts, split squats (using an arm for balance), and cable kickbacks.