3.1-Data-Management

Introduction to Applied Data Science

  • Instructor: Seth Bullock (H/T Edmund Hunt, Luis Vaquero, lan Nabney)

  • Website: bristol.ac.uk

Learning Objectives

  • By the end of the lecture, students should be familiar with:

    • Different data storage types

    • Different database types

    • How data management relates to applications

Data Management: A Historical Perspective

  • Concept: Data management is intertwined with the history of technology and society, reflecting on how data has been used through time.

  • Cultural Reference: Stephen Fry's "Great Leap Years" illustrates the evolution of data management techniques.

Historical Milestones in Data Management

Jacquard Looms

  • Year: 1804

  • Description: A stored 'programme' controls weaving patterns, marking advancement in programmable machines.

Herman Hollerith and Punched Cards

  • Description: Introduced punched cards for data processing.

  • Milestones (1890 to 2016):

    • 1890: Hollerith Punched Cards

    • 1924: CTR becomes IBM

    • 1970: Development of SQL

    • Modern frameworks: PostGreSQL, Bigtable, MongoDB, etc.

Database Trends

  • Trends focus on different database technologies and their popularity.

  • Key Players (2014-2024): Oracle, MySQL, Microsoft SQL Server, PostgreSQL, MongoDB, Snowflake, etc.

Data Architecture

Layers

  • Management Layer: Handles data and operational processes separately.

  • Analytics Layer: Retrieves data efficiently without affecting the management processes.

  • Challenges: The architecture should address scalability and complexity.

Storage Types and Technologies

Types of Storage

  • Blob Storage: Unstructured data, accessed via REST APIs, suitable for data lakes (e.g., images, videos).

  • Disk Storage: Mountable on Virtual Machines, often used as front for Blob storage (e.g., Azure).

  • File Storage: Functions like a file system, supports file transfers via protocols like SMB.

Summary of Storage Types

  • Blob Storage: For unstructured data; use when scaling data lakes.

  • Disk Storage: For persistent attached storage; use for local applications.

  • File Storage: For applications interacting through traditional file system APIs.

Database Categories

Relational Databases

  • Characteristics:

    • Structured as tables

    • Use SQL for querying

    • Ensure ACID transactions

  • Challenges: Performance issues with increasing complexity.

Non-Relational Databases (NoSQL)

  • Emerged for handling big data and real-time applications.

  • BASE Properties: Basically Available, Soft state, Eventually consistent.

Data Warehouses vs. Data Lakes

Feature

Data Warehouse

Data Lake

Types

Structured, Relational

Structured, Unstructured, Semi-structured

Schema

Schema on Write

Schema on Read

Data Profile

Processed, Vetted

Raw, Unfiltered

Sources

Application, Transactional Data

Big Data, IoT, Streaming Data

Scalability

Difficult and Expensive to Scale

Easy and Cheap to Scale

Typical Users

Data Warehouse Professionals

Data Scientists, Data Engineers

Use Cases

Core Reporting, Business Intelligence

ML, Predictive Analytics

/

Database Classifications

Types

  • Key-Value, Document, Column, Graph

Key-Value Stores

  • Description: Like a dictionary, with a strict format for keys and opaque values.

  • Advantages:

    • Low latency

    • High throughput

Document Stores

  • Description: Stores JSON-like documents; allows for partial data retrieval and complex queries.

  • Use Case: Shopping cart in e-commerce.

Column Stores

  • Description: Efficiently sorts and compresses data in multiple levels of keys, enabling space optimization.

Graph Databases

  • Represents complex relationships through nodes, edges, and properties, ideal for navigating interconnected data sets.

Choosing Database Technologies

  • Consider All Factors: Including scale, consistency, availability, and requirements of applications.

  • CAP Theorem: Limits of distributed systems regarding consistency, availability, and partition tolerance.

Conclusion

  • Recap: Understanding of data management in the context of applied data science is vital. Students should now understand various storage and database types and their applications.

References for Further Reading

  • Microsoft Azure Storage Introduction: https://learn.microsoft.com/en-us/azure/storage/common/storage-introduction

  • NoSQL Database Survey: Gessert, F., et al., 2017

  • Consistency Trade-offs: Abadi, D., 2012

  • Real-world Communication Failures: Bailis, P., & Kingsbury, K., 2014

Thank You

  • Contact: bristol.ac.uk