#2 (2.1-2.3) Chapter 2: Data Management

1) The Era of Big Data is Here, 2) Data Managment Systems, 3)Enterprise Data Architecture

What is Big Data?

Massive, high-velocity, and diverse datasets generated from digital sources (e.g., transactions, IoT sensors, social media) that require specialized tools, infrastructure, and algorithms to process and analyze.

What is Smart Data?

Data that has been cleaned, filtered, organized, and contextualized to provide actionable insights for strategic decision-making. Big Data becomes a business asset only when converted into Smart Data.

What are the 5 V's of Big Data?

Volume, Variety, Veracity, Velocity, and Value.

Define Volume in Big Data and its key challenge.

• Definition: Massive quantities of data generated and stored over time or per time unit.

• Challenge: Managing large storage capacity requirements and querying massive datasets efficiently.

Define Variety in Big Data and its key challenge.

• Definition: Diversity of data formats ranging from structured (databases) to unstructured (text, video, audio).

• Challenge: Integrating disparate data types into a unified, cohesive database or model.

Define Veracity in Big Data and its key challenge.

• Definition: The degree of trustworthiness, reliability, and precision of the data.

• Challenge: Handling messy data, missing values, measurement inconsistencies, and inaccuracies.

Define Velocity in Big Data and its key challenge.

• Definition: The speed at which new data is generated, transmitted, and processed in real time.

• Challenge: Systems can get inundated rapidly; requires fast infrastructure to enable real-time response strategies.

Define Value in Big Data and its key challenge.

• Definition: The degree to which extracted insights contribute tangible or intangible business benefits.

• Challenge: Requires aligning analytics questions directly with core organizational goals to ensure utility.

List the 9 computer storage measurement units in order from smallest to largest.

: Byte (B) → Kilobyte (KB) → Megabyte (MB) → Gigabyte (GB) → Terabyte (TB) → Petabyte (PB) → Exabyte (EB) → Zettabyte (ZB) → Yottabyte (YB).

Why is cross-functional collaboration necessary for data governance?

It breaks down functional silos. Marketing asks the strategic business questions, while IT and Data Science manage infrastructure and algorithms. Collaboration prevents local optimizations that lead to system-wide inefficiencies.

What is the difference between "Garbage In, Garbage Out" (GIGO) and "Garbage In, Hallelujah Out"?

• GIGO: Flawed input data inevitably leads to flawed analytics outputs regardless of model complexity.

• Garbage In, Hallelujah Out: Accepting flawed outputs as accurate simply because they were generated by complex, validated algorithms.

How should data quality issues like missing data and outliers be handled?

• Outliers: Evaluate before deleting; removing them blindly strips away insights, but leaving true errors skews predictions.

• Missing Data: Investigate why data is missing rather than blindly imputing values (e.g., substituting with the mean), which can introduce bias.

What is a Database?

A structured collection of operational company data organized for efficient access, retrieval, and analysis across functional departments.

What is a Database Management System (DBMS)?

The software framework used to create, maintain, manipulate, and query database structures and data.

What is a Database Schema?

The structural framework defining a database setup, including tables, columns, data types, relationships, primary/foreign keys, stored procedures, and validation rules.

What is a Relational Database?

: A DBMS that stores data in structured 2D tables (rows and columns) with predefined schemas, requiring SQL for querying and updating.

What is a Non-Relational (NoSQL) Database?

A flexible, schema-free database that stores structured or unstructured data vertically (e.g., key-value or document formats), making it ideal for dynamic data like social media or IoT feeds.

Compare Relational vs. Non-Relational (NoSQL) Databases on Flexibility and Querying.

• Relational: Inflexible (schema changes require modifying table structures); allows fast, complex drill-down queries using SQL.

• NoSQL: Highly flexible and scalable for dynamic data; makes complex drill-down queries more difficult.

What is the difference between Columns (Fields/Variables) and Rows (Records) in a relational table?

• Columns: Store specific attributes or properties describing an entity (e.g., region, average price).

• Rows: Individual instances or data points stored within the table, identified by unique keys.

What is a Primary Key?

A unique identifier assigned to each row/record in a table to ensure no duplicate entries exis

What is a Foreign Key?

A column (or set of columns) in one table that references the Primary Key of another table, linking records across tables.

What is Structured Query Language (SQL)?

A database management language used to join, select, manipulate, update, and retrieve data stored in relational databases.

Why do most modern companies use both Relational and NoSQL databases?

Relational databases are used for consistent, structured operational records (e.g., CRM, sales), while NoSQL databases handle high-velocity, dynamic data sources (e.g., mobile apps, social media).

What is Enterprise Data Architecture?

A structural framework that enables companies to systematically organize, integrate, store, and utilize data across the organization for operational and strategic decision-making.

What is Streaming Data?

The continuous, real-time transfer of data generated from numerous disparate sources in varying formats (e.g., mobile GPS logs, store sensors, web clickstreams).

What is Data Management?

The lifecycle management of data across an organization—from initial acquisition and verification to secure storage, access control, and ultimate disposal.

What does ETL stand for in data integration?

Extract, Transform, Load.

Explain the 3 steps of the ETL process.

• Extract: Pulls raw data from source systems (CRM, ERP, transaction systems, streaming feeds).

• Transform: Cleans, formats, and conforms data into a standardized structure for storage.

• Load: Deposits the transformed data into a storage repository (Data Warehouse, Data Mart, Data Lake).

What is Apache Hadoop, and how does it process big data?

An open-source, distributed software framework that uses MapReduce to divide big data workloads across networked computers, handling structured, semi-structured, and unstructured data at low cost.

What is MapReduce in Hadoop?

A two-step programming platform:

1. Map: Divides data into manageable subsets and distributes them across networked computers for processing.

2. Reduce: Combines the results from the individual nodes into a single final solution.

: What is HIVE?

A data warehouse infrastructure built on top of Hadoop that provides SQL-like querying capabilities across distributed file systems.

What is a Data Warehouse?

A central, highly structured repository containing enterprise-wide historical data organized under a single schema for high-speed querying and strategic reporting.

What is a Data Mart, and what are its two types?

A domain-specific subset of data focused on a specific department (e.g., Marketing).

• Dependent Data Mart: Sourced directly from an Enterprise Data Warehouse.

• Independent Data Mart: Sourced directly from operational systems or external providers.

What is a Data Lake?

A flexible storage repository that holds massive amounts of raw data in its native format (structured, semi-structured HTML, unstructured video/audio, machine logs) for fast exploration and archiving without pre-load delays.

What is Data Democratization?

The practice of breaking down departmental "data silos" to provide broad, secure access to unified data across functional teams, empowering employees company-wide to make data-driven decisions.