#3 (2.4-2.5) Chapter 2: Data Management

2.4 Data Quality, 2.5 Data Understanding, Preparation, and Transformation

What is Data Quality?

The degree to which a dataset meets requirements for timeliness, completeness, accuracy, consistency, and format.

What is the fundamental principle of "Garbage In, Garbage Out" (GIGO)?

Poor-quality or erroneous input data inevitably leads to poor-quality decisions, flawed analytics, and unreliable strategic insights.

What are the 5 core dimensions of Data Quality?

Timeliness, Completeness, Accuracy, Consistency, and Format.

Define Timeliness as a data quality dimension and give its risk.

• Definition: Data is accessible and up-to-date when needed for decision-making.

• Risk: Delays prevent real-time responses to customer behavior, social media sentiment, or competitor pricing shifts.

Define Completeness as a data quality dimension and give its risk.

• Definition: Maintaining the necessary breadth, depth, and scope of data for a specific task.

• Risk: Missing fields (e.g., email, name) prevent the execution of targeted campaigns or complete customer analysis.

: Define Accuracy as a data quality dimension and give its risk.

• Definition: The extent to which data is error-free, reliable, and precisely measured.

• Risk: Human input errors, bad measurements, or corrupted feeds lead to flawed strategic choices.

Define Consistency as a data quality dimension and give its risk.

• Definition: Standardized values and definitions across different functional departments, systems, or databases.

• Risk: Inconsistent definitions lead to integration confusion, uncoordinated operations, and improper billing/pricing reconciliation.

Define Format as a data quality dimension and give its risk.

• Definition: The extent to which information is adequately presented or delivered for efficient understanding.

• Risk: Clumsy or hard-to-read dashboards obscure critical data, leading to operational failures and oversights.

How do human input errors like misspelled names affect customer data?

They create duplicate accounts for the same individual (e.g., Jonh Smith vs. John Smith), splitting transaction history across CRM records and distorting customer value metrics.

What is Data Understanding?

The process of thoroughly exploring and verifying available dataset fields before analysis to ensure correct business problem alignment and prevent erroneous reporting.

What is Overfitting, and what causes it?

• Definition: When an overly complex model captures noise or sample-specific spurious effects rather than true relationships.

• Cause: Including too many unnecessary, highly correlated, or irrelevant features (variables).

• Consequence: Reduces model generalizability to new, unseen data.

What is a Statistical Power Calculation?

A statistical method used to determine the minimum sample size required to estimate outcomes with a sufficient level of precision.

What sample size is typically needed for predictive modeling vs. estimating effect size?

Predictive modeling typically requires a large sample of several thousand records to accurately forecast new outcomes, whereas estimating effect sizes can be done with smaller, representative samples.

What is a Unit of Analysis?

he primary entity being analyzed (the "what, when, and who"), such as a customer (for churn), a brand (for satisfaction), or a subscription (for plan pricing).

What are the 3 options for handling missing values in a dataset?

1. Imputation: Replacing missing entries with estimated values (mean, median, mode, or regression/decision tree predictions).

2. Record Omission: Removing entire incomplete rows/observations.

3. Variable Exclusion: Dropping an entire column/feature if missingness is widespread and the variable is non-essential.

: When is mean, median, or mode imputation recommended for missing data?

When values are missing completely at random and represent a very small portion of the overall dataset.

What is the statistical rule of thumb for identifying an outlier?

Any observation that falls more than 3 standard deviations away from the mean or at a considerable distance from other clusters in cluster analysis.

When should outliers be removed versus retained?

• Remove: When outliers are data entry errors or noise (e.g., age = 140).

• Retain: When rare extreme events are the actual target of the analysis (e.g., fraud detection models).

What is Data Aggregation?

Applying summary operations to data (e.g., summing weekly sales into monthly totals) to align granular raw data with the required unit of analysis.

What is Data Normalization and how is it calculated?

Rescaling variables to bring them into the same range by subtracting the mean and dividing by the standard deviation (z=σx−μ​). It prevents high-range variables from dominating distance calculations in techniques like cluster analysis.

What is Feature Construction?

Creating new variables derived from existing fields (e.g., extracting day of the week, month, or season from a sales timestamp) to uncover underlying predictive patterns.

What is Dummy Coding?

Re-coding nominal categorical (nonmetric) variables (e.g., regions like Northeast, Southwest) into binary (0 or 1) indicator variables for statistical modeling.