Chapter 5:

Data Wrangling Process

  • Definition: Process of preparing source data for analysis.

  • Six Steps:

    • Step 1: Discovering - Familiarizing with source data.

    • Step 2: Structuring - Uniform formats for data features.

    • Step 3: Cleaning - Removing/replacing missing/outlier values.

    • Step 4: Enriching - Creating new features by combining existing data or appending new data.

    • Step 5: Validating - Ensuring dataset consistency and accuracy.

    • Step 6: Publishing - Making the dataset accessible for other users.

Comparison with ETL

  • ETL (Extract, Transform, Load):

    • Automated process for handling large data volumes.

    • Typically uses databases other than static datasets.

    • More structured compared to data wrangling.

  • Data Wrangling:

    • Informal and manual, better suited for small datasets.

Tools and Techniques

  • Pandas: Key library for data wrangling in Python.

    • DataFrame: Central data structure; consists of rows and columns.

    • Common methods: df.read_csv(), df.fillna(), df.dropna(), df.rename(), etc.

Data Cleaning Types

  • Missing Data: Represented as NaN, NA, None, etc.

  • Outliers: Values significantly outside the average, often > 2 or 3 standard deviations from the mean.

  • Duplication: Removing identical entries in a dataset.

Imputation Strategies

  • Mean Imputation: Replacing missing values with mean.

  • Regression Imputation: Using regression models to predict missing values.

  • Hot/Cold Deck Imputation: Randomly selecting values from similar data instances.

Data Enrichment Techniques

  • Appending data: Incorporating external datasets to augment existing ones.

  • Deriving new features: Creating metrics or categorical data from existing features (e.g., income per capita from total income and population).

Feature Scaling

  • Standardization: Centers features around the mean.

    • Resulting values called z-scores.

  • Normalization: Rescales features to a range of [0, 1].

Conclusion

  • Data wrangling is essential for turning raw data into a usable format for analysis, integrating multiple steps from cleaning to enrichment. Mastery over tools like Pandas is key for efficient data handling.