CH3- Data Management & Data Quality – Key Notes

Page $1$

• Course: Introduction to Data Analytics and Applications

• Chapter focus: Data Management and Data Quality

• Prepared by Miss Noor Assyikin Binti Alias, focusing on foundational concepts crucial for effective data utilization.

Page $2$

• Agenda covers essential stages of data processing: Data Preprocessing, Data Mining, Data Sampling, and Data Sub-setting. Each step contributes to transforming raw data into actionable insights.

Page $3$

• Data preprocessing is the critical first step that transforms raw, often messy, data into a clean, structured, and usable format.

• Its primary goal is to ensure high data quality before any further analysis or modeling is performed, as the quality of input directly impacts the reliability of output.

• This multi-stage process typically involves four key activities: cleaning (handling errors and inconsistencies), integration (combining data from various sources), transformation (converting data into suitable formats), and reduction (decreasing data volume while retaining information).

Page $4$

• Real-world data is inherently complex and often suffers from various quality issues:

  • Incomplete data: Missing values (e.g., a customer's age is not recorded), unavailable attributes (a column was never captured).

  • Noisy data: Contains errors or outliers (e.g., incorrect data entry like a negative age or a sensor malfunction leading to extreme readings).

  • Inconsistent data: Discrepancies in naming conventions or formats across different data sources (e.g., "New York" vs. "NY"; date formats like "MM/DD/YYYY" vs. "DD-MM-YY").

  • Causes: Recording errors during data collection, accidental or intentional data deletions, and a lack of proper historical data tracking.

Page $5$

• Data preprocessing is a mandatory step that systematically occurs during the initial data preparation stage of any data analysis or machine learning project.

• This thorough preparation is crucial to produce reliable, accurate, and unbiased mining results, preventing the propagation of errors from raw data into insights.

Page $6$

• The key data processing pipeline typically involves several sequential stages:

  • Load data: Gathering raw data from various sources into a working environment.

  • Clean/integrate/transform/reduce: Applying preprocessing techniques to enhance data quality and usability.

  • Split train–test: Dividing the processed dataset into subsets for model training and evaluation.

  • Model training & evaluation: Building and assessing predictive or descriptive models using the prepared data.

Page $7$

• The main steps of data preprocessing are:

  • Cleaning: Addressing quality issues (missing values, duplicates, errors).

  • Integration: Combining data from disparate sources into a unified view.

  • Selection/Reduction: Identifying and retaining only the most relevant features or samples.

  • Transformation: Converting data into appropriate formats for specific algorithms.

• These techniques are often not mutually exclusive and can be strategically combined or iterated upon for optimal data quality and performance.

Page $8$

• Data cleaning specifically focuses on improving the quality of data by addressing common imperfections:

  • Handling missing values: Strategies include removing rows or columns with excessive missing data, or imputing (filling in) missing values using statistical measures like the mean, median, or mode, or more advanced methods like regression imputation.

  • Detecting and correcting duplicates: Identifying and removing redundant records to prevent skewed analysis.

  • Resolving inconsistencies: Standardizing entries (e.g., converting all state abbreviations to full names) and correcting formatting errors (e.g., ensuring all dates follow a consistent YYYY-MM-DD pattern).

  • Managing outliers: Identifying and sometimes treating extreme data points that deviate significantly from other observations, as they can disproportionately influence model training.

Page $9$

• The output of comprehensive data cleaning is clean data, which is fundamental for several reasons:

  • It directly prevents biased analysis by ensuring that erroneous or inconsistent data doesn't skew statistical inferences or model parameters.

  • It strongly supports accurate modeling outcomes, as machine learning algorithms perform optimally on high-quality, consistent data.

Page $10$

• Data integration involves combining data from multiple, disparate sources into a cohesive and unified dataset.

• This process often presents significant challenges:

  • Entity matching: Identifying and linking records that refer to the same real-world entity but are represented differently across sources (e.g., matching customer records from a sales database and a marketing database).

  • Resolving differing value formats: Standardizing data types and formats that vary across sources (e.g., merging financial data where one source uses USD and another uses a local currency, or differing date formats like "$01/25/2023$ vs. "$25-Jan-2023$").

Page $11$

• Achieving proper alignment and consistency across all integrated sources is paramount, especially when dealing with large, diverse datasets.

• This ensures that data can be correctly joined, analyzed, and modeled without introducing errors or misinterpretations arising from structural or semantic differences.

Page $12$

• Data selection (or feature selection) is the process of choosing only the most relevant attributes or data points from the dataset.

• The goal is to enhance efficiency by reducing dimensionality and improve accuracy by focusing on informative features.

• Data can be broadly categorized into:

  • Quantitative data (numerical): Measurable quantities, such as age ($e.g., 30$ years), income ($e.g., \$50,000$), or temperature ($e.g., 25^{\circ}C$).

  • Qualitative data (categorical): Descriptive information that can be observed but not measured numerically, including text (e.g., customer reviews), images, audio recordings, and video files.

Page $13$

• The benefits of effective data selection and reduction techniques are substantial:

  • Faster computation: Reduced dataset size means algorithms run more quickly, especially crucial for large datasets.

  • Higher model accuracy: By removing irrelevant or redundant features, models can focus on important patterns, reducing noise and overfitting.

  • Improved data quality indirectly: Focusing on relevant data can highlight existing quality issues within that subset.

  • Simpler interpretation: Models built on fewer, more meaningful features are often easier to understand and explain.

Page $14$

• Practical examples of data selection:

  • In credit risk assessment, focusing solely on financial variables like income, credit score, and loan history, rather than irrelevant demographic data, can streamline analysis.

  • For marketing campaign analysis, focusing only on customer interaction data from the most recent promotion (e.g., last $3$ months) can provide more actionable insights than analyzing all historical data.

Page $15$

• Data transformation is the process of converting raw data into a suitable format or structure compatible with specific analytical algorithms or desired outcomes.

• Common transformation techniques include:

  • Normalization: Scaling numerical features to a standard range (e.g., Min-Max normalization scales values between $0$ and $1$) to prevent features with larger values from dominating those with smaller values.

  • Scaling: Adjusting the range of features (e.g., Z-score standardization transforms data to have a mean of $0$ and a standard deviation of $1$), which is crucial for distance-based algorithms.

  • Aggregation: Summarizing data (e.g., calculating daily sales from hourly transactions).

  • Discretization: Dividing continuous attributes into intervals or bins (e.g., converting age into age groups).

  • Attribute construction: Creating new attributes from existing ones (e.g., calculating Body Mass Index from height and weight).

• This ensures compatibility with various algorithms (many algorithms assume data is normalized or scaled) and effectively reduces computational complexity.

Page $16$

• The outcomes of effective data transformation include:

  • Better accuracy: Many machine learning algorithms perform better when data is properly scaled or transformed.

  • Lower dimensionality: Through techniques like PCA (Principal Component Analysis), data can be transformed into a lower-dimensional space while retaining most of its variance.

  • Faster processing: Transformed data can streamline computations.

  • Easier interpretation: Transforming complex data into more intuitive formats can aid in understanding.

Page $17$

• Data mining is the process of discovering meaningful patterns, relationships, and insights from large datasets, utilizing a blend of statistical methods, artificial intelligence, and machine learning algorithms.

• Key data mining tasks commonly performed include:

  • Association: Discovering relationships between variables (e.g., market basket analysis: "customers who buy bread also buy milk").

  • Classification: Categorizing data into predefined classes (e.g., spam detection, predicting customer churn).

  • Prediction (Regression): Forecasting continuous values (e.g., predicting house prices, stock values).

  • Clustering: Grouping similar data points together based on their inherent characteristics without predefined classes (e.g., customer segmentation).

  • Time-series analysis: Analyzing data points collected over time to identify trends, cycles, or forecast future values (e.g., predicting sales over time).

Page $18$

• Data mining possesses several distinct characteristics:

  • Automated/Semi-automated: It often involves sophisticated algorithms that can automatically identify patterns, though human input and iterative refinement are usually required.

  • Extracts hidden information: Unlike simple queries, it uncovers non-obvious patterns and insights that are not immediately apparent.

  • Handles big data: Designed to process and derive insights from massive and complex datasets that traditional methods struggle with.

  • Supports predictive & descriptive goals: Can be used to forecast future trends (predictive) or to understand past behavior and relationships (descriptive).

Page $19$

• The typical data mining process flow involves:

  • Mining: Applying algorithms to the prepared data to discover patterns.

  • Pattern evaluation: Assessing the significance and interestingness of the discovered patterns (e.g., using statistical measures, domain knowledge).

  • Knowledge representation: Presenting the extracted knowledge in an understandable and actionable format, often through visual aids such as charts, graphs, dashboards, or reports.

Page $20$

• Data mining has diverse applications across various industries:

  • Business/Marketing:

  • Market segmentation: Grouping customers with similar traits for targeted campaigns.

  • Product recommendations: Suggesting products based on past purchases (e.g., "customers who bought this also bought…").

  • Fraud detection: Identifying unusual patterns in transactions that may indicate fraudulent activity.

  • Finance:

  • Credit scoring: Assessing the creditworthiness of loan applicants.

  • Risk management: Predicting financial risks and optimizing investment portfolios.

  • Stock market analysis: Forecasting stock prices and identifying trading opportunities.

Page $21$

• More applications of data mining:

  • Healthcare:

  • Disease prediction: Identifying individuals at high risk of certain diseases based on their medical history and lifestyle.

  • Drug discovery: Analyzing molecular data to identify potential new drugs.

  • Outcome analysis: Evaluating the effectiveness of treatments and interventions.

  • Retail:

  • Market-basket analysis: Understanding product co-occurrence in customer purchases.

  • Inventory demand forecasting: Predicting future product demand to optimize stock levels and supply chains.

Page $22$

• Despite its power, data mining faces significant challenges:

  • Data quality: The "Garbage In, Garbage Out" principle applies directly; poor input data leads to unreliable insights.

  • Scalability: Processing and analyzing ever-increasing volumes of big data efficiently can be computationally intensive.

  • Privacy and security: Handling sensitive personal or proprietary information raises ethical and legal concerns regarding data privacy and protection.

  • Interpretability: Complex models (e.g., deep learning) can be difficult to interpret, making it challenging to understand why a particular prediction or classification was made, which can hinder trust and adoption.

Page $23$

• Distinguishing between data preprocessing and data mining:

  • Preprocessing vs. Mining: Preprocessing is the prior stage focused on data preparedness and quality improvement; mining is the subsequent stage focused on extracting actionable insights and patterns.

  • Preprocessing handles issues: It specifically addresses raw data problems like missing values, inconsistencies, and noise.

  • Mining involves algorithms and interpretation: It involves selecting appropriate algorithms (e.g., classification, clustering) and interpreting the patterns they uncover to generate knowledge.

Page $24$

• Data sampling is a technique used to select a representative subset (or sample) from a larger dataset or population. Its primary purpose is to reduce the volume of data to be processed while ensuring that the chosen subset accurately reflects the characteristics of the entire population.

• This is often done to lower computational costs, save time, and manage resources, especially with very large datasets.

• Example: Conducting a survey of only $1,000$ people to understand the opinions of an entire city population, rather than surveying every single resident.

Page $25$

• Sampling methodologies are broadly categorized into two main types:

  • Probability Sampling: Every item or individual in the population has a known, non-zero chance of being selected. This method ensures statistical representativeness and allows for generalization of findings to the larger population.

  • Non-Probability Sampling: Selection is based on non-random criteria, such as convenience or researcher judgment. This method is often quicker and less expensive but may not produce statistically representative samples, limiting generalizability.

Page $26$

• Probability Sampling Methods:

  • Simple Random Sampling: Every item or individual in the population has an exactly equal chance of being selected for the sample. This is often done using random number generators.

  • Stratified Random Sampling: The population is divided into distinct subgroups (strata) based on shared characteristics (e.g., age groups, gender, income levels).

  • Samples are then drawn proportionally (or disproportionately, depending on the research goal) from each stratum to ensure representation of specific subgroups.

Page $27$

• More Probability Sampling Methods:

  • Cluster Sampling: The population is divided into clusters (e.g., geographical areas, schools). A random sample of these clusters is selected, and all individuals within the chosen clusters are included in the sample.

  • Multistage Sampling: An extension of cluster sampling, where instead of surveying all individuals within selected clusters, further sampling is performed within those clusters at multiple levels.

Page $28$

• An illustration of Multistage Sampling:

  • Imagine surveying opinions across a country: initially, a random sample of countries is selected.

  • From the chosen countries, a random sample of regions is selected.

  • Within those regions, a sample of cities is chosen.

  • Finally, within the selected cities, a sample of individual participants is randomly chosen for the survey.

Page $29$

• Another Probability Sampling Method:

  • Systematic Sampling: Items are selected from the population at regular intervals, often after a random starting point. For example, selecting every $k^{th}$ item from an ordered list (e.g., every tenth customer record from a database), where $k$ is the sampling interval.

Page $30$

• Non-Probability Sampling Methods:

  • Convenience Sampling: Participants are selected based on their easy accessibility and proximity to the researcher. This is often used for preliminary research due to its speed and cost-effectiveness, but it carries a high risk of bias.

  • Consecutive Sampling: Involves including all subjects who meet the inclusion criteria and are readily available until a predetermined sample size is met. Similar to convenience sampling but aims to capture all available subjects over a specific period.

Page $31$

• More Non-Probability Sampling Methods:

  • Purposive (Judgmental) Sampling: The researcher intentionally selects participants based on their expert judgment and specific criteria relevant to the study's objectives. This is useful for niche studies where specific expertise or characteristics are required.

  • Quota Sampling: Similar to stratified sampling but non-random. The researcher sets quotas for different subgroups (e.g., $50$ men, $50$ women) and then uses non-random methods (like convenience sampling) to fill those quotas until the desired number of participants from each subgroup is reached.

Page $32$

• Key issues and challenges associated with data sampling:

  • Information loss: Reducing the dataset size inherently means some original detail or nuance from the full dataset might be lost, potentially affecting the richness of analysis.

  • Bias: If the sample is not truly random or representative, it can introduce systematic errors, leading to skewed or misleading results that do not accurately reflect the overall population.

  • Imbalance: Particularly critical in classification tasks, an imbalanced sample may have a disproportionate representation of majority vs. minority classes, which can lead to models that perform poorly on the under-represented class (e.g., fraud detection where fraudulent transactions are very rare).

Page $33$

• Data sub-setting refers to the process of extracting a smaller, specifically relevant portion of a larger dataset for focused analysis or particular tasks.

• Unlike sampling, which aims for statistical representation of the whole, sub-setting is about focusing on a specific segment that meets certain criteria, thereby reducing complexity and required computational resources for that specific task.

Page $34$

• The primary purposes and benefits of data sub-setting include:

  • Faster processing_: Analyzing only a relevant subset dramatically reduces the computational time required for queries, model training, or visualization.

  • Focus on relevant data: Allows analysts to concentrate their efforts and insights on specific segments or periods that are most pertinent to a particular business question or problem.

  • Simplified analysis: Working with smaller, more focused datasets can make complex analyses more manageable and less prone to errors.

  • Resource optimization: Reduces memory usage, disk I/O, and CPU cycles, especially beneficial in environments with limited computational resources.

Page $35$

• Common scenarios where data sub-setting is applied:

  • Demographic targeting: Analyzing only data for customers within a specific age range or geographic region for targeted marketing campaigns.

  • Fraud window analysis: Examining transaction data only within a specific high-risk time window (e.g., $24$ hours after an unusual login) to detect fraudulent activities.

  • Specific patient groups: Studying medical data exclusively for patients with a particular condition (e.g., diabetes) or those who received a specific treatment.

  • Holiday sales analysis: Focusing entirely on sales data from the Black Friday to Cyber Monday period to understand peak purchasing behavior.

Page $36$

• Trivia: The financial implications of poor data quality are significant, with estimates suggesting it costs the global economy approximately $3.1$ trillion USD annually.

• This underscores the critical principle of "Garbage In, Garbage Out" (GIGO), meaning the quality of output is only as good as the quality of input data.

• In the healthcare sector alone, poor data quality results in estimated losses of around $17$ billion USD annually, highlighting its pervasive impact across industries.

Page $37$

• End of content – gratitude expressed for engagement and attention to data management principles.