Data Mining and Machine Learning Notes
Fundamentals of Data, Information, and Knowledge
Data: Elementary descriptions of things, events, activities, and transactions that are recorded, classified, and stored but not organized to convey any specific meaning. Serves as the most basic unit of observation (e.g., raw sensor logs, numeric values, or transaction timestamps).
Information: Data that has been organized, structured, or processed in computer files or relational structures so that it possesses context, relevance, and operational purpose for a user.
Knowledge: Processed and contextualized information that conveys actionable understanding, synthesized learning, integrated expertise, and analytical insights applicable to solving complex real-world problems or driving organizational activities.
Perspectives and Classifications of Data
Business Perspective of Data:
Captured, stored, and managed by an enterprise as a high-value digital asset to support executive decision-making, strategic planning, competitive positioning, and daily operational workflows.
Often originates and exists in a raw, fragmented, and unorganized state prior to analytical processing.
Technical Perspective of Data:
Formatted sequences of symbols, signals, bits, or bytes that act as structured inputs for computational and machine processes, yielding valuable information as analytical output.
Business Data Characteristics and Categories:
Stored in enterprise databases and content repositories that are either directly machine-readable or rendered into formats intended for human consumption (e.g., user interfaces, dashboards, structured documents, reports).
Key Business Data Types:
Audit trail: Sequential, time-stamped logs tracking system access, user activities, and data modifications for security and compliance.
Dark data: Operational data collected and stored during business activities that remains unanalyzed and unutilized.
Machine data: Automated telemetry and logs generated directly by hardware systems, smart devices, and applications without human intervention.
Master data: Core enterprise entities shared across an organization (e.g., customers, products, suppliers, employees).
Reference data: Permissible values used to classify or characterize other data elements (e.g., country codes, transaction currencies, status flags).
Transactional data: High-volume records detailing commercial exchanges or business events (e.g., point-of-sale receipts, web purchases, wire transfers).
Major Categories in an Organizational Frame:
Quantitative Data: Measurable numeric values that quantify counts, amounts, lengths, or monetary figures.
Qualitative Data: Non-numeric, descriptive attributes capturing qualities, categories, opinions, or textual feedback.
Classification based on Time Dimension:
Cross-sectional Data: Observations collected across multiple subjects or entities at a single, specific point in time.
Time Series Data: Sequential measurements recorded for a single entity or set of variables at regular, successive time intervals.
Classification based on Structural Complexity:
Structured: Rigorously organized in fixed schemas and relational tables with predefined rows and columns (e.g., Point of Sale [POS] transaction logs, financial ledgers, structured customer databases).
Unstructured: Expressed without a predefined conceptual model or relational schema (e.g., freeform text files, video clips, audio recordings, photographs).
Semi-structured: Contains organizational markers, metadata, or self-describing tags without conforming to rigid row/column boundaries (e.g., XML files, JSON logs, HTML documents).
Classification based on Data Source:
Internal Data: Sourced directly from systems within organizational boundaries (e.g., historical sales records, enterprise CRM logs, internal financial accounts).
External Data: Acquired from third-party or public sources outside the organization (e.g., social media activity streams, macroeconomic indicators, government census data, spatial/geographic maps).
Principles and Motivation of Data Mining
Commercial Drivers for Data Mining:
Explosive Data Accumulation: Massive warehousing of modern datasets across e-commerce transactions, retail point-of-sale registers, banking/credit card operations, automated teller machine (ATM) logs, and web logs.
Hardware Advancements: Exponential power expansion of parallel processing hardware coupled with steadily decreasing storage costs.
Competitive Market Pressures: Fierce market competition demanding customized services, churn prevention, and superior Customer Relationship Management (CRM).
Core Definitions of Data Mining:
The computational extraction of implicit, previously unknown, non-trivial, and potentially useful information or patterns from massive data repositories.
The exploration and systematic analysis, by automatic or semi-automatic algorithm execution, of vast quantities of data to discover meaningful structural relationships.
Equivalently termed Knowledge Discovery in Databases (KDD): The non-trivial process of identifying valid, novel, useful, and ultimately understandable patterns in large databases.
Alternative Terminology: Knowledge extraction, data/pattern analysis, data archeology, data dredging, information harvesting, and Business Intelligence (BI).
Distinctions — What is NOT Data Mining:
Standard deductive database querying (e.g., looking up a subscriber's phone number in an electronic directory, querying a web search engine for direct keywords like "Amazon").
Static rule-based expert systems or standard descriptive statistical calculations (e.g., computing a basic mean or variance).
Valid Data Mining Examples:
Discovering geographic distribution clusters of specific surnames (e.g., identifying a statistically significant concentration of names like O'Brien, O'Rourke, and O'Reilly in the Boston area).
Contextual grouping/clustering of search engine results based on semantics (e.g., automatically segregating search results into distinct context clusters such as the Amazon rainforest versus Amazon.com).
Origins, Disciplines, and Architecture of Data Mining
Origins and Technological Foundations:
Data mining represents an interdisciplinary confluence of multiple computational fields: Machine Learning/AI, Pattern Recognition, Statistics, Database Systems, Information Science, and Data Visualization.
Failure of Traditional Analytical Techniques:
Conventional manual methods or basic statistical packages fail when applied to modern enterprise data due to:
Enormity: Scale and volume reaching terabytes or petabytes.
High Dimensionality: Hundreds or thousands of attribute variables per instance.
Heterogeneity: Complex, non-standard, distributed, and unstandardized data formats.
Dimensions of Decision-Making in Data Mining:
Databases Mined: Relational, transactional, object-oriented, active, spatial, time-series, text, multimedia, legacy, and web environments.
Knowledge Types Mined: Characterization, discrimination, association rules, classification, clustering, trend analysis, deviation analysis, and outlier detection.
Techniques Utilized: Database-oriented strategies, Online Analytical Processing (OLAP/MDA), machine learning, statistical modeling, data visualization, and neural networks.
Application Domains: Retail target marketing, telecommunications network management, banking credit risk analysis, healthcare analytics, DNA sequence mining, stock market prediction, and weblog stream analysis.
Core Tasks in Data Mining
Prediction Tasks:
Utilize explanatory input variables to predict unknown, missing, or future values of target variables.
Key Predictive Tasks:
Classification: Maps observations into discrete, predefined category labels.
Regression: Estimates continuous numeric target values.
Deviation / Outlier Detection: Identifies instances that stray significantly from expected normative behaviors.
Description Tasks:
Discover human-interpretable patterns and structural relationships that describe the intrinsic properties of the underlying dataset.
Key Descriptive Tasks:
Clustering: Groups unlabeled observations based on distance or feature similarity.
Association Rule Discovery: Uncovers co-occurrence relationships between attributes (e.g., market basket analysis).
Sequential Pattern Discovery: Identifies recurring temporal sequences or chronological event series.
The Knowledge Discovery in Databases (KDD) Process
Data mining serves as the central analytical core of the comprehensive KDD framework.
Sequential Steps in the KDD Process:
Learning the Application Domain: Understanding relevant domain knowledge, prioritizing prior constraints, and defining project goals.
Creating a Target Data Set: Selecting data subsets from raw operational databases or enterprise data warehouses.
Data Cleaning and Preprocessing: Consumes up to of total analytical effort. Handles noise reduction, missing value strategy execution, and inconsistency removal.
Data Reduction and Transformation: Executing feature selection, dimensionality reduction, discretization, and attribute normalization.
Choosing Mining Functions: Establishing whether descriptive or predictive objectives apply (e.g., summarization, classification, regression, association, clustering).
Choosing Mining Algorithms: Selecting specific algorithmic models aligned with data characteristics and task goals.
Data Mining Execution: Running search algorithms to discover empirical patterns of interest.
Pattern Evaluation and Knowledge Presentation: Interpreting extracted patterns using visualization tools, removing trivial or redundant rules, and filtering by interestingness.
Use of Discovered Knowledge: Integrating discovered patterns into decision support systems, operational workflows, or corporate strategy.
Business Intelligence Pyramid
A hierarchical view demonstrating how data processing transforms raw assets into executive decision potential:
Making Decisions (End User Focus — Highest business value potential)
Data Presentation / Visualization Techniques (Business Analyst Focus)
Data Mining / Information Discovery (Data Analyst Focus)
Data Exploration / Statistical Analysis, Querying, Reporting (Data Analyst Focus)
Data Warehouses / Data Marts / OLAP, MDA (Database Administrator Focus)
Data Sources (Paper records, files, transactional databases [OLTP] — Database Administrator Focus)
Major Issues and Challenges in Data Mining
Mining Methodology and User Interaction:
Extracting diverse knowledge types across multiple abstraction levels iteratively.
Incorporating expert background knowledge directly into algorithmic search procedures.
Formulating expressive, ad-hoc data mining query languages.
Resolving the "interestingness problem" (filtering out trivial or unhelpful patterns).
Performance and Scalability:
Algorithmic efficiency and time/space scalability relative to data volume growth.
Developing parallel, distributed, and incremental mining algorithms for real-time streaming data.
Diversity of Complex Data Types:
Managing spatial-temporal data, multimedia formats, non-relational graphs, and heterogeneous global web structures.
Social Impacts and Operational Integration:
Knowledge Fusion: Merging newly discovered patterns seamlessly with existing knowledge bases.
Safeguarding data security, intellectual property, and individual personal privacy.
Machine Learning Taxonomy and Learning Paradigms
Machine Learning Definition: The simulation of human intelligence by machines, enabling computational systems to autonomously learn, adjust parameters, and improve performance from data experience without explicit rule programming.
Taxonomic Hierarchy: \n\text{Artificial Intelligence} \supset \text{Machine Learning} \supset \text{Deep Learning} \supset \text{Generative AI}\n
Three Core Machine Learning Paradigms:
Supervised Learning: Model training executed on a fully labeled dataset where every feature vector is paired with a ground-truth response label (e.g., training a visual classifier using image instances explicitly labeled as "CAT").
Unsupervised Learning: Model training conducted on unlabeled data instances to discover intrinsic groupings, underlying manifold structures, or cluster boundaries based on feature similarities.
Reinforcement Learning: Algorithmic policy optimization where an autonomous agent interacts with a dynamic environment, receiving environmental reward or penalty feedback to learn an optimal action trajectory.
Detailed Problem Categories in Machine Learning
Classification vs. Clustering:
Classification (Supervised): Assigns incoming inputs into predefined, discrete category labels (e.g., explicitly categorizing animal images into discrete labels like hen, rabbit, or turkey).
Clustering (Unsupervised): Partitions unlabeled inputs into groupings based on mathematical distance or metric similarity (e.g., naturally separating animal instances into mammals versus birds without prior labels).
Regression Analysis:
Definition: Fitting a mathematical function, line, or surface through continuous data points to model relationships between predictor attributes and continuous numeric outcomes.
Mathematical Terminology:
: Feature, covariate, independent variable, or predictor attribute (e.g., property area in square feet, ).
: Observation, dependent variable, or continuous response (e.g., market price in dollars, \).
Limitations of Simple Matching and Binning Strategies:
Exact Feature Matching: Direct table lookup fails because unobserved future samples rarely share identical predictor values .
Local Interval Range Averaging: Computing local target averages discards historical observations outside the narrow interval and remains vulnerable to sparse local sample sizes.
Linear Regression Model:
Single Variable Formula: \nf(x) = w_0 + w_1 x\n
represents the -intercept parameter.
represents the slope parameter.
Multivariate Linear Regression Formula: \nf_{\mathbf{w}}(x) = w_0 + w_1 x_1 + w_2 x_2 + \dots + w_d x_d\n
Example feature setup: , .
Model Order, Complexity, and Overfitting:
Linear models underfit if real-world physical relationships are non-linear (e.g., quadratic curves).
High-order polynomial models can minimize the training Residual Sum of Squares (RSS) to zero, but suffer severe overfitting and fail to generalize on unseen evaluation data.
Model Evaluation via Data Splitting:
Data is partitioned into a Training Set (used to fit parameters) and a Test Set (heldout data used to evaluate generalization performance).
Real-World Regression Applications:
Stock Market Valuation: Predicting continuous future share prices using covariates such as historical trading volume, price momentum, news sentiment scores, and commodity market indices.
Social Engagement (Tweet Popularity): Predicting retweet counts based on total follower count, second-degree network reach, text linguistic features, hashtag trending scores, and historic virality metrics.
Algorithmic Implementations in Machine Learning
Supervised Classification Workflow:
Define Problem: Establish objective (e.g., binary discrimination between dogs and cats).
Find Data: Collect representative labeled datasets.
Clean Data: Remove noisy, corrupted, or mislabelled records.
Choose Model: Select algorithm architecture.
Train Model: Fit parameters on clean training data.
Logistic Regression (Binary Classification):
Advantages:
Makes no restrictive distributional assumptions regarding class locations in feature space.
Extends to multiclass scenarios via multinomial logistic regression.
Provides explicit probabilistic output predictions.
Fast training computation and rapid evaluation speed.
High accuracy on linearly separable datasets.
Resistant to overfitting when regularized.
Disadvantages:
Inability to directly model continuous target variables.
Fails when independent predictors exhibit negligible correlation with the binary target.
Requires large sample sizes to reach parameter stability.
Decision Trees:
Partitions feature space into logical, rule-based splits without requiring underlying numeric distance metrics.
Executes recursive logical conditions based on feature properties (e.g., evaluating boolean pathways such as "Is color red?" or "Is text underlined?").
Random Forest Ensembles:
Constructs an ensemble of individual decision trees trained on randomized feature sub-samples.
Combines individual decision tree predictions using Majority Voting to produce the final classification, significantly reducing model variance and overfitting risk.
Data Preprocessing and Feature Engineering
Data Analysis Pipeline Sequence: \n\text{Data Preprocessing} \longrightarrow \text{Data Mining Core} \longrightarrow \text{Result Post-Processing}\n
Pipeline Stage Details:
Preprocessing: Required because real-world raw data is inherently noisy, incomplete, and inconsistent. Key techniques include Sampling, Dimensionality Reduction, and Feature Subset Selection.
Post-Processing: Converts raw model outputs into actionable insights via statistical significance testing and visualization.
Attribute Fundamentals:
Attribute Values: Symbolic or numeric values assigned to an attribute.
Mapping Dynamics:
The same underlying attribute can be represented by different attribute values (e.g., measuring height in meters versus feet).
Distinct attributes can use the same numerical range while maintaining different logical properties (e.g., ID numbers and age values both exist as integers, but ID values have no upper bound while age values are bounded by human lifespans).
Attribute Types:
Discrete Attribute: Possesses a finite or countably infinite set of values (e.g., postal zip codes, word counts, document term frequencies). Typically stored as integer variables.
Continuous Attribute: Represented by real numbers (e.g., temperature, height, weight). Digital storage relies on finite floating-point precision.
Data Quality Issues:
Noise: Random modification distortion of original values (e.g., acoustic distortion during voice signal processing).
Outliers: Data instances exhibiting traits that deviate significantly from the general statistical distribution of the dataset.
Missing Values:
Causes: Omission during collection (e.g., survey non-response) or non-applicability (e.g., income attributes for minors).
Remediation Methods: Dropping instances, imputing/estimating missing entries, ignoring missing components during specific calculations, or replacing with probability-weighted candidate values.
Duplicate Data: Presence of identical or nearly-identical records (e.g., a single customer possessing multiple registered email addresses across merged databases). Addressed through data deduplication.
Techniques for Handling Noise:
Binning: Sorting continuous data and partitioning into local bins. Smoothed using bin means, bin medians, or boundary ranges.
Equal-Width Binning: Divides total attribute range into intervals of equal distance. Simple to implement, but vulnerable to outlier distortions.
Equal-Depth (Equal-Frequency) Binning: Partitions data into intervals such that each bin contains an equal count of records. Effectively handles skewed data distributions.
Regression Smoothing: Fits a linear or polynomial function to smooth out variations or predict missing entries (e.g., ).
Clustering: Isolates and removes spatial outliers.
Combined Machine-Human Inspection: Automated detection flags suspicious observations for expert human verification.
Data Transformation and Feature Selection
Feature Scaling Motivations:
Neural Networks: Initial weight matrices are initialized randomly near zero; convergence improves when features share uniform scales.
Regularized Linear Models: Techniques like Ridge and Lasso regression rely on scaled features to apply penalties uniformly.
Unsupervised Learning: Dimensionality reduction algorithms (such as Principal Component Analysis [PCA]) are sensitive to relative feature scales.
Mathematical Normalization Formulations:
Min-Max Normalization: \nv' = \frac{v - \min}{\max - \min} (\text{new\_max} - \text{new\_min}) + \text{new\_min}\n
Z-Score Normalization (Standardization): \nv' = \frac{v - \text{mean}}{\text{stand\_dev}}\n
Decimal Scaling Normalization: \nv' = \frac{v}{10^j}\n
where is the smallest integer such that \n\max(|v'|) < 1\n
Feature Selection Strategies:
Goal: Eliminate uninformative or redundant features to reduce computational complexity and prevent model degradation.
Screening Metrics:
Correlation coefficients for continuous variable interactions.
Analysis of Variance (ANOVA) testing for categorical factors, selecting attributes meeting significance thresholds (e.g., ).
Collinearity between Features:
Including highly correlated features introduces collinearity issues in linear models.
Correlation matrices and scatter plot heatmaps help identify and prune redundant predictor groups.
Near-Zero-Variance Predictors:
Features containing near-constant values across all observations provide zero explanatory variance.
Model Impact:
Tree-based models are inherently resistant to zero-variance features as they are ignored during split evaluations.
Linear models experience computational instabilities and matrix inversion failures when zero-variance predictors are included.