Machine Learning: Data Representation and Exploration Notes

Machine Learning Overview and Historical Context

• Artificial intelligence (AI) encompasses historical developments and significant scientific breakthroughs that led to modern applications.

• Historical Milestones:

  • Perceptron (1958): A foundational neural network structure. The mathematical model defines the output ($o$) as $o = sign(∑_k w_k i_k)$, where $i_k$ are inputs, $w_k$ are weights, and outputs are typically in the set $\{-1, +1\}$.

  • Backpropagation (1986): A critical development in training neural networks, enabling deeper architectures and complex function mappings, such as $f(4) = 5$, $f(8) = 5$, $f(12) = 9$, and $f(16) = 17$.

• Modern AI Applications:

  • Consumer AI: Generative models for content creation.

  • Research AI: Applications in Protein Folding (biology), Game Engines (simulation), and Autonomous Driving (transportation).

Fundamental Data Representation

• Objects of interest (e.g., cars, persons, countries) are assumed to be represented as $d$-dimensional column vectors unless otherwise noted.

• Example: Porsche Playing Cards: A car is represented as a 6-dimensional column vector ($d = 6$):

  • Porsche 959 (1988):

  • Model Year: $1988$

  • Displacement: $2849\,cm^3$

  • Power Output: $331\,kW$

  • Acceleration: $3.7\,s$

  • Top Speed: $315\,km/h$

  • Weight: $1450\,kg$

  • Porsche 911 GT1 (1997): [$1997, 3163, 400, 3.7, 310, 1120$]

  • Porsche Carrera GT (2003): [$2003, 5733, 450, 3.9, 330, 1380$]

• The Data Matrix ($X$):

  • All representations are collected into a matrix $X$.

  • Rows: Called samples (denoted $n$).

  • Columns: Called features (denoted $d$).

  • Notation:

  • $X = \begin{pmatrix} x_1^T \\ \vdots \\ x_i^T \\ \vdots \\ x_n^T \end{pmatrix}$

  • $x_i^T$: The $i$-th sample.

  • $x_{\cdot j}$: The $j$-th feature (entire column).

  • $x_{ij}$: The $j$-th feature of the $i$-th sample.

Feature Types and Encodings

• Features are categorized based on their data properties:

• Categorical Features (Unordered):

  • Binary: Takes two values, e.g., $x_{ij} \in \{\text{true, false}\}, \{\text{yes, no}\}, \{\text{pass, fail}\}, \{\text{dead, alive}\}$.

  • Nominal: Takes multiple unordered values, e.g., $x_{ij} \in \{\text{English, German, French}\}, \{\text{cat, dog, rabbit}\}, \{\text{biology, physics, chemistry}\}$.

• Numerical Features (Ordered):

  • Ordinal: Ordered labels without consistent spacing, e.g., $x_{ij} \in \{\text{low income, medium income, high income}\}, \{\text{disagree, neutral, agree}\}$.

  • Discrete: Countable values, e.g., $x_{ij} \in \{0, 1, 2, \dots\}$.

  • Continuous: Real-valued numbers, e.g., $x_{ij} \in \mathbb{R}_{\ge 0}$ (mass, distance).

• 1-of-k Encoding (One-Hot Encoding): Categorical data is converted to numerical vectors. For example, animal types [cat, dog, rabbit] become binary columns where a "cat" is [$1, 0, 0$] and a "rabbit" is [$0, 0, 1$].

Complex Data Structures

• Text Data: Represented using Bag-of-Words.

  • Words are counted after ignoring capitalization and punctuation based on a defined dictionary.

  • Pro: Allows distance/similarity computation between documents.

  • Con: Ignores word order (e.g., "The policeman handcuffed the robber" looks the same as "The robber handcuffed the policeman").

• Image Data: Represented as sequences of grayscale pixel intensities.

  • Values range from $0$ (black) to $255$ (white).

  • Dimensionality is extremely high; a $(1080, 1920)$ resolution image has over $2,000,000$ pixels.

• Graph Data: Represented via Adjacency Matrices.

  • Entries can be binary (presence of an edge) or numerical (edge weights).

Operations on Features

• Aggregation: Grouping raw features to reduce combinations (e.g., grouping specific cities into countries like Germany or France).

• Transformation (Discretization/Binning): Turning numerical data into categorical data (e.g., converting ages into bins like <20, $[20, 25)$, and $\ge 25$).

• Selection: Removing irrelevant features (e.g., StudentID) to treat similar samples equally.

• Data Size Rule of Thumb: A popular approximation for the required number of samples is n > 10 \times d.

• Data Cleaning: Identifying and fixing issues including noise (distortion), outliers (entry errors), missing values, and duplicated data.

Data Exploration and Categorical Statistics

• Exploration aims to find outliers, corrupt data, feature correlations, and understand data distribution/ranges using statistics and visualization.

• Frequency: The fraction of entries belonging to a specific class $k$.

• Distribution: The set of frequencies for all classes.

• Mode: The most frequent class in the dataset.

• Entropy ($H(x_{\cdot j})$): Measures the randomness or "average units of information" in a sample.

  • Formula: $H(x_{\cdot j}) = -\sum_{c=1}^K p(c) \cdot \log(p(c))$

  • Range: $0$ (no randomness) to $\log(K)$ (total randomness, where all classes have equal frequency).

  • Exercise (Binary Entropy): If a binary feature is "1" with probability $0.9$, the entropy is $-(0.9 \cdot \log_2(0.9) + 0.1 \cdot \log_2(0.1)) \approx 0.47$.

Numerical Statistics

• Range: Defined by the minimum and maximum: $(\min_{i} x_{ij}, \max_{i} x_{ij})$. Anomalies (negative mass, unit mismatch) can be identified here.

• Arithmetic Mean ($\mu_j$): A measure of central tendency: $\mu_j = \frac{1}{n} \sum_{i=1}^n x_{ij}$.

• Standard Deviation ($\sigma_j$): A measure of statistical dispersion: $\sigma_j = \sqrt{\frac{1}{n} \sum_{i=1}^n (x_{ij} - \mu_j)^2}$.

• Sample Variance: Denoted as $\sigma_j^2$.

• Robust Statistics:

  • Median ($med_j$): The middle value of a sorted list. For odd $n$, it is at position $\frac{n+1}{2}$. For even $n$, it is the average of positions $\frac{n}{2}$ and $\frac{n}{2} + 1$. It is less sensitive to outliers than the mean.

  • Interquartile Range ($IQR_j$): The range between the first quartile ($Q_1$, $0.25$-quantile) and third quartile ($Q_3$, $0.75$-quantile). $IQR_j = [Q_1, Q_3]$.

• Quantiles: A value $z$ is a $q$-quantile if a fraction $q$ of entries are $\le z$.

  • $0.00 \rightarrow \min$, $0.25 \rightarrow Q_1$, $0.50 \rightarrow med$, $0.75 \rightarrow Q_3$, $1.00 \rightarrow \max$.

Correlation and Distances

• Sample Covariance: $cov(x_{\cdot j}, x_{\cdot k}) = \frac{1}{n} \sum_{i=1}^n (x_{ij} - \mu_j)(x_{ik} - \mu_k)$.

• Correlation Coefficient ($\rho$): Measures linear correlation: $\rho(x_{\cdot j}, x_{\cdot k}) = \frac{cov(x_{\cdot j}, x_{\cdot k})}{\sigma_j \cdot \sigma_k}$.

  • Range: $[-1, +1]$.

  • Invariance: Invariant under linear transformations $a \cdot x + b$.

• Distances:

  • Euclidean Distance: $d(x_{\cdot j}, x_{\cdot k}) = \|x_{\cdot j} - x_{\cdot k}\|_2 = \sqrt{\sum_{i=1}^n (x_{ij} - x_{ik})^2}$.

  • Hamming Distance: Counts the number of positions where values differ: $d(x_{\cdot j}, x_{\cdot k}) = \sum_{i=1}^n [x_{ij} \neq x_{ik}]$ (useful for categorical data).

Limitations of Statistics and Visualization Tools

• Anscombe's Quartet & Datasaurus Dozen: Datasets with identical summary statistics (mean, variance, correlation) but vastly different visual distributions, highlighting the necessity of visualization.

• Visualization Methods:

  • Quantile Plot: Maps data values against their $q$-quantiles.

  • Box Plot: Displays $\{min, Q_1, med, Q_3, max\}$ and $IQR$ visually.

  • Histogram: Visualizes counts aggregated into bins.

  • Line Plot: Shows one variable as a function of another.

  • Scatter Plot: Explores joint distribution of two features; color can represent a third variable.

  • Matrix Plot: Heatmap showing all matrix entries or pairwise correlations.

Introduction to Supervised Learning

• Motivation: Determining cause-and-effect in complex scenarios (e.g., tracking food intake to identify allergies).

• Setup:

  • Input: Features of samples (e.g., quantities of food eaten).

  • Output: Class labels (e.g., sick or not sick).

  • Goal: Find a model to accurately predict labels for new samples.

• Classification: A branch of supervised learning where labels are categorical. The resulting model is a classifier.