Machine Learning Midterm 2

Neural Networks Big Picture

Flexible Approximators

• nonlinear mapping from input vectors to targets

High-Dimensional Power

• state of the art accuracy for complex data like images, language, sensors

Neural Networks Architecture

Weighted Sum

• computes bias plus weighted input sum

Activation Function

• introduces non-linearity via ReLU, sigmoid, tanh

Parallel Layers

• Stacked units composing hierarchical feature representations

Neural Network Output Layers

Regression

• linear output for continuous target variables

Binary Classification

• Sigmoid output predicting probability

Multiclass

• Softmax output distribution across multiple classes

Forward Pass

Data flows forward through linear combinations and activations to generate predictions

Gradient Descent

Minimizes loss functions by nudging weights downhill via chain rule

Neural Network Overfitting Measures

Weight Decay

• penalize large weights for smoother functions

Dropout

• Randomly drop units during training

Early Stopping

• halt training when validation loss plateaus

Neural Network Pros and Cons

Pros

• High accuracy for nonlinear data

• flexible feature interactions

• supports regression and classification

Cons

• minimal interpretability

• computationally intensive training

• prone to overfitting without tuning

Why Cross Validation

Training Error Bias

• training error underestimates true out of sample error

External Benchmark

• Independent test sets are rare, resampling techniques simulate true out of sample evaluation

Model Evaluation

• reliably compare hyperparameters and architectures without overfitting and training data

Bias-Variance Tradeoff

Bias and Variance are inversely related

You have to try and find the point that jointly minimizes the two

CV Resampling Tools

• Validation Split

• K-Fold CV

• Repeated K-Fold

Validation Split

Random train/validation split set by the computer

• Pros

  • fastest and simplest method

• Cons

  • High variance estimate

K-Fold CV

Partitions the data into K number of folds. The standard is a 5- or 10-fold model

This is the optimal bias-variance tradeoff

Repeated K-Fold

Run K-fold multiple times with different partitions

Reduces the estimate variability

Stratified K-fold

Preserves class proportions across every fold

Vital for imbalanced data and prevents empty positive classes in folds

Group-Aware CV

Keeps entity groups in single folds (patients, customers)

Prevents data leakage from correlated rows

Time-Series CV

Respects temporal order: train on past, validate on future

Spatiotemporal CV

Validation folds defined by geographic blocks

Grid Search Problem Solving

Optimistic Bias

• Training error is optimistic: a single validation split is noisy and unreliable

Variance Reduction

• K-fold CV averages across folds while grid search ensures systematic comparison

End-to-End Tuning Process of Grid Search

Define Metric

• align metric with real business objectives

Choose CV

• match scheme to data-generating process

Specify Grid

• select meaningful log spaced parameter values

Evaluate

• run CV, rank models, and test on holdout data

Choosing CV Design

Standard K-Fold

• 5 or 10 are robust defaults, repeated CV stabilizes small datasets

Stratified CV

• Preserves class proportions; keeps rare classes present and reduces variance

Group and Time Series

• Group K-fold prevents leakage; time-series validates future on past

Grid Metrics

Avoiding Optimistic Bias

Using the same CV scores for choosing hyperparameters and final performance introduces mild selection bias

Real-World ML Sequential Processing

Multi-Stage Workflows

• Models rarely work directly on raw data, there often has to be some sort of transformation

Data Leakage Defense

• Preprocessing before validation leaks test fold statistics, producing overly optimistic results

Unified Parameter Tuning

• GridSearchCV can simultaneously optimize preprocessing steps and model hyperparameters

pipe.fit(x,y)

Calls fit.transform() sequentially through intermediate steps, passing modified data forward

pipe.predict()

Transforms test data using fitted transformers before executing predictions with the final estimator

Double Underscores

Use stepname__parameter syntax to assign parameters to specific pipeline steps

make_pipeline utility

Automatically generates step names based on lowercased class names

Duplicate Step Classes

Appends numbers sequentially when class types repeat

named_steps

Dictionary mapping of stepnames to fitted estimator objects

best_estimator

Returns the full optimal pipeline instance from GridSearch

What is Unsupervised Learning

Extracting structural patterns and hidden representations directly from features, without relying on target labels

4 Primary Unsupervised Task Families

Clustering

Dimensionality Reduction

Density and Anomalies

Association Rules

Clustering Quality Metrics

Within Cluster SSE

• measures internal cluster compactness

• Monotonically decreases as k grows

• use elbow method to select optimal k

Silhouette Coefficient

• a_i = Mean distance to same cluster instances

• b_i mean distance to nearest neighbor cluster

• Range [-1,1] = Higher values signal well-separated bounds

Dimensionality and Association Metrics

PCA Explained Variance

• 90%+ is the target threshold

Association Lift

• >1 indicates positive feature dependence, it means we have increased the value

• =1 indicates there is a random chance of cooccurrence

Unsupervised Learning Workflow

Scale Features

Handle Outliers

Encode

Deduplicate

Validation/Robustness Checks for Unsupervised Learning

Seed Stability

• rerun models across initializations, structure should remain persistent

Sensitivity Analysis

• test sensitivity to metric selection

Downstream Utility

• verify downstream performance gains when feeding supervised pipelines

Unsupervised Learning Pitfalls

Over-Interpretation

• Clusters are hypotheses, not ground truth

Curse of Dimensionality

• Distance metrics lose contrast in high-dimensional spaces

• reduce dimensionality prior to clustering

Pipeline Data Leakage

• fit unsupervised transformers on training splits only before transformation

What is Principal Component Analysis (PCA)

Data compression via rotation. Re-express high dimensional data into new, orgothonal, uncorrelated axes ordered by total variance explained

• PC1 - Direction of maximum variance

• PC2 - Maximizes remaining variance subject to zero correlation with PC1

• 
  

Key Benefits of PCA

Dimensionality Reduction

• Faster model training and lower computational overhead

Multicollinearity Relief

• Removes correlation between colinear predictors completely

Noise Reduction

• Drops trailing components capturing high-frequency noise

Data Visualization

• Projects complex data on 2D/3D maps to spot clusters

PCA Fitting Process

Standardize and Center

• Subtract mean and scale to unit variance before decomposition

Find Directions

• Eigen decomposition of covariance matrix of SVD of data matrix

Project Data

• Project centered data onto top K directions to get Z scores

PCA Formula

Choosing Number of Components for PCA

Cumulative Explained Variance

• Target 90% of the variance as your threshold and choose the smallest K that achieves it

Scree / Elbow Plot

• Plot explained variance v component index. Find the point the slope decreases sharply

Cross Validation

• Treat k as a hyperparameter, optimize k directly against downstream task metric

K-Means Clustering

Core Objective

• partitions observations into k distinct clusters by minimizing within cluster squared distance to learned centroids

K-Means Clustering Optimal Applications

Compact spherical grouping, behavioral segmentation, vector quantization, and automated feature engineering

K-Means Clustering Limitations

Fails on non-spherical clusters, varying density, and non-numeric categorical data lacking distance metrics

Lloyd’s Optimization Method

Initialization

• Select k initial centroids randomly or via k-means seeding scheme

Assignment

• assign each observation to its closest centroid

Update Step

• recompute centroid as the arithmetic mean of assigned points

Convergence

• Repeat steps 2-3 until assignments freeze or J drops below tolerance

K-Means Clustering Data Prep

Feature Scaling

• variables with large variance distort distance metrics, must standardize data

Outlier Handling

• Extreme observations drag centroids away from dense regions

PCA Pre-Processing

• High dimensions dilute dimensions

• retain 90% variance via PCA before clustering

K-Means++

Seeds centroid distance from each other using probability distribution proportional to squared distance

Mini-Batch K-Means

Uses random subsamples per iteration

Significantly accelerates convergence on massive datasets with minimal metric loss

Spherical K-Means

Normalizes feature vectors to unit length and replaces Euclidean distance with cosine similarity

Tailored for text vectors

K-Means Metrics

Inertia

• Internal variance sum, the lower the better

Calinski-Harabasz Index

• Ration of between to within dispersion, the higher the better

Davies-Bouldin Index

• Average similarity between clusters, the lower the better

K-Means Clustering Pitfalls

Spherical Assumption

Complex Shape failure

Outlier Sensitivity

Fixed k Required

Agglomerative Hierarchical Clustering

Bottom Up Merge

• start with every observation as its own cluster

Iterative Pair Linkage

• Recursively merge the two closest clusters based on distance metric

Dendrogram Cut

• Slice the tree horizontally at desired height to yield target cluster count

Types of Linkage Criteria

Single

• Minimum distance

• Good for elongated shapes, prone to chaining

Complete

• Maximum Distance

• Yields compact, equal sized clusters

Average

• Mean Pairwise

• Balances single and complete

Ward’s

• Minimum SSE

• Minimizes withing cluster variance growth, similar output to k-means

DBSCAN Core Concept

Eps Neighborhood

• Maximum neighborhood search distance around any given data point

Core Point Condition

• Minimum points required within eps radius to form a dense region

Core Point

Has minimum number of samples inside its eps neighborhood

Border Point

Within epsilon of a core point, but lacks enough neighbors itself

Noise Point

Does not meet any criteria, explicitly filtered out as an outlier

min_samples rule of thumb

set it equal to Dimensionality +1

Use larger values for noisy data or high dimensions to enforce higher core density

Unsupervised Method Comparison to Remember

What is Text Mining

Core Transformation

• Unstructured text to Structured representations for algorithms

Scalable Reading

• Automates document reading to detect emerging patterns, risks, and sentiment at scale

Business Impact

• Powers intelligent routing, risk scoring, QA automation, and product designs

Text Mining Workflow

Collect Text

Preprocess

Represent

Analyze

Evaluate

Text Mining Preprocessing

Normalization

• Lowercasing, stripping URLs, emojis, and boilerplate text

• the purpose is to make it all look the same

Tokenization

• Splitting raw text into terms or character n-grams

Stopwords

• Reducing word variants to dictionary root forms

• ex) Running to run

Guardrail

• Avoid over-cleaning that destroys critical signals

Feature Engineering Choices

Term-Based (TBM)

• Simple word counts and TF-IDF weighting

Phrase-Based (PBM)

• Multiword n-grams for localized text

Concept-Based (CBM)

• Linguistic POS tags and domain ontologies

Pattern Taxonomy (PTM)

• Frequent sequential rules and hierarchical trees

Core Text Analysis Tasks Chart to remember

Information Extraction (IE)

The goal is to convert raw text into structures schema records

Extracts: entities, attributes, relations

Use Cases: Contract analytics, KYC, clinical notes

Evaluations: Precision, recall, entity-level F1

Information Retrieval (IR)

The goal is to retrieve matching documents for user intent

The model ranks: corpus documents based on relevance metrics

Use Cases: Search bars, knowledge bases, help desks

Evaluations: Precision at k, MAP, nDCG

Supervised Categorization

Assigns predefined labels

Logistic regression and linear SVM

Handles high sparsity and dimensionality well

Unsupervised Clustering

Discovers latent themes without existing labels

Uses algorithms: K-Means, Agglomerative, DBSCAN, LDA

Key Metrics: Silhouette Score, cluster stability

Feature Engineering Trade Offs

Text Method Selection Examples