Machine Learning - Cap 5610 - Midterm

Machine Learning (ML)

Learn from data

ACM

Association for Computing Machinery

ACM Turing Award

Equivalent to the “Nobel Prize of Computing”

Alan Mathison Turing (1912–1954)

Father of theoretical computer science and artificial intelligence who articulated the mathematical foundation and limits of computing

Supervised Learning

A type of machine learning where both input and output are given (labeled data)

Unsupervised Learning

A type of machine learning where only input is given and no labels (unlabeled data)

Support Vector Machine (SVM)

A supervised learning model that classifies data by finding the optimal hyperplane

K-means Clustering

An unsupervised learning algorithm that groups data into clusters based on similarity

Feature

An independent variable

Observation (Row)

An example or data point in the dataset

Accuracy

(TP+TN)/(TP+TN+FP+FN) – proportion of correct predictions

Precision

TP/(TP+FP) – proportion of predicted positives that are truly positive

Recall (Sensitivity/TPR)

TP/(TP+FN) – proportion of actual positives correctly predicted

Specificity

TN/(TN+FP) – proportion of actual negatives correctly predicted

F1 Score

2 * (Precision * Recall) / (Precision + Recall) – harmonic mean of precision and recall

Confusion Matrix

A table showing counts of TP

True Positive (TP)

Correctly predicted positive cases

False Positive (FP)

Incorrectly predicted as positive

False Negative (FN)

Incorrectly predicted as negative

True Negative (TN)

Correctly predicted negative cases

ROC Curve

A graphical plot of True Positive Rate (TPR) vs. False Positive Rate (FPR) at various thresholds

AUC (Area Under Curve)

A measure of the overall performance of a classifier across thresholds

Cross-validation

A resampling method (e.g., 5-fold) to evaluate model performance on different data splits

LASSO (Least Absolute Shrinkage and Selection Operator)

A regression-based method for feature selection

Concrete Autoencoder (CAE)

A deep learning method for feature selection using neural networks

Feature Extraction

Process of creating a new, smaller set of features that captures the most useful information from the original data

Principal Component Analysis (PCA)

A dimensionality reduction method for feature extraction

Autoencoder (AE)

A neural network that compresses and reconstructs data for feature extraction

Pan-cancer Analysis

Studying multiple cancer types to find common biomarkers using machine learning

Stochastic Behavior of CAE

The variability in results due to randomness in training runs of Concrete Autoencoder

Data Representation (Computer Science)

Data is represented as 0 and 1 in general computing

Data Representation (Machine Learning)

Data is represented as vectors and matrices

Scalar

A single number represented by a lowercase italic letter (e.g.

Vector

An ordered array of numbers

Matrix

A 2-D array of numbers identified by two indices

Tensor

An array with more than two axes

Categorical Variable

Discrete/qualitative variable

Nominal Variable

Categorical variable with two or more categories that have no intrinsic order

Ordinal Variable

Categorical variable with two or more categories that can be ordered or ranked

Continuous Variable

A variable that takes values on a continuous scale

Sample

An item to process (classify or cluster)

Feature Vector

An N-dimensional vector of numerical features representing a sample

Feature Selection

Process of filtering out irrelevant or redundant features while keeping a subset of the original features

Structured Data

Data organized in rows and columns (e.g.

Unstructured Data

Data without a predefined structure (e.g., text data), often converted into structured form like Bag-of-Words

Input Vector (𝑥𝑖)

Independent variable representing the ith sample

Response Variable (𝑦)

Dependent variable representing the outcome

Binary Classification (𝑦 ∈ {−1, 1} or {0, 1})

Task of predicting one of two classes

Multi-label Classification (𝑦 ∈ ℤ)

Task of predicting multiple discrete labels

Regression (𝑦 ∈ ℝ)

Predicting a continuous value

Principal Component Analysis (PCA)

A dimensionality reduction technique that transforms high-dimensional data into uncorrelated variables (principal components) capturing maximum variance

Principal Component (PC)

A linear combination of original variables that explains variance in data

PC1

Principal component explaining the largest variance in the dataset

PC2

Principal component explaining the next largest variance

Scree Plot

A plot showing the variance explained by each principal component

Step 1 of PCA

Standardization and centering data so variables share the same scale

Step 2 of PCA

Compute covariance matrix to identify relationships between variables

Step 3 of PCA

Eigen decomposition to identify eigenvectors (principal components) and eigenvalues (variance explained)

Eigenvector

A vector indicating a direction of maximum variance in data

Eigenvalue

A scalar indicating the amount of variance explained by its corresponding eigenvector

Step 4 of PCA

Select significant principal components to create a feature vector

Step 5 of PCA

Project data onto the new feature vector space (dimensionality reduction)

Applications of PCA

Data visualization

t-SNE (t-Distributed Stochastic Neighbor Embedding)

A nonlinear dimensionality reduction technique for visualizing high-dimensional data in 2D or 3D

t-SNE Purpose

Creates a visual “map” of high-dimensional data to reveal patterns and clusters

t-SNE Strength

Preserves local structure of data

Difference PCA vs. t-SNE

PCA preserves global structure (deterministic) while t-SNE preserves local structure (stochastic)

KL Divergence in t-SNE

Measures the difference between probability distributions in original and reduced dimensions

Model Parameters

Values determined using the training dataset

Hyperparameters

Values set before training that control the learning process (e.g.

t-SNE Hyperparameter: Components

Dimension of the embedded space

t-SNE Hyperparameter: Perplexity

, Effective number of nearest neighbors (recommended 5–50, must be < number of samples)

t-SNE Hyperparameter: Iterations

Number of optimization steps (≥250 recommended)

Scatter Plot

A plot showing relationships between two variables (e.g., Iris dataset)

Line Plot

Visualization showing how a variable changes with another

Histogram

Approximate representation of the distribution of numerical data (introduced by Karl Pearson)

Density Plot

Plot showing proportions of values in a distribution

Bar Plot

Visualization effective for categorical data with fewer than 10 categories

Box Plot

Visualization showing distributions with quartiles and medians (five-number summary: min, Q1, median, Q3, max)

Violin Plot

Visualization combining box plot and density plot to show distribution and probability density

LN_IC50

Log normalized IC50 representing drug dose sensitivity or resistance

Low IC50

Indicates tumor cells are more sensitive to the drug (better response)

High IC50

Indicates tumor cells are resistant to the drug (worse response)

Regression Models

Predict a continuous target variable such as drug response (IC50)

Five-Fold Cross Validation

Model evaluation method splitting data into 5 folds for training/testing

MAE (Mean Absolute Error)

Average of absolute differences between predictions and true values

MSE (Mean Squared Error)

Average of squared differences between predictions and true values

RMSE (Root Mean Squared Error)

Square root of MSE

R² (Coefficient of Determination)

Proportion of variance explained by the model

PCC (Pearson Correlation Coefficient)

Measure of linear correlation between predicted and actual values

CatBoost

Gradient boosting decision tree model shown to perform best for Docetaxel regression

SHAP (SHapley Additive exPlanations)

Explainable AI method for feature importance attribution

Global Feature Importance (Regression)

Average of absolute SHAP values across all samples

Local Feature Importance (Regression)

SHAP values for an individual sample prediction

Top 10 Genes (Docetaxel)

Most important genomic features identified by SHAP for drug response

Force Plot (SHAP)

Visualization showing how each feature contributes positively or negatively to prediction

Classification

Predicting categorical labels (e.g., cancer type) using input features

Five-Fold Cross Validation (Classification)

Evaluation method where classifier is trained/tested across 5 splits

Evaluation Metrics (Classification)

Accuracy

LightGBM

Gradient boosting framework identified as best classifier in experiments