Biomedical PhD Candidacy Exam: Applied Machine Learning for Biomedical Data Science

MSBD 550: Applied Machine Learning for Biomedical Data Science - Summer 2024 PhD Candidacy Exam Study Guide

Essential Machine Learning Concepts and Definitions

• Unsupervised Learning

  • Definition: A type of machine learning where the algorithm learns from data without labeled responses. It identifies hidden patterns or intrinsic structures in the input data.

  • Example: K-means clustering is a primary example of unsupervised learning, used to group data points into $K$ clusters based on similarity.

  • Contrast: Linear regression, Decision Trees, and Support Vector Machines (SVMs) are supervised learning methods because they require labeled target variables.

• Algorithms for Dual Tasks (Classification and Regression)

  • K-nearest neighbors (KNN): This algorithm is versatile and can be applied to both classification (predicting a discrete label) and regression (predicting a continuous value) tasks.

  • Other mentioned algorithms like Principal Component Analysis (PCA) and K-means are primarily for dimensionality reduction and clustering, while the Apriori Algorithm is used for association rule learning.

• Model Performance: Overfitting

  • Overfitting occurs when a model learns the noise and details in the training data to the extent that it negatively impacts the performance of the model on new data.

  • Characteristic: The model performs well on training data but poorly on unseen or new data.

• Ensemble Methodologies: Bagging vs. Boosting

  • Bagging (Bootstrap Aggregating): Builds multiple models independently and in parallel. The primary goal is to decrease variance. An example is the Random Forest algorithm.

  • Boosting: Builds models sequentially where each new model attempts to correct the errors made by the previous model. The primary goal is to decrease bias. Examples include AdaBoost and Gradient Boosting.

• Fundamental Principles (True/False)

  • Activation Functions: In neural networks, activation functions are essential because they introduce non-linearity into the model, allowing it to learn complex patterns.

  • Confusion Matrix: This tool is specifically used to evaluate the performance of a classification model, not a regression model. Regression models typically use metrics like Mean Squared Error (MSE).

  • Gradient Descent: This is a first-order iterative optimization algorithm used for finding the minimum of a loss function.

  • Naive Bayes Classifiers: These classifiers operate on the "naive" assumption that every feature is independent of any other feature, given the class label.

Biomedical Case Study: Emergency Room Risk Stratification

• Problem Scenario

  • A hospital emergency room measures 17 variables (demographics, clinical data) for newly admitted patients.

  • The goal is to decide whether to admit a patient to the Intensive Care Unit (ICU).

  • Priority: Patients with a survival probability of less than one month are prioritized due to high ICU costs.

  • Objective: Discriminate high-risk patients from low-risk patients.

• Features and Target Identification

  • Features: The 17 measured variables, including demographics and clinical variables.

  • Target Variable: Risk status (Binary classification: High-risk/ICU vs. Low-risk/No ICU).

• Machine Learning Workflow Steps

  1. Data Collection: Gathering the 17 clinical and demographic variables for ER patients.

  2. Data Preprocessing: Handling missing values, normalizing clinical measurements, and encoding categorical demographics.

  3. Exploratory Data Analysis (EDA): Checking distributions of variables and correlations with survival outcomes.

  4. Feature Selection/Engineering: Selecting the most predictive clinical variables among the 17 measured.

  5. Model Selection: Choosing appropriate algorithms (e.g., Logistic Regression, Random Forest, SVM) for binary classification.

  6. Training: Splitting data into training and validation sets to train the model.

  7. Evaluation: Using metrics like sensitivity (recall), specificity, and F1-score, given the high cost of false negatives (missing a high-risk patient).

  8. Deployment: Integrating the model into the ER decision-making process.

Ensemble Learning Theory

• Definition

  • Ensemble learning is a machine learning paradigm where multiple models (often called "weak learners") are trained to solve the same problem and combined to get better results. The main hypothesis is that when weak models are correctly combined, we can obtain more accurate and robust models.

• Types of Ensembles

  • Bagging (Bootstrap Aggregating)

    • Mechanism: It involves taking multiple samples from the training dataset with replacement (bootstrapping) and training a separate model on each sample.

    • Example: Random Forest, which builds multiple decision trees independently and aggregates their results (voting for classification, averaging for regression).

  • Boosting

    • Mechanism: It is an iterative technique that adjusts the weight of an observation based on the last classification. If an observation was classified incorrectly, it tries to increase the weight of this observation.

    • Example: XGBoost or AdaBoost. These are used extensively in competitive machine learning to achieve high accuracy by reducing bias.

  • Stacking (Stacked Generalization)

    • Mechanism: It involves training a learning algorithm to combine the predictions of several other learning algorithms. First, all of the other algorithms are trained using the available data, then a combineder algorithm is trained to make a final prediction using all the predictions of the other algorithms as additional inputs.

Comparative Analysis of Machine Learning Paradigms

• Inference vs. Prediction

  • Inference: Focused on understanding the relationship between the independent variables and the output. It asks how $Y$ is affected as $X$ changes (e.g., used in clinical studies to find disease drivers).

  • Prediction: Focused on accurately forecasting the output $Y$ for new observations. The internal "black box" relationship is less important than the accuracy of the output (e.g., predicting if a specific patient has a disease).

• Supervised vs. Reinforcement Learning

  • Supervised Learning: The algorithm learns from a labeled dataset with clear input-output pairs provided by a "teacher."

  • Reinforcement Learning: The algorithm learns by interacting with an environment. It receives rewards or penalties for its actions and aims to maximize the long-term cumulative reward.

• Overfitting vs. Underfitting

  • Overfitting: High variance/Low bias. The model is too complex and captures the noise in the data.

  • Underfitting: Low variance/High bias. The model is too simple to capture the underlying trend of the data (e.g., fitting a linear line to quadratic data).

• Batch vs. Online Learning

  • Batch Learning: The system is incapable of learning incrementally. It must be trained using all available data at once. To learn about new data, a new version of the system must be trained from scratch on the full dataset.

  • Online Learning: The system learns incrementally by feeding it data instances sequentially, either individually or in small groups called mini-batches. It is ideal for systems that receive data in continuous flows or have limited computing resources.

California Housing Dataset Analysis

• Context

  • Derived from the 1990 U.S. Census.

  • Unit of analysis: Block groups (population of 600 to 3,000 people).

• Attributes (9 Predictors)

  1. housingMedianAge: Median age of houses in the block.

  2. totalRooms: Total rooms within the block.

  3. Latitude: Geographic coordinate.

  4. Longitude: Geographic coordinate.

  5. totalBedrooms: Total bedrooms within the block.

  6. population: Total residents in the block.

  7. households: Total households (groups residing in a home unit) per block.

  8. medianIncome: Measured in tens of thousands of US Dollars ($10,000s$).

  9. oceanProximity: Categorical location relative to the sea.

• Target Variable

  • medianHouseValue: Expressed in hundreds of thousands of dollars ($100,000s$).

• Modeling Considerations

  • Features of Relevance: medianIncome is typically the strongest predictor of house value. oceanProximity and geographic coordinates (Latitude/Longitude) are crucial due to high values in coastal areas.

  • Suited Models: Regression models are required (e.g., Linear Regression, Random Forest Regressor, or Gradient Boosting) because the target variable is continuous.

Decision Tree Learning and Information Theory

• Dataset for Risk Prediction

  • Sample Size: $N = 14$ people.

  • Attributes: Credit History (Bad, U, Good), Debt (H, L), Collateral (None, Ok), Income (L, M, H).

  • Target: Risk (H, M, L).

• Entropy Calculations

  • Individual Entropy: The entropy of a single class $i$ in the set $S$ is calculated based on the proportion $p_i$:         $H(S) = - \text{p}_{\text{High}} \text{log}_2(\text{p}_{\text{High}}) - \text{p}_{\text{Medium}} \text{log}_2(\text{p}_{\text{Medium}}) - \text{p}_{\text{Low}} \text{log}_2(\text{p}_{\text{Low}})$

  • Overall Entropy of the Dataset: This is the expected information needed to classify a member of the set $S$. If there are 14 examples, and the counts for High, Medium, and Low risk are $n_H$, $n_M$, and $n_L$:         $\text{Entropy}(S) = - \frac{n_H}{14} \text{log}_2(\frac{n_H}{14}) - \frac{n_M}{14} \text{log}_2(\frac{n_M}{14}) - \frac{n_L}{14} \text{log}_2(\frac{n_L}{14})$

• Information Gain ($IG$)

  • Definition: The reduction in entropy achieved by partitioning the set on an attribute $A$.

  • Formula:         $IG(S, A) = \text{Entropy}(S) - \textstyle\sum_{v \text{ in Values}(A)} \frac{|S_v|}{|S|} \text{Entropy}(S_v)$

  • Criteria for Best Split: The attribute with the highest Information Gain is selected as the best split point in the decision tree. This attribute reduces the uncertainty (entropy) about the target class the most.