Machine Learning Foundations - Probability Distributions and Supervised Learning

ML and Probability

Definition of Probability: Probability is defined as the likelihood of an event occurring. It is a fundamental concept used in daily life as well as in computing.
Everyday Examples of Probability:
- Likelihood of flipping a coin and getting a Head.
- Likelihood of flipping a coin 3 times and getting all Heads.
- Probability of rain tomorrow morning.
- Chances of a person (e.g., James) bringing their lunch to school.
- The likelihood of a friend passing a quiz.
Mathematical Formula:

$\text{Probability of that event} = \frac{\text{Number of that event happening}}{\text{Total number of all events}}$

Relationship between Machine Learning and Probability:
- Machine Learning is primarily a predictive tool.
- It uses past data to predict future outcomes, provided the data has a consistent pattern or trend.
- Probability is the branch of mathematics that enables these predictions through calculations of various outcomes for a given event.
- In ML, the goal is often to predict the future by selecting the most likely outcome based on past data patterns.

Probability for Random Variables: Joint, Marginal, and Conditional

Single vs. Multiple Random Variables:
- Finding the probability for a single random variable is relatively straightforward.
- In Machine Learning, multiple random variables often interact, requiring more complex calculations involving joint, marginal, and conditional probabilities.
Joint Probability: Represents the probability of event A and event B happening together.
- Notation: $P(A \cap B)$ or $P(A \text{ and } B)$ .
- Deck of Cards Example: If event A is picking a 6 and event B is all red cards, the joint probability is the intersection (picking the 6 of Hearts or the 6 of Diamonds).
- Calculation:

$P(A=6) = \frac{4}{52}$

$P(B=\text{red}) = \frac{26}{52}$

$P(6 \cap \text{red}) = P(6) \times P(\text{red}|6) = \frac{1}{26}$

Marginal Probability: Represents the probability of a specific event occurring, regardless of any other event.
- Notation: $P(A)$ .
- Deck of Cards Example: The probability of getting a red card regardless of the number.
- Calculation:

$P(\text{red}) = \frac{26}{52} = \frac{1}{2}$

Conditional Probability: Represents the probability of event A occurring, given that event B has already happened.
- Notation: $P(A|B)$ .
- Formula:

$P(A|B) = \frac{P(A \cap B)}{P(B)}$

Example Calculation (6 given Red): Using the previous card data:

$P(6|\text{red}) = \frac{P(6 \cap \text{red})}{P(\text{red})} = \frac{1/26}{1/2} = \frac{1}{13}$

Probability Distributions

Definition: A probability distribution shows all possible values of a random variable and how often they occur.
List of Common Distributions:
- Bernoulli Distribution
- Uniform Distribution
- Binomial Distribution
- Normal Distribution
- Poisson Distribution
- Exponential Distribution

Bayes Theorem

Core Utility: Bayes Theorem is used to find the conditional probability $P(B|A)$ when the conditional probability $P(A|B)$ is already known.
Mathematical Derivation:
1. $P(A|B) = \frac{P(A \cap B)}{P(B)}$ therefore $P(A \cap B) = P(A|B) \times P(B)$
2. $P(B|A) = \frac{P(A \cap B)}{P(A)}$ therefore $P(A \cap B) = P(B|A) \times P(A)$
3. Equating both: $P(A|B) \times P(B) = P(B|A) \times P(A)$
4. Final Theorem Form:

$P(B|A) = \frac{P(A|B) \times P(B)}{P(A)}$

Components of the Theorem:
- Posterior ( $P(B|A)$ ): The conditional probability we want to find.
- Likelihood ( $P(A|B)$ ): The conditional probability we already know.
- Prior ( $P(B)$ ): The knowledge or probability we already possess about B.
- Normalization Constant ( $P(A)$ ): The marginal probability of A.
Machine Learning Context: Usually, B refers to labels (targets) and A refers to features (inputs).
Real-World Application Examples:
- Medical Diagnosis: If we know the probability a patient drinks alcohol given they have liver cancer ( $P(\text{drink} | \text{cancer})$ ), we can find the probability they have cancer given they drink ( $P(\text{cancer} | \text{drink})$ ).
- Daily Activities: Finding the probability of a friend playing tennis given it is raining, based on the known likelihood of rain when they play tennis.

Na#ve Bayes Classifier

Handling Multiple Features: While standard Bayes Theorem works on single features, the Na#ve Bayes Classifier extends this to multiple features ( $A_1, A_2, \dots, A_n$ ).
Simplification Assumption: It assumes that features are independent of one another (the "na#ve" assumption).
Formula for Multiple Features:

$P(B|A_1, A_2, \dots, A_n) = \frac{P(A_1, A_2, \dots, A_n | B) \times P(B)}{P(A_1, A_2, \dots, A_n)}$

Simplified Calculation:

Benefits:
- Fast and easy to predict classes for test datasets.
- Performs well in multi-class predictions.
- Often outperforms other models when the independence assumption holds.
Metaphorical Example: Deciding whether Sarah will watch a movie based on features like "is it a blockbuster," "is it romance," "does it have Tom Cruise," and "has her sister watched it."

Machine Learning and Decision Trees

Definition: A Decision Tree is a supervised machine learning technique that iteratively splits data according to specific conditions.
Structure: Each split represents a choice based on a condition, leading to final outcomes at the ends of the branches.
Key Terminologies:
- Root Node: The very first node at the top; it has no parent nodes.
- Internal Node: A node inside the tree with one parent node and at least two child nodes.
- Leaf Node: The end of a branch, assigned a class label (either pure or based on majority vote).
- Parent and Child Nodes: When a node splits, the original is the parent, and the resulting nodes are children.
Binary Decision Tree: A specific type where setiap node splits into exactly two branches (Boolean: Yes/No, True/False). For $n$ features, there can be a maximum of $2^n$ potential splits.
Benefits:
- Provides comprehensive analysis by displaying all possible outcomes.
- Easy to use by focusing on one decision at a time.
- High interpretability (easy to understand how an outcome was derived).

Building a Decision Tree: The CART Algorithm

Algorithms for Decision Trees: Major algorithms include CART, ID3, Chi-Square, and Reduction in Variance. This guide focuses on CART (Classification and Regression Trees).
Gini Index (Gini Coefficient):
- A measure of distribution and variance between data points.
- Calculation Formula: $1 - \sum (P_i)^2$
- Meaning: Higher Gini index indicates larger inequality/impurity. A Gini index of $0$ means the data is perfectly pure (all points belong to the same outcome).
CART Process Step-by-Step:
1. Calculate the Gini Index for each individual attribute.
2. Calculate the weighted sum of the Gini Index for the attribute.
3. Select the attribute with the lowest Gini Index value as the splitting node.
4. Update the tree and repeat the process for sub-nodes.

Example: CART Calculation (Tennis Dataset)

Goal: Predict "Play Tennis" (Yes/No) based on features: Outlook, Temperature, Humidity, and Wind.
Outlook Calculations:
- Sunny: 5 instances (2 Yes, 3 No). $\text{Gini} = 1 - (2/5)^2 - (3/5)^2 = 0.48$
- Overcast: 4 instances (4 Yes, 0 No). $\text{Gini} = 1 - (4/4)^2 - (0/4)^2 = 0$
- Rain: 5 instances (3 Yes, 2 No). $\text{Gini} = 1 - (3/5)^2 - (2/5)^2 = 0.48$
- Weighted Sum for Outlook:

$\text{Gini}(\text{Outlook}) = \frac{5}{14} \times 0.48 + \frac{4}{14} \times 0 + \frac{5}{14} \times 0.48 = 0.343$

Final Attribute Comparison (Recap):
- Gini(Outlook) = 0.343
- Gini(Temperature) = 0.441
- Gini(Humidity) = 0.367
- Gini(Wind) = 0.428
- Selection: Outlook is chosen as the Root Node because it has the lowest Gini value.
Sub-node Splitting: Once Outlook is the root, the data is partitioned (e.g., the "Sunny" branch). We then repeat Gini calculations for the remaining features (Temperature, Humidity, Wind) within that specific branch.
- Within the "Sunny" branch: Gini(Temp) = 0.2, Gini(Humidity) = 0, Gini(Wind) = 0.467.
- Selection: Humidity becomes the next node under Sunny.

Python Implementation of Decision Trees

Key Libraries: scikit-learn (specifically DecisionTreeClassifier), pandas (for data manipulation), and graphviz (for visualization).
Label Encoding: DecisionTreeClassifier only accepts numeric input. Strings (e.g., "Sunny", "Rain") must be converted to numbers using Label Encoding or One-Hot Encoding.
Process Flow:
1. Read Data: Usually through pandas.read_csv().
2. Preprocessing: Encode labels and features into numeric format.
3. Split Data: Use train_test_split to create training and testing sets (X for features, y for labels).
4. Modeling: Use clf.fit(X_train, y_train).
5. Prediction: Use clf.predict().
6. Evaluation: Calculate accuracy scores and generate classification reports.
Graphical Output Interpretation:
- Samples: The number of data points at that node.
- Value: Array showing the count of each label (e.g., [# No, # Yes]).
- Class: The majority label at that node.
- Condition: The feature threshold used for the split (e.g., Outlook <= 1.5).

Questions & Discussion

Building a Probability Machine:
- Question: Build the model based on flipping a coin 20 times. Using that, what will your next flip be? Does it match your expectation?
Bayes Theorem Scenarios: Write the theorem for:
- The probability of an autonomous car stopping, given it sees a pedestrian walking.
- The probability of a patient being pandemic positive, given they are coughing.
- The probability of credit card fraud, given the transaction amount is $100.
Decision Tree Benefits: Beyond comprehensive analysis and ease of use, what other benefits can you think of?
Application 1: Customer Bank Loan Application:
- What features determine the tree? (Income, credit score, employment history).
- Collection difficulty? (Standard banking procedures).
- Ethics? (Bias against certain demographics, transparency in loan denial).
Application 2: Prioritizing Patients in ER:
- Features? (Vital signs, age, severity of injury).
- Collection? (Emergency intake).
- Ethics? (Fairness in life-and-death decisions, algorithm bias).
Application 3: Student Admission or Scholarship:
- Features? (GPA, test scores, extracurriculars).
- Ethics? (Socioeconomic bias, transparency in shortlisting).
Python Evaluation:
- What happens to accuracy when you rerun the code? (Randomness in train_test_split can change results).
- How can accuracy be improved? (Modifying hyperparameters in the classifier).
- What are the impacts of modifying parameters like tree depth?

Definition of Probability: Probability is defined as the likelihood of an event occurring. It measures the uncertainty associated with the occurrence of random events and is expressed as a number between 0 (impossible event) and 1 (certain event). This fundamental concept is widely used in various fields including statistics, finance, and machine learning to model uncertainty in data and decision making.
Everyday Examples of Probability:
- The likelihood of flipping a coin and getting a Head (probability = 0.5).
- The likelihood of flipping a coin 3 times and getting all Heads (probability = $\frac{1}{2} \times \frac{1}{2} \times \frac{1}{2} = \frac{1}{8}$ ).
- The probability of rain tomorrow morning (this can vary anywhere from 0 to 1 based on weather data).
- The chances of a person (e.g., James) bringing their lunch to school can be evaluated by considering historical data on James’s lunch habits.
- The likelihood of a friend passing a quiz depends on their study habits and previous performance, which can also be quantified as a probability.
Mathematical Formula: $\text{Probability of that event} = \frac{\text{Number of that event happening}}{\text{Total number of all events}}$ This formula can also be extended to complex scenarios involving multiple events and conditions.
Relationship between Machine Learning and Probability:
- Machine Learning is primarily a predictive tool that relies heavily on probabilistic models.
- It uses past data to predict future outcomes, provided the data has a consistent pattern or trend. The efficiency of predictions in machine learning is highly dependent on the underlying probability distributions of the data.
- Probability is the branch of mathematics that enables these predictions through calculations of various outcomes for a given event, which allows for assessing the chances of different scenarios occurring.
- In ML, the goal is often to predict the future by selecting the most likely outcome based on past data patterns, aiding in applications such as recommendation systems, fraud detection, and more.