Comprehensive Study Guide on Decision Trees and Supervised Medical Classification

Fundamentals of Supervised Classification and Decision Trees

• Supervised machine learning models are designed to learn mapping functions from labeled training datasets to assign unseen inputs to target outputs.

• Decision Trees are non-parametric supervised learning models utilized for both classification and regression tasks.

• Predictions in a Decision Tree are generated by recursively splitting input data into sequential branches based on specific feature thresholds until arriving at a final decision, known as a leaf node.

• At each internal node, the tree evaluates a specific feature test condition to split the dataset, aiming to maximize child node purity.

• Optimal dataset splits are evaluated and determined using mathematical purity metrics, including Gini Impurity, Entropy (Information Gain), or Mean Squared Error (MSE).

• Key Strengths of Decision Trees:

  • Highly intuitive and simple to interpret or visualize, operating conceptually like a human decision flowchart.

  • Seamlessly handles both numerical (continuous) and categorical data types without requiring specialized encoding upfront.

  • Requires no feature scaling or normalization steps prior to model training.

  • Capable of capturing complex non-linear relationships without explicit feature engineering.

• Core Limitations of Decision Trees:

  • Highly prone to overfitting and memorization of training data if allowed to grow without depth restriction.

  • Model structures can be unstable; minor variations or small changes in training data can produce drastically different tree configurations.

  • Predictive power and generalization are often substantially improved by deploying ensemble methods such as Random Forests.

• Practical Implementation Frameworks:

  • Implemented in Python via Scikit-Learn using DecisionTreeClassifier() for categorical targets and DecisionTreeRegressor() for continuous targets.

  • Model complexity and depth are controlled using explicit hyperparameters such as max_depth, min_samples_split, min_samples_leaf, and max_leaf_nodes.

Structure and Architecture of Decision Trees

• Root Node:

  • The foundational starting node of the decision tree structure, representing the entire undivided training dataset prior to any feature splitting.

• Internal (Decision) Nodes:

  • Intermediate structural nodes that perform an explicit conditional test on a selected attribute or feature.

  • Possess incoming branches from parent nodes and outgoing branches leading to subsequent child nodes based on test outcomes.

• Branches:

  • Directed connections that represent the possible outcome paths resulting from a conditional test applied at a decision node.

• Leaf (Terminal) Nodes:

  • The final terminating nodes of a decision tree that do not contain further splits.

  • Represent the ultimate predicted class output in classification or continuous target value in regression.

Mathematical Metrics for Split Selection and Impurity

• Split Decision Goal:

  • At every recursive step, the model identifies the specific feature and numerical threshold that achieves the maximum separation of target classes.

  • Selection is governed by node purity: higher purity indicates that a node contains a predominantly higher proportion of a single class.

• Gini Impurity:

  • The default criterion utilized in Scikit-Learn for evaluating classification splits.

  • Quantifies the probability that a randomly chosen sample from the node set would be incorrectly labeled if it were randomly classified according to the distribution of targets in the node.

  • Mathematical Formula:

$Gini = 1 - \sum_{i=1}^{C} p_i^2$

• In this equation, $p_i$ represents the proportion of samples belonging to class $i$ present within the specific node, where $C$ is the total number of classes.

• Operational Numerical Examples:

  • Node A with a $50\%$ Setosa and $50\%$ Versicolor distribution:

$Gini = 1 - ((0.5)^2 + (0.5)^2) = 1 - (0.25 + 0.25) = 0.5$

* Node A reflects high impurity ($Gini = 0.5$).
* Node B with a $100\%$ Setosa distribution:

$Gini = 1 - ((1.0)^2 + (0.0)^2) = 1 - 1.0 = 0$

* Node B reflects complete purity ($Gini = 0$

• Entropy and Information Gain:

  • Entropy quantifies the level of disorder, randomness, or uncertainty within a specific node.

  • Mathematical Formula:

$Entropy = -\sum_{i=1}^{C} p_i \log_2(p_i)$

• Operational Values:

  • A completely pure node containing exclusively one class yields an entropy value of $H = 0$

  • A completely balanced node with a 50/50 mix between two classes yields maximum entropy of $H = 1\text{ bit}$

• Information Gain represents the net drop in total entropy achieved by partitioning a parent node into child sub-nodes.

• The splitting algorithm systematically selects the feature threshold that maximizes Information Gain or minimizes Gini Impurity.

• Recursive splitting halts when maximum node purity is reached, or when user-defined constraints stop tree growth.

Execution and Decision Logic on the Iris Dataset

• Iris Dataset Feature Specifications:

  • Features evaluated: sepal length, sepal width, petal length, and petal width.

  • Target classes evaluated: Setosa (class $0$), Versicolor (class $1$), and Virginica (class $2$).

• Rule Extraction Example:

  • Decision trees derive interpretable logic mimicking human decision logic:

  • Rule 1: If petal length < 2.45\,cm, classify as Setosa.

  • Rule 2: Else if petal width < 1.75\,cm, classify as Versicolor.

  • Rule 3: Else, classify as Virginica.

• Step-by-Step Recursive Splitting Process:

  • Step 1: Root Node Evaluation

  • Starting set contains all $150$ dataset samples (PETALLEN).

  • Evaluates Petal Length $\le 2.45\,cm$ as the initial split.

  • This primary split provides the greatest reduction in impurity, perfectly isolating short-petaled Setosa samples from all other classes.

  • Step 2: Left Subtree Branch

  • Samples satisfying Petal Length $\le 2.45\,cm$ route left.

  • Yields $50$ samples, all classified as Setosa ([Setosa]).

  • Impurity metrics for this node evaluate to $Gini = 0$ and $Entropy = 0$ (pure node; execution terminates on this branch).

  • Step 3: Right Subtree Branch

  • Samples with Petal Length > 2.45\,cm route right, leaving $100$ remaining samples composed of Versicolor and Virginica.

  • Second decision node evaluates Petal Width $\le 4.9$.

  • Step 4: Middle Node Partitioning

  • Sub-branch with $54$ samples (PETALWI) remains mixed.

  • Evaluates sub-split at Petal Width $\le 1.65$

  • If Petal Width $\le 1.65$, routes to a pure leaf of $47$ samples classified as Versicolor ([Versicolor]).

  • If Petal Width > 1.65, splits further based on Sepal Length ($\le 5.95$). Both child splits resolve cleanly into Virginica leaf nodes.

  • Step 5: Rightmost Branch Partitioning

  • Sub-branch with $46$ samples (PETALWI) evaluates Petal Width $\le 1.75$

  • If Petal Width > 1.75, routes directly to a leaf node containing $40$ samples classified as Virginica ([Virginica]).

• Impurity Convergence:

  • Regardless of whether Gini Impurity or Entropy is selected, shallow tree constraints (such as max_depth=2) systematically identify Petal Length $\le 2.45\,cm$ as the root split.

  • This occurs because the initial gain generated by isolating Setosa is identical and optimal under both formulas.

Decision Boundaries, Overfitting, and Regularization

• Nature of Decision Boundaries:

  • Decision trees partition feature space into orthogonal, rectangular regions using binary threshold conditions (e.g., Petal Length $\le 2.45\,cm$).

  • Unlike Support Vector Machines, which construct smooth or curved hyperplanes, decision trees construct axis-aligned linear segment boundaries.

  • Final predictions within any bounded rectangular partition are decided by the majority target class of training samples contained within that space.

• Overfitting Dynamics:

  • An unconstrained decision tree continues splitting until every leaf contains minimal samples (down to a single sample per leaf).

  • Deep trees memorize individual training observations and noise, resulting in perfect training accuracy but degraded generalization performance on test data.

• Regularization Hyperparameters:

  • max_depth: Enforces a hard limit on the maximum vertical depth of the tree structure.

  • min_samples_split: Defines the minimum number of samples a node must contain to consider executing another internal split.

  • min_samples_leaf: Establishes the minimum sample threshold allowed inside a candidate terminal leaf node.

  • max_leaf_nodes: Restricts the total absolute quantity of terminal leaf nodes generated across the entire tree.

• Depth Scaling Behavior:

  • Small tree depths (e.g., max_depth=3 or max_depth=4): Capture generalized macro-structures, remain highly interpretable, and maintain robust test set performance.

  • Large tree depths (e.g., max_depth=10 or greater): Achieve extreme training set accuracy, create excessively fine partition grids, and experience severe overfitting.

The CART Algorithm Architecture

• Definition:

  • CART stands for Classification and Regression Trees, representing the foundation algorithm implemented in Scikit-Learn for building decision trees.

• Operational Algorithmic Steps:

  1. Commences with the total training set residing in a singular root node.

  2. Iterates over every individual feature and evaluates all possible numerical split thresholds.

  3. Measures resulting child node purities for each potential split using Gini Impurity or Entropy.

  4. Commits to the single split option that maximizes child node purity.

  5. Recursively applies steps 2–4 to child nodes.

  6. Halts processing once the maximum allowable depth limit is reached, or when no candidate split produces a non-zero improvement in purity.

Medical Classification Protocols and Clinical Case Study

• Supervised Medical Classification:

  • Machine learning classification assigns medical observational samples into discrete pre-defined diagnostic categories.

  • Case Study Focus: Binary classification of breast tumors as either Benign (non-cancerous, target value $0$) or Malignant (cancerous, target value $1$).

• Medical Significance:

  • Breast cancer represents one of the most prevalent cancers globally, making rapid, early diagnosis critical for patient survival rates.

  • Automated machine learning tools serve as diagnostic aids to physicians by detecting subtle high-dimensional data patterns that evade manual inspection, thereby reducing human diagnostic errors and accelerating screening pipelines.

• Dataset Specifications (Breast Cancer Wisconsin - Fine Needle Aspiration):

  • Fine Needle Aspiration (FNA) is a minimally invasive diagnostic procedure collecting fluid and cell samples directly from breast tissue masses.

  • Features quantify physical geometric properties of cell nuclei derived from digitized tissue images, including radius, texture, smoothness, and symmetry.

  • Total Dataset Size: $569$ instances.

  • Dimensionality: $30$ continuous numerical features per instance.

  • Target Classes: Binary classification target indicating Benign or Malignant.

• Preprocessing Pipeline Protocols:

  • Step 1: Missing Value Verification

  • Process executed via data.isnull().sum() to verify complete data integrity.

  • Confirmed zero missing values present across all $569$ samples.

  • Step 2: Feature Normalization (Min-Max Scaling)

  • Features are rescaled into a standardized closed interval of $[0, 1]$.

  • Mathematical Transformation:

$x_{scaled} = \frac{x - x_{min}}{x_{max} - x_{min}}$

* Purpose: Normalization prevents features with large absolute ranges from dominating model loss functions. Normalization ensures gradient stability in Logistic Regression and is strictly required for Support Vector Machines due to distance sensitivity.

• Step 3: Train-Test Partitioning

  • Dataset is split using train_test_split(X, y, test_size=0.2, random_state=42) or standard $70\%$ training / $30\%$ testing allocations.

  • Standardized setting random_state=42 guarantees exact experimental reproducibility.

Logistic Regression Model Architecture

• Fundamental Definition:

  • Logistic Regression is a parametric statistical model used for binary classification that computes the posterior probability of an instance belonging to a specific target class.

• Mathematical Formulation:

  • Rather than predicting raw linear response values, linear combinations of weighted features are transformed through the Sigmoid (Logistic) Function.

  • Sigmoid Function Equation:

$P(x) = \frac{1}{1 + e^{-z}}$

• Linear Predictor Equation:

$z = \theta_0 + \theta_1 x_1 + \theta_2 x_2 + \dots + \theta_n x_n$

• In these equations, $e$ represents Euler's number and $\theta_i$ represents the learned weight coefficient for feature $x_i$.

  • Decision Rules:

• Probability output $P(x)$ is bounded strictly between $0$ and $1$

• If predicted probability P(x) > 0.5, the sample is classified as class $1$ (Malignant).

• If predicted probability $P(x) \le 0.5$, the sample is classified as class $0$ (Benign).

  • Properties and Advantages:

• Performs optimally on linearly separable feature spaces.

• Highly transparent and computationally lightweight.

• Individual regression coefficients ($\theta_i$) directly communicate the directional impact and strength of each corresponding feature on target probabilities.

  • Implementation Hyperparameters in Scikit-Learn:

• C (Inverse Regularization Strength): Controls parameter magnitude penalty. Smaller C values enforce stronger regularization (simpler model), while larger C values reduce regularization penalties (closer fit to training points). Optimal setting found: C = 10.

• penalty='l1': Implements Lasso Regularization, driving uninformative feature weights to absolute zero to perform embedded feature selection.

• solver='liblinear': Optimization algorithm best suited for small-scale clinical datasets and $\text{L1}$ penalties.

Support Vector Machine (SVM) Hyperplane Optimization

• Fundamental Mechanics:

  • Support Vector Machine is a robust non-parametric algorithm that constructs an optimal decision boundary (hyperplane) separating two distinct classes within feature space.

  • Operates by maximizing the geometric margin, defined as the shortest perpendicular distance between the decision hyperplane and the nearest training data samples from any class.

  • Support Vectors: The critical boundary data points located directly on the edge of the margin that determine the precise position and orientation of the decision hyperplane.

• Non-Linear Classification and Kernels:

  • SVM handles non-linear class distributions by utilizing kernel functions to project input data into higher-dimensional Hilbert spaces where linear separation becomes possible.

  • Radial Basis Function (RBF) Kernel: Formulates non-linear curved decision boundaries that wrap cleanly around isolated spatial data clusters.

• Hyperparameter Controls:

  • C (Regularization Parameter):

  • Balances maximum margin width against training classification error penalties.

  • Low C: Enforces a wider margin, accepting small training errors to maximize generalizability.

  • High C: Enforces a narrow margin, penalizing training errors heavily (increasing overfitting risks).

  • gamma ($\gamma$):

  • Defines the distance radius of influence exerted by individual training support vectors.

  • Low $\gamma$: Means a single sample's influence reaches far, generating smooth, low-variance decision boundaries.

  • High $\gamma$: Restricts influence tightly around support vectors, generating complex, wiggly decision boundaries sensitive to local noise.

Systematic Hyperparameter Tuning via Grid Search

• Methodology:

  • Hyperparameter optimization is conducted systematically using GridSearchCV.

  • Evaluates grid cross-validation combinations of hyperparameters to isolate optimal settings that maximize test accuracy while guarding against overfitting.

• Optimized Experimental Results:

  • For the Breast Cancer Wisconsin dataset, joint parameter tuning established optimal diagnostic performance at C = 10 and gamma = 0.1 (gamma = 0.1).

  • The resulting RBF kernel generates a smooth, curved spatial boundary around data clusters guided strictly by critical support vectors.

Quantitative Evaluation Metrics and Confusion Matrix Framework

• Accuracy:

  • Represents the proportion of total correct predictions relative to total dataset predictions.

  • Formula:

$\text{Accuracy} = \frac{\text{True Positives} + \text{True Negatives}}{\text{Total Predictions}}$

• Best applied when diagnostic classes are balanced across the dataset.

  • Precision:

• Measures the proportion of predicted positive cases that were genuinely positive.

• Formula:

$\text{Precision} = \frac{\text{True Positives}}{\text{True Positives} + \text{False Positives}}$

• Clinical Impact: High precision minimizes False Positives (reducing unnecessary medical alarms and patient stress).

  • Recall (Sensitivity):

• Measures the proportion of actual positive cases that were correctly identified by the model.

• Formula:

$\text{Recall} = \frac{\text{True Positives}}{\text{True Positives} + \text{False Negatives}}$

• Clinical Impact: High recall minimizes False Negatives (reducing missed malignant cancer diagnoses).

  • F1-Score:

• The harmonic mean that balances Precision and Recall into a singular scalar performance metric.

• Formula:

$\text{F1-Score} = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}$

• Critical for medical evaluation when equal trade-offs between false alarms and missed cases must be enforced.

  • Confusion Matrix Structural Framework:

• True Positive (TP): Actual Malignant sample correctly predicted by model as Malignant.

• False Negative (FN): Actual Malignant sample incorrectly predicted by model as Benign (critical diagnostic failure).

• False Positive (FP): Actual Benign sample incorrectly predicted by model as Malignant (false alarm).

• True Negative (TN): Actual Benign sample correctly predicted by model as Benign.

Comparative Performance Analysis: Logistic Regression vs SVM

• Comparative Results:

  • Both Support Vector Machines and Logistic Regression yield exceptional, robust diagnostic performance when classifying benign versus malignant breast tumors.

  • Performance differences between the two models remain marginal across standard metrics (Accuracy, Precision, Recall, F1-Score).

• Model Specific Distinctions:

  • Support Vector Machine (SVM): Demonstrates slightly higher overall robustness. Effectively manages complex non-linear feature interactions via the RBF kernel and offers superior margin optimization, slightly favoring precision gains and reducing misclassification of malignant cases.

  • Logistic Regression: Delivers comparable performance while maintaining full linear transparency. Provides highly interpretable coefficient metrics directly actionable for clinical decision support systems.

  • Impact of Preprocessing: The minimal performance gap confirms that simple linear models perform nearly on par with complex kernel models when paired with complete feature scaling and data preprocessing.

  • Both classifiers maintain balanced precision and recall metrics, confirming equal proficiency at minimizing false positive alarms and dangerous false negative omissions.

• Citation Details:

  • Study Reference: K. Elmazi and D. Elmazi, "Evaluating Support Vector Machine and Logistic Regression: A Machine Learning-Based Statistical Study for Breast Cancer Diagnosis," 2024 International Conference on Computing, Networking, Telecommunications & Engineering Sciences Applications (CoNTESA), Tirana, Albania, 2024, pp. 1-6, doi: 10.1109/CoNTESA64738.2024.10891283.