Data Preprocessing and Machine Learning Foundations

Professional Background and Industry Context

• The field of data science and machine learning requires extensive practical experience. Professionals in this space often possess dual expertise in IT and advanced analytics, such as having thirteen-plus years in IT and ten-plus years specifically in Data Science and Artificial Intelligence/Machine Learning (AI/ML).
• Current specialized focuses include agentic identity and the fine-tuning of Small Learning Models (SLMs) and Large Language Models (LLMs).
• Data scientist roles frequently involve cloud certifications and a deep understanding of the full data lifecycle.

The Core Concept and Importance of Data Preprocessing

• Data preprocessing is the foundational step in training any machine learning model. It involves transforming raw data into a usable format to ensure the model functions optimally and produces accurate results.
• Definition: Preprocessing is the process of making data usable by removing redundancies, cleaning errors, encoding string values, and handling missing or repeated data.
• The Vegetable Analogy: Using raw data is compared to purchasing a potato at a market. A potato cannot be used in its raw form; it must first be washed, peeled, and chopped (processed) before it can be used for cooking (training a model).
• Time Allocation: Data cleaning is the single biggest factor in model performance. Approximately $60\,$ to $70\,$ of a data scientist's time is dedicated purely to data cleaning and preprocessing.
• Garbage In, Garbage Out (GIGO): If data is not cleaned or processed correctly, the resulting model output will be messy, noisy, and unpredictable. Even high-precision studies can fail if preprocessing is neglected.

Real-World Case Study: 2017 MIT Cancer Detection Model

• In 2017, researchers at MIT developed a deep learning model designed to detect cancer cells from images.
• The Dataset Flaw: During the training phase, the images containing cancer cells featured a ruler placed behind the tumor to measure its length. The healthy cell images did not include a ruler.
• The Consequence: The model did not learn to identify cancer cells; instead, it learned to identify rulers.
• Resulting Failures:
  • If a ruler was placed next to a healthy cell, the model predicted it as cancerous.
  • If a ruler was omitted from a cancerous cell image, the model predicted it as healthy.
• The Lesson: This failure highlights the necessity of thorough preprocessing, data cleaning, and feature engineering to ensure clarity and valid annotations before training.

Data Preprocessing Workflow and the Electricity Analogy

• Standard Workflow:
  • Raw Data: Collected directly from sources; often messy and noisy.
  • Preprocessing: Cleaning data, handling null values, and performing feature engineering.
  • Model Creation: Building the algorithmic framework.
  • Prediction: Utilizing the trained model on new data.
  • Useful Results: Interpretable and applicable outcomes.
• The Electricity Analogy:
  • Modern society perceives AI or data as the "new electricity." Raw electricity from power grids or generation houses can reach voltages of $11,000\,V$ up to $100,000\,V$.
  • Direct usage of raw electricity would destroy appliances and harm individuals (turning a TV into "Lay's chips").
  • Sub-stations act as the "preprocessing" layer, breaking down high voltage into usable formats like $440\,V$ or $220\,V$ for household appliances.

Handling Missing Values

• Missing values represent one of the most significant challenges in data handling. For example, in a class test dataset containing physics, chemistry, math, and English scores, several students may lack individual subject grades.
• Imputation Techniques:
  • Mean: Used when data follows a normal distribution. If one value is missing among scores like 78, 32, and 93, the mean is approximately $\frac{78 + 32 + 93}{3} \approx 67.67$.
  • Median: Preferred for skewed data or when significant outliers are present. It represents the literal middle point ($n/2 + 1$ for ascending sequences) and captures variance without being biased by extreme values.
  • Mode: Specifically used for categorical or string-based data (e.g., shoe sizes in a store). It represents the most frequent value.
  • Advanced Methods: Production-level environments often use Forest-based classifiers, regression models, or deep learning models to predict and fill missing values. While expensive, these provide maximum accuracy for sensitive sectors like finance.
  • Back-filling/Forward-filling: Taking the next or previous value to fill the gap (often reserved for Proof of Concepts or POCs).
• Dropping Data Criteria:
  • If a column contains more than $20\,$ to $25\,$ missing values, it is generally considered noise and should be dropped.
  • Keeping columns with excessive missing data ($50\,$ or more) leads to wrong assumptions because any imputation becomes more representative of the "imputed noise" than the actual population.

Statistical Distribution and Feature Characterization

• Before imputing values, data scientists plot the data to determine distribution:
  • Normal Distribution: Data is centered around the mean.
  • Skewed Distribution: Data is shifted left or right; median is the better representative here.
• Retail Application of Mode: A shoe shop near a college uses mode to determine which sizes ($7$ or $8$) to keep in stock and price higher based on frequency of demand.

Correlation and Redundancy

• Definition: Correlation measures the extent to which two variables are interrelated and how a change in one affects the other.
• The Relationship Analogy: If a boy is friends with two girls and tells both they are his "only best friend" and the "most beautiful," he cannot have both in the same room without bias or conflict.
• Mathematical Application: In features, if two columns are highly correlated, they provide redundant information. Keeping both biases the model judgment. Data scientists must select only one to avoid multicollinearity.

Inconsistent Formatting and Data Types

• Inconsistent Formatting: Common in dates (e.g., $05/06/2026$ vs. $06/05/2026$) and string inputs (e.g., "Ravi" vs. "RAVI").
  • Pydantic Library: A Python library used for schema validation. It allows for the definition of classes where columns and data types are strictly expected (e.g., Name as String with $30$ characters).
  • Normalization: Using methods like .upper() or .lower() ensures that categorical comparisons remain uniform.
• Wrong Data Types: Defining a PIN code ($560037$) as a number rather than a string can lead to the model incorrectly relative-ranking locations (assuming one area is "better" because the number is higher).

Feature Scaling: Standardization vs. Normalization

• Scales and Attributes: In a dataset with Age ($40\text{--}55$), Salary ($70,000\text{--}130,000$), and Weight ($62\text{--}73\,kg$), the raw distances between salary values are exponentially larger than age values.
• Problem: Distance-based algorithms will find these features incomparable and fail to converge.
• Scaling Methods:
  • Standard Scaler: Utilizes mean and standard deviation to transform data.
  • Min-Max Scaler: Rescales data to a fixed range, typically $0$ to $1$.
• When Scaling is Required:
  • Distance/Kernel Based: K-Nearest Neighbors (KNN), Principal Component Analysis (PCA), Support Vector Machines (SVM) with radial bias functions, and K-means.
  • Gradient Descent Optimized: Linear Regression, Logistic Regression, and Neural Networks. Scaling prevents zigzagging and allows the model to converge at the optimal point faster.
• When Scaling is NOT Required:
  • Tree-Based Algorithms: Random Forest, XGBoost, CatBoost, and LightGBM. These use specific criteria (thresholds) for splitting rather than calculating continuous distance.

Outlier Detection and Treatment

• Definition: Outliers are data points that significantly differ from the rest of the dataset and do not follow the established pattern.
• Genuine Outliers vs. Typos:
  • Typo: An age of $-1$ or a Blood Pressure (BP) of $190$ for a $20$-year-old is likely an error or equipment malfunction.
  • Genuine Case: A student with an IIT JEE rank of $2,000$ choosing a local college instead of an IIT to stay near parents. While this point deviates from the college's average rank pattern, it is a valid, real-world case.
• Procedure: Consult a Subject Matter Expert (SME) before treating outliers to understand the cause. Do not simply "blanket scale" or delete them.

Exploratory Data Analysis (EDA) on the Titanic Dataset

• Initial Setup: Libraries used include pandas (pd), numpy (np), matplotlib.pyplot (plt), and seaborn (sns).
• Loading Data: Using sns.load_dataset('titanic') results in a dataframe (df) with $891$ rows and $15$ columns.
• Inspection Commands:
  • df.head(10): Views the first $10$ rows.
  • df.tail(10): Views the last $10$ rows.
  • df.shape: Returns the dimensions.
  • df.columns: Lists all features.
  • df.info(): Provides data types. "Object" type often indicates mixed data or strings.
  • df.describe(): Generates summary statistics (mean, std dev, min, max, percentiles). Use .transpose() to flip for better readability.
  • df.isnull().sum(): Identifies missing values (Age: $177$, Deck: $688$, Embarked: $2$).
• Categorical Inspection: df['column_name'].value_counts() identifies class distributions.
  • Example: In the "Alive" column, $549$ people died and $342$ survived.

Class Imbalance and Banking Fraud

• Class Imbalance: This occurs when one target class significantly outnumbers another.
• Example: In a banking dataset of $1,000$ transactions, $995$ might be normal and $5$ might be fraudulent.
• The Accuracy Trap: A model that simply predicts "Non-fraud" for everything will be $99.5\,$ accurate but $100\,$ useless. In these cases, the Key Performance Indicator (KPI) is the ability to identify the minority class (fraud).
• Threshold Rule: If the minority class contains less than $50\,$ of the majority class's population, it is technically imbalanced.

Principles of Object-Oriented Programming (OOP)

• Definition: OOP is a paradigm where everything is treated as an "object" with specific attributes and methods to mimic real-world entities.
• Application in Finance: A "User" object in a banking app includes attributes like Date of Birth, Gender, Balance, and Account Number.
• Advantages over Functional Programming:
  • Maintainability: Functions are grouped within classes (e.g., all user-related functions in a User class) rather than spread across multiple files.
  • Scalability: Functional programming becomes hard to debug as complexity grows; OOP allows for easier tracking of coupling and cohesion.
• Mathematical Foundations for Machine Learning:
  • Matrix Operations: Multiplication, upper/lower column matrices, Eigenvalues, Eigenvectors, and Decomposition.
  • Calculus: Specifically the Chain Rule for optimizing weights in neural networks.

Questions & Discussion

• Question: When should we drop a column versus imputing values?

• Response: Generally, if missing values are less than $20\,$, impute them. If they exceed $25\,$, dropping the column is safer to avoid creating a model that learns from artificial "noise." However, always check the correlation with the target variable first.

• Question: Which mathematical topics are most critical for deep learning?

• Response: Matrix operations are paramount. Everything in machine learning happens in matrices. Calculus is also vital, especially the chain rule for backpropagation.

• Question: Can we use VS Code instead of Jupyter for these tasks?

• Response: Yes, VS Code, Jupyter Notebook, and Google Colab are all acceptable environments for Python-based data science workflows.

• Question: How do we handle scaling in XGBoost?

• Response: You do not need to scale data for XGBoost or any tree-based algorithm. These algorithms use thresholds to create partitions, so the raw magnitude of the numbers does not affect the split logic as it does in distance-based algorithms like KNN.