Data Preparation

1. Handling Missing Data

What is missing data?

Missing data occurs when some values in your dataset are empty or NaN (Not a Number). This can cause problems when analyzing or training a machine learning model.

How to check for missing data?

You can use the isnull() function to check where data is missing.

import pandas as pd

data = {'Name': ['Alice', 'Bob', 'Charlie', 'David'],
        'Age': [25, None, 30, 28],  # Missing value for Bob
        'Salary': [50000, 60000, None, 55000]}  # Missing value for Charlie

df = pd.DataFrame(data)

# Check missing values
print(df.isnull())  # Returns True for missing values
print(df.isnull().sum())  # Count missing values per column

Output:

    Name    Age  Salary
0  False  False  False
1  False   True  False
2  False  False   True
3  False  False  False

Age       1
Salary    1
dtype: int64

Here, Bob's age is missing, and Charlie's salary is missing.

How to handle missing values?

1. Remove missing values (dropna())

   • If only a few values are missing, you can remove those rows.

df_cleaned = df.dropna()
print(df_cleaned)

1. Fill missing values (fillna())

   • You can replace missing values with a fixed value.

df_filled = df.fillna(0)  # Replace missing values with 0
print(df_filled)

1. Use Mean/Median/Mode to fill missing values

   • Mean: Average of column values.

   • Median: Middle value.

   • Mode: Most frequently occurring value.

df["Age"].fillna(df["Age"].mean(), inplace=True)  # Replace missing Age with mean
df["Salary"].fillna(df["Salary"].median(), inplace=True)  # Replace missing Salary with median

Using SimpleImputer for Missing Values

Instead of fillna(), Scikit-Learn's SimpleImputer helps handle missing values automatically.

from sklearn.impute import SimpleImputer

imputer = SimpleImputer(strategy='mean')  # Can use 'median' or 'most_frequent'
df[['Age', 'Salary']] = imputer.fit_transform(df[['Age', 'Salary']])

print(df)

2. Data Preprocessing (Scaling & Normalization)

Why do we need scaling?

• In ML models, different units (e.g., salary in thousands, age in years) can cause issues.

• Normalizing or standardizing helps bring data into a common scale.

Types of Scaling:

1. Standardization (StandardScaler)

   • Converts data to mean = 0, standard deviation = 1.

   • Used for algorithms like SVM, Logistic Regression, K-Means.

2. Min-Max Normalization (MinMaxScaler)

   • Converts data to range 0 to 1.

   • Useful for Neural Networks.

Example of Scaling:

from sklearn.preprocessing import StandardScaler, MinMaxScaler

data = {'Age': [20, 25, 30, 35, 40], 'Salary': [30000, 50000, 70000, 90000, 110000]}
df = pd.DataFrame(data)

# Standardization
scaler = StandardScaler()
df['Age_Standardized'] = scaler.fit_transform(df[['Age']])

# Min-Max Scaling
min_max_scaler = MinMaxScaler()
df['Salary_Normalized'] = min_max_scaler.fit_transform(df[['Salary']])

print(df)

Output:

   Age  Salary  Age_Standardized  Salary_Normalized
0   20  30000         -1.41                0.0
1   25  50000         -0.71                0.25
2   30  70000          0.00                0.50
3   35  90000          0.71                0.75
4   40  110000         1.41                1.0

3. Encoding Categorical Data (Converting Text to Numbers)

Why do we need encoding?

• Machine learning models can't understand text.

• We convert categorical data (like 'Male', 'Female') into numbers.

Types of Encoding

1. Label Encoding (LabelEncoder)

   • Converts each unique value into a number.

   • Example: {Male: 0, Female: 1}.

2. One-Hot Encoding (OneHotEncoder)

   • Converts categories into separate columns (binary).

Label Encoding Example:

from sklearn.preprocessing import LabelEncoder

df = pd.DataFrame({'Gender': ['Male', 'Female', 'Male', 'Female']})

label_encoder = LabelEncoder()
df['Gender_Label'] = label_encoder.fit_transform(df['Gender'])

print(df)

Output:

   Gender  Gender_Label
0    Male             1
1  Female             0
2    Male             1
3  Female             0

One-Hot Encoding Example:

from sklearn.preprocessing import OneHotEncoder

df = pd.DataFrame({'Gender': ['Male', 'Female', 'Male', 'Female']})

one_hot_encoder = OneHotEncoder(sparse=False, drop='first')  # drop='first' to avoid redundancy
encoded_array = one_hot_encoder.fit_transform(df[['Gender']])

df_encoded = pd.DataFrame(encoded_array, columns=['Gender_Male'])
df = pd.concat([df, df_encoded], axis=1)

print(df)

Output:

   Gender  Gender_Male
0    Male          1.0
1  Female          0.0
2    Male          1.0
3  Female          0.0

4. File Handling in Python

Reading CSV Files (pd.read_csv())

import pandas as pd

df = pd.read_csv("data.csv")  # Read CSV file
print(df.head())  # Print first 5 rows

Dynamically Reading Files (sys.path)

You can take user input for file names.

import sys

def read_file():
    file_name = input("Enter file name: ")
    df = pd.read_csv(file_name)
    print(df.head())

read_file()

Final Summary