Coding: Python Syntax Brush-Up

🔰 0⃣ Import Libraries

python

CopyEdit

import pandas as pd import numpy as np import datetime as dt

Explanation:

  • pandas for dataframes (core for ML feature engineering).

  • numpy for array computations.

  • datetime for date-based feature creation.

1⃣ Reading Files / Importing Data

python

CopyEdit

df = pd.read_csv("data.csv")

Explanation:

  • Reads CSV into pandas dataframe.

  • Supports additional params like delimiter=, encoding=, nrows=, etc.

python

CopyEdit

df = pd.read_excel("data.xlsx")

  • Reads Excel file directly.

2⃣ Inspecting the Data

python

CopyEdit

df.head()

df.tail()

df.info()

df.describe()

Explanation:

  • head(): Preview first rows.

  • info(): Data types, null counts.

  • describe(): Summary stats (mean, std, min, percentiles).

python

CopyEdit

df.columns.tolist()

df.shape

  • Get column names & dataframe shape.

3⃣ Basic Filtering

python

CopyEdit

df[df["salary"] > 50000]

df[(df["department"] == "HR") & (df["salary"] > 60000)]

Explanation:

  • Apply boolean conditions to filter rows.

4⃣ Selecting Columns

python

CopyEdit

df["salary"] # Single column (Series)

df[["salary", "bonus"]] # Multiple columns (DataFrame)

5⃣ Missing Value Handling

python

CopyEdit

df.isnull().sum()

df["salary"].fillna(df["salary"].median(), inplace=True)

df.dropna(subset=["salary"], inplace=True)

Explanation:

  • Check missing values.

  • Fill with median.

  • Drop rows where salary is null.

6⃣ Creating New Columns (Feature Creation)

Arithmetic features

python

CopyEdit

df["total_comp"] = df["salary"] + df["bonus"]

Conditional features

python

CopyEdit

df["high_salary"] = np.where(df["salary"] > 100000, 1, 0)

Date features

python

CopyEdit

df["hire_date"] = pd.to_datetime(df["hire_date"])

df["tenure_days"] = (pd.Timestamp.today() - df["hire_date"]).dt.days

df["hire_year"] = df["hire_date"].dt.year

Explanation:

  • Create tenure, extract year from date, create flags.

7⃣ One-Hot Encoding (Categorical to Numerical)

Using pandas:

python

CopyEdit

df_encoded = pd.get_dummies(df, columns=["department"], drop_first=True)

Explanation:

  • Converts department column into binary columns.

  • drop_first=True avoids multicollinearity (reference encoding).

Using sklearn (more production-like):

python

CopyEdit

from sklearn.preprocessing import OneHotEncoder

encoder = OneHotEncoder(sparse_output=False, drop="first")

encoded = encoder.fit_transform(df[["department"]])

encoded_df = pd.DataFrame(encoded, columns=encoder.get_feature_names_out())

df = pd.concat([df.drop("department", axis=1), encoded_df], axis=1)

Explanation:

  • drop="first" avoids dummy variable trap.

  • Use this for pipelines in ML model training.

8⃣ Label Encoding (For tree models)

python

CopyEdit

from sklearn.preprocessing import LabelEncoder

le = LabelEncoder()

df["department_encoded"] = le.fit_transform(df["department"])

Explanation:

  • Converts categories into integers.

  • Can be useful for trees, but risky for linear models.

9⃣ Binning / Bucketing (Discretization)

python

CopyEdit

df["income_bin"] = pd.cut(df["salary"], bins=[0, 50000, 100000, 200000], labels=["low", "mid", "high"])

Explanation:

  • Groups continuous values into bins.

  • Can reduce variance or simplify models.

🔟 GroupBy Aggregations (Feature Engineering)

python

CopyEdit

agg = df.groupby("department")["salary"].agg(["count", "mean", "sum"])

Explanation:

  • Aggregates salary for each department: count, mean, sum.

  • Used heavily for feature aggregation, especially customer-level features.

Create user-level features:

python

CopyEdit

df_agg = df.groupby("customer_id").agg({

    "purchase_amount": ["count", "sum", "mean", "max"],     "days_since_signup": "max"

}).reset_index()

df_agg.columns = ["customer_id", "purchase_count", "purchase_sum", "purchase_mean", "purchase_max", "max_days"]

Explanation:

  • Multi-column groupby aggregation — common in customer churn, credit scoring, risk models.

1⃣1⃣ Rolling Aggregations (Time-series Features)

python

CopyEdit

df = df.sort_values(by="transaction_date") df["rolling_sum"] = df["purchase_amount"].rolling(window=3).sum()

Explanation:

  • Rolling sum over previous 3 rows — creates lagging features for sequence data.

1⃣2⃣ Merge / Joins (Master This)

python

CopyEdit

df_new = pd.merge(df1, df2, how="inner", on="customer_id")

Explanation:

  • SQL-style joins: inner, left, right, outer.

  • Essential for feature engineering from multiple tables.

1⃣3⃣ Sorting

python

CopyEdit

df.sort_values(by=["customer_id", "purchase_date"], ascending=[True, False])

1⃣4⃣ Apply (Row-wise Feature Engineering)

python

CopyEdit

df["net_salary"] = df.apply(lambda row: row.salary - row.tax, axis=1)

Explanation:

  • Applies row-level custom calculations.

  • Avoid overusing for large dataframes — vectorization preferred.

1⃣5⃣ Datetime Engineering (Very common in production ML)

python

CopyEdit

df["transaction_date"] = pd.to_datetime(df["transaction_date"])

df["transaction_month"] = df["transaction_date"].dt.to_period("M")

df["day_of_week"] = df["transaction_date"].dt.dayofweek

df["is_weekend"] = df["day_of_week"].isin([5, 6]).astype(int)

Explanation:

  • Derive temporal patterns: seasonality, weekday vs weekend behavior, month trends.

1⃣6⃣ Outlier Detection (Basic Rule-based)

python

CopyEdit

q1 = df["salary"].quantile(0.25)

q3 = df["salary"].quantile(0.75)

iqr = q3 - q1

lower = q1 - 1.5 * iqr

upper = q3 + 1.5 * iqr

df_filtered = df[(df["salary"] >= lower) & (df["salary"] <= upper)]

Explanation:

  • Standard IQR rule for removing outliers.

1⃣7⃣ Pipeline Concept (For production ML pipelines)

python

CopyEdit

from sklearn.pipeline import Pipeline

from sklearn.preprocessing import StandardScaler

from sklearn.linear_model import LogisticRegression

pipeline = Pipeline([

    ("scaler", StandardScaler()),

    ("model", LogisticRegression())

])

pipeline.fit(X_train, y_train)

Explanation:

  • Full production-ready feature engineering + model pipeline.