Exploratory Data Analysis | data mining!

created by claude

What is exploratory data analysis (EDA)?

Looking at data before modeling it: checking shape, errors, outliers, and relationships.

What's a CS analogy for EDA?

Profiling before optimizing: you don't model data you haven't looked at.

What's the first step in choosing a plot?

Label each variable as numeric or categorical, then pick the plot for that combination.

Which plot shows one numeric column?

A histogram (or a boxplot).

Which plot shows one categorical column?

A bar chart of counts: df["col"].value_counts().plot.bar()

Which plot compares a numeric variable across categories?

Side by side boxplots, one box per category.

Which plot shows the relationship between two numeric variables?

A scatterplot.

How do you compare two categorical variables?

A crosstab: pd.crosstab(df["a"], df["b"], normalize="index"), or a grouped bar chart.

Why is a scatterplot a bad choice for hours_per_week vs income?

income is categorical, so the plot collapses into two vertical stripes of overlapping dots. Use side by side boxplots.

Why can't you use a boxplot for sex vs income?

A boxplot needs a numeric variable, and both are categorical. Use a crosstab.

Which seaborn function draws a histogram?

sns.histplot (not histoplot).

Application: write the code for the distribution of age.

sns.histplot(data=df, x="age")

Application: write the code to compare hours_per_week between income groups.

sns.boxplot(data=df, x="income", y="hours_per_week")

Application: write the code for age vs hours_per_week.

sns.scatterplot(data=df, x="age", y="hours_per_week")

Application: write the code for the share of >50K earners by sex.

pd.crosstab(df["sex"], df["income"], normalize="index")

In seaborn, why does data="df" fail?

It passes the string "df", not the DataFrame. Use data=df with no quotes.

Is ">50K" a column name in the Adult dataset?

No. It's a value in the income column, so use df["income"]. Run df.columns when unsure.

What is Anscombe's quartet?

Four datasets with nearly identical means, SDs, and correlation that look completely different when plotted.

What does Anscombe's quartet teach?

Summary statistics hide shape, outliers, and curves. Plot first, because the plot tells you which model to use.

Do mean and SD describe a dataset's shape?

No. Skew, multiple peaks, and outliers are invisible to them.

Does reporting a standard deviation mean the data is bell shaped?

No. SD can be computed on any data.

Application: the Adult age histogram has a spike at exactly 90. What's the likely cause?

Capping (top coding): everyone 90 or older was recorded as 90.

Why wouldn't df.describe() reveal the spike at age 90?

It reports only about 8 numbers, and max = 90 looks perfectly normal. Only a plot shows hundreds of people sitting at 90.

What does df.info() catch during EDA?

Column types, such as zip codes loaded as int64.