Data Types and Tidy Data | data mining!

created by claude

What is a nominal variable?

A categorical variable whose categories have no order (e.g. occupation, zip code).

What is an ordinal variable?

A categorical variable whose categories have an order but unequal gaps between them (e.g. education level).

What's the difference between discrete and continuous numeric variables?

Discrete values are countable (number of children); continuous values can be anything in a range (weight).

What's the quick test for whether a column is numeric or categorical?

Ask whether arithmetic on it means anything. If not, it's categorical, even if it's made of digits.

Application: a dataset has a zip_code column made of digits. Numeric or categorical?

Categorical. The average of two zip codes is meaningless.

Why is it dangerous to leave a categorical code column as int64 when using sklearn?

sklearn treats it as a quantity. For example, k nearest neighbors would compute distances between zip codes as if their magnitudes meant something.

Why can one hot encoding zip codes be a problem?

There are tens of thousands of possible values, so it creates thousands of columns. It's common to group them first (e.g. by region or first 3 digits).

Why is education_num (1–16) ordinal rather than numeric?

The order is real, but the gaps between levels aren't equal.

Application: the Adult dataset's income column holds "<=50K" or ">50K". What type is it?

Categorical (binary). Type depends on how the column is recorded, not what it represents.

Is hours_per_week a categorical variable?

No. It's a numeric (discrete) variable. Categorical and numeric columns get completely different preprocessing, so the word matters.

In tidy data, each variable is a ___, each observation is a ___, and each value is a ___.

Column; row; cell.

How do you figure out what one observation is in a table?

Ask what identifies one row, e.g. station + month, or runner + mile.

Application: a table has columns runner, mile1_split, mile2_split, mile3_split. What is one observation?

One runner running one mile, so each runner gets 3 rows in tidy form.

Application: a weather table has columns station, jan_temp, feb_temp, mar_temp. What is one observation?

One station's temperature in one month. Include every variable that identifies a row, not just the time.

Why does tidy format matter for pandas and sklearn?

These tools operate on columns. Anything you group, filter, or plot by must be a column; in wide format, variables like "which month" are hidden in the headers.

Why is adding new data easier in tidy format?

New data adds rows instead of columns, so the structure and your code stay the same (like INSERT vs ALTER TABLE).

A wide table with Test1, Test2, Test3 columns resembles which database design mistake?

Repeating group columns (like phone1, phone2, phone3), which violate first normal form.

In df.melt(), what does id_vars do?

Names the column(s) kept as is and repeated on every row.

In df.melt(), what do var_name and value_name do?

var_name names the new column holding the old column names; value_name names the new column holding the old cell values.

What's the correct melt() argument for naming the column that holds the old column names: var_name or vars_name?

var_name. Using vars_name throws a TypeError.

Why does melt(id_vars="station", ...) on its own fail?

melt has to be called on the DataFrame: df.melt(...) or pd.melt(df, ...).

Application: fix a table with columns station, jan_temp, feb_temp, mar_temp.

df.melt(id_vars="station", var_name="month", value_name="temp")

In df.melt(), which argument names the values column: var_value or value_name?

value_name. var_value throws a TypeError.

Why avoid spaces in column names like "stock name"?

Spaces break attribute access (df.stock name won't work) and are easy to mistype. Use "stock" or stock_name.