Data Mining 1

Tabular Data

Data that’s found in a “table” format.

This is the:

Data Science Lifecycle

Tabular Row

A single observation with a group of features.

Tabular Column

A single feature of a group of observations.

pandas

A python library used to manipulate tabular data.
Similar to R.

Dataframe

A table defined by pandas.

Series

A single column of a table as defined by pandas.
Contains an object that holds an index and a value.

Given a series s, s\text{[“a”]} returns:

The value in s associated with index \text{a}.

Given a series s, s\text{[[“a”,”c”]]} returns:

A series that contains the rows in s that are associated with indices \text{a} and \text{c}.

Given a series s, s>0 returns:

A series that contains every index in s and a bool telling whether or not the value is >0.

Given a series s, s[s>0] returns:

A series that contains every row in s whose value is >0.

Dataframe Index

Defines where a particular row will appear in a dataframe.
Does not have to be numeric or unique.

Given a dataframe df, df.index wil return:

Returns every row from df associated with the index.

Given a dataframe df, df.columns returns:

The name of every column in df.

Given a dataframe df, df.shape returns:

The width and height of df.

Given a dataframe df, df.head(n) returns:

The first n rows from df.

Given a dataframe df, df.tail(n) returns:

The last n rows from df.

Given a dataframe df, df.loc[rows, cols] returns:

All rows specified in rows with only the columns specified in cols from df.

The difference between iloc and loc:

iloc uses the position of the column and row, while loc uses the identifier associated with the column and row.

Given a dataframe df, df[0:3] returns:

Every row from df associated with indices 0 to 3 inclusive.

Given a dataframe df, df[cols] returns:

Every row from df with only the columns specified in cols.

Given a dataframe df, df.loc[:9, :] returns:

The first 10 rows from df with every column included.

Given a dataframe df, df[[True,False,True]] returns:

The 1st and 3rd rows from df.

Given a dataframe df, df[(df[“Sex”]==”F”)] returns:

All rows from df where Sex is equal to F.

Given a dataframe df that has no column testCol, df[“testCol”]=testData will:

Add a column called testCol to df with the rows specified in testData.

Given a dataframe df, df[“col”]=df[“col”]-1 will:

Subtract 1 from every row in the column col from df.

Given a dataframe df, df.rename(columns={“oldCol“:newCol}) returns:

A version of df where oldCol is instead called newCol.

Given a dataframe df, df.drop(“oldCol“, axis=”columns”) returns:

A version of df without oldCol.

Given a dataframe df, df.size returns:

The number of individual elements in df.
I.e. Width \times Height.

Given a dataframe df, df.describe() returns:

A general summary of df.

Given a dataframe df, df.sample(n) returns:

A random sample of n rows from df without replacement.
Calling this again will not returns any of the same n rows.

Given a dataframe df, df[“col”].value\_counts() returns:

How many times each value in col occurs.

Given a dataframe df, df[“col”].unique() returns:

The first occurance of every value in col from df.

Given a dataframe df, df[“col”].sort\_values() will:

Sorts the values of col in numeric or alphabetical order.

Given a dataframe df, df.groupby(“Year“) will:

Group the rows in df by the data in Year.

Aggregate Function

A function that looks at more than one row/column at once.

Given a dataframe df, df.groupby(“Year“).agg(sum) returns:

The sum of each unique year in df.

Given a dataframe df, df.groupby(“Year”).size() returns:

The number of rows in df associated with every unique year.

Given a dataframe df, df.groupby(“Year”).count() returns:

Returns the number of values in each column with a non-missing value associated with every unique year.

Given a dataframe df and a command
df.groupby(col).filter(\text{FUNC}), what goes in \text{FUNC}?

A lambda function that returns either True or False.

Given a dataframe df, df.groupby(val) returns an object of type:

\text{DataFrameGroupBy}

Given a dataframe df, df.groupby(val).agg(func) returns an object of type:

DataFrame

Given two dataframes df1 and df2, pd.merge(left=df1, right=df2, left\_on=”col1”, right\_on=”col2”) returns:

A dataframe that merges df1 and df2 based on col1 and col2.

Given a series S, S.map(func) will:

Apply func to each element in S.

Probability

The frequency with which an event occurs in a collection of independent but identical tries.

P(A|B) stands for:

The probability that A happens due to B happening.

P(A|B)=

\frac{P(B|A) \times P(A)}{P(B)}

Census

A complete count or survey of a population.

Population

The complete set of individuals being studied.

Survey

A set of questions or measurments.

Sample

A subset of the population.

Inference / Prediction

The act of drawing conclusions about a population based on a sample.

Sampling Frame

A subset of the population that could possibly be in a sample.

Selection Bias

Systematically excluding particular groups.

Response Bias

Respondants don’t always respond truthfully.

Population Parameter

A number that describes something about the population.

Sample Statistic

An estimate of the number computed on a sample.

Cenertal Limit Theorem

Given a sample is large enough, its distribution will always resemble a normal distribution and will be centered at the population mean.

Sample Space

All possible outcomes for some random event.

Until an experiment occurs a random variable:

Does not hold a value.

Probabilities

The chance that a random variable will take each possible value.

X \sim F(p) means:

The random variable X has a distribution F with a parameter p.

Null Hypothesis

The “default” hypothesis given a scenario.

P-Value

The probability that a given hypothesis could occur.

Standard Deviation (SD) =

\frac{\text{Population SD}}{\sqrt{Sample Size}}

Square Root Law

Increasing the sample size by a factor will decrease the SD by the square root of the factor.

Convergence

When two or more series of values drift towards the same value.

Expectation

The weighted average of the possible values of a random variable.
The weights are the probabilities of the values.

Given X is a random variable and x is a possible value of X, Expectation =

\sum_{\text{all possible x}}{xP(X=x)}

Exploratory Data Analysis (EDA)

The process of iteratively asking and answering more questions about a dataset.

The EDA process:

Question
Investigate
Interpret
Repeat

Model

An idealized representation of a system.

Deterministic Physical Models

Laws that govern how the world works.

Reasons for building models:

To explain complex phenomena.
To make accurate predictions.
To make casual inferences.

Model Evaluation Statistics:

Error
Bias
Variance

Bias

How close a model is to the estimate (on average) to the parameter.

Variance

How spread out the estimate is.

Mean Squared Error (MSE) given \hat \theta =

E((\hat \theta - \theta)²)

Bias given \hat \theta =

E(\hat \theta - \theta)=E(\hat \theta) - \theta

Variance =

\frac{1}{n} \sum^{n}_{i=1}(x_i - \bar x)²

Simple Linear Regression (\hat y) =

\theta_0 + \theta_1 x

The Modeling Process:

Choose a model.
Choose a loss function.
Fit the model.
Evaluate model performance.

Loss Function

Characterizes the cost / error / fit resulting from a particular choice of model and parameters.

Squared Loss / L2 Loss (L(y,\hat y)) =

(y - \hat y)^2

Absolute Loss / L1 Loss (L(y,\hat y)) =

|y - \hat y|

Covariance =

\frac{1}{n} \sum_i (y_i - \bar y)(x_i - \bar x)

Multiple Linear Regression (\hat y) =

\theta_0 + \theta_1 x_1 + … + \theta_p x_p

L2 Vector Norm given an n^{\text{th}} dimensional vector =

\sqrt{ \sum^{n}_{i=1} (x^2_i) }

Span

The set of all possible linear combinations between two columns of a matrix.

Deductive Reasoning

Reasoning based on nature / tradition.

Inductive Reasoning

Reasoning based on the observations made.

\frac{dz}{dx}=

\frac{dz}{\hat{y}} \times \frac{\hat{y}}{dx}

Multiple Linear Regression assumes that…

Every included parameter has no relationship.

Gradient Descent

Finding the lowest value by stepping in either direction based on the slope.

Given gradient descent and a learning rate \alpha, x^{t+1}=

x^t - \alpha \frac{d}{dx}f(x^t)

Gradient Descent stops after…

A fixed number of updates or the change in results is too low.

MSE for Linear Regression (\hat{R}(\theta)) =

\frac{1}{n} \sum ^{n} _{i=1} (y_i - (\theta_0 + \theta_1 x))^2