DATA 110 Quiz 1 Content

Data

Pieces of Information

Structured Data Types

Tables; most common type of data; usually .csv files; rows are called observations and columns are called features

Semi-structured Data Types

Essentially somewhat structured data with some data; examples include html and xml

Unstructured Data Types

Includes images, videos and audio; text is usually in bag-of-words form

Bag-Of-Words

when python basically counts frequency of words to process the content; doesn’t consider word order and which words actually hold value but can process large amounts of data

Nominal Data

categorical data; uses labels or names; can be numeric or non-numeric (always nominal)

Ordinal Data

same concept as categorical data but follows ranking system; can be numeric or non-numeric

Interval Data

quantitative data for measurement; always numeric

Data Quality Issues

• missing values

• miscoded values

• duplicate values

• outliers

• invalid values

• date/time issues

Personally Identifiable Information (PII)

any info that can identify a specific person, either directly or indirectly; should be protected and remain classified

Direct Identifiers

You can tell who it is immediately; examples include full name, SSN, driver’s license #, passport #, etc.

Indirect (Quasi) Identifiers

combinations of these variables may inadvertently allow you to identify someone; examples include age, birth date

Methods to protect information

• remove column

• mask values

• random IDs

• hash identifiers (ids turned into undecipherable values)

• aggregate

• generalize

• noise

• synthetic data

programming/coding

writing instructions for the computer to complete a task

debugging

the process of finding mistakes and fixing the instructions

vibe coding

a software development workflow where you rely almost entirely on AI to write and refine code while you act as a guiding visionary

pros of vibe coding

• lowers the barrier to entry for those w/minimal coding knowledge

• speeds up the application development process even for experienced coders

cons of vibe coding

• lack of deep understanding

• inconsistent/inefficient code

• hidden security vulnerabilities

compilers/interpreters

computer programmers that actually do the translation

packages

data sci. tools; have built-in types but you can make your own

string method

functions associated with a particular data type

capitalize()

‘professor balan’ → ‘Professor Balan’

lower()

‘Professor Balan’ → ‘professor balan’

upper()

‘Professor Balan’ → ‘PROFESSOR BALAN’

Importing Pandas and Defining a Function

import pandas as pd 
pd.read_csv('Name'.csv)
df_name=pd.read_csv('Name'.csv)
type(df_name)
df_name

Displays a Subset of the table

df_name.head()
df_name.tail()
df_name.sample()

Displays Info about the Table (like number of columns/rows, Python Data Types, etc.)

df_name.info()
df_name.shape()
df_name.dtypes()

Displays Descriptive Stats

df_name.describe()

Selecting a Column

df_name.column
df_name.['column']

Frequency Distribution for Categorical Columns (Counts for ea. Column)

df_name.column_title.value_counts()

Aggregation on Numerical Columns (Stats for ea. Column)

df_name.column_title.min()
df_name.column_title.max()
df_name.column_title.mean()
df_name.column_title.sum()

Selecting/Dropping Columns

df_name.filter(['column_name','column_name_1'])
df_name.drop(columns=['column_name'])

(The functions don’t change the original file. If you want to reuse the update file, you'll need to assign it to a new name.)

#example 
uncbb_filtered=uncbb.filter(['NAME','PTS'])
#creates new file 
uncbb=uncbb.filter(['NAME','PTS'])
#changes original file 

Indexing

x: str='pickles'
print x[0]
print x[len(x)-1] 

allows us to identify the location of a character or group of characters w/in a string

Slicing

x: str='pickles'
print x[2:5]
print x[2:3:5]

allows us to access/extract a substring from a string

Working w/ Columns

df.filter()
df.drop()

Boolean Indexing

#example 
uncbb_guards=uncbb[uncbb['POS']=='G']
print(uncbb_guards)

returns a table w/only the rows for which the corresponding bool is True

• and (&) → both conditions must be True

• or (|) → at least one of the conditions must be True

• not (~) → condition is NOT True

Group-By

getting information about one variable within the confines of another variable (ex. total number of baggage complaints per airline)

#example
df.groupby('airlines')['complaints'].sum()
#the function is determining the sum of complaints based on the airline 
df.groupby('key')['variable_to_analyze'].operation()

Functions for Missing Values

#pulling up the basic info about the dataset
df.info()
#calculates the total number of missing values 
df.isna().sum()
#calculates the percentage of missin values 
df.isna().sum()*100
#drops all missing values 
df.dropna()
#replaces missing values with smth else lik mean or mode or unknown for categorical data
df.fillna()

(Could use filter function but most ideal for smaller data sets)

Pythons Tools for Data Viz

Matplotlib, Seaborn, Altair

Matplotlib

like a car engine for making graphs and charts

import matplotlib.pyplot as plt 

Seaborn

the dashboard that Matplotlib has under the hood

import seaborn as sns 

Function of Data Viz

• depends on whether data is categorical (nominal/ordinal) and numerical (interval)

• should convey the most amount of information in the smallest amount of time with the least amount of ink and space

Frequency Distributions

used to summarize categorical variables; lists unique values of the variable w/ their frequency (value_counts)

#example 
lot_count = df['LotConfig'].value_counts()
print(lot_count)

Bar Charts

#example
plt.figure()
sns.barplot(x=lot_count.index, y=lot_count.values)
plt.ylabel('Number of Homes')
plt.xlabel('Lot Configuration')
plt.xticks(rotation=20)
plt.title('Distribution of Lot Configuration')
plt.show()

Distributions of Single Categorical Variable

Bar Charts and Pie Charts

Distributions of Single Numeric Variable

• can be represented by histograms and box plots

• can be described as center, spread, location and skewness (the shape of the data)

Histograms

• data values are places into distinct, inclusive, equivalent bins

• seaborn chooses bin width based on range of data displayed

• height of the bar represents the # of observations in the bin

#example
plt.figure()
sns.histplot(data=df, x='Sales Price')
plt.title('Distributions of Sales Price')
plt.xlabel('Sales Price ($)')
plt.ylabel('Frequency')
plt.show()

Boxplots

• good for representing 5-Number Summary (Min, Q1, Med, Q2, Max)

• Visualizes IQR of data and the spread

plt.figure()
sns.boxplots(data=df, y='SalesPrice")
plt.title("Distributions of Sales Price")
plt.ylabel('Sales Price ($)')
plt.show()

Displaying Relationships

• For Categorical vs. Numeric Data → Comparative Boxplots

• For Numeric vs. Numeric → Scatterplots

• For Categorical vs. Categorical → Heatmaps

Comparative Boxplots

#example
plt.figure()
sns.boxplot(data=df, x='LotConfig', y='SalePrice')
plt.title('Distribution of Sales Price by Lot Configuration')
plt.xlabel("Lot Configuration")
plt.ylabel("Sales Price ($)")
plt.show()

Scatterplots

#example
plt.figure()
sns.scatterplot(data=df, x='Lot Area', y='SalesPrice')
plt.title('Lot Size vs. Sale Price')
plt.xlabel('Lot Size in Sq. Ft.')
plt.ylabel('Sale Price ($)')
plt.show()

Heatmaps

• assigns colors to values and cells are painted w/such

• visually displays patterns, gradients, and outliers

#example method 1
ct=pd.crosstab(df["BldgType"],df["CentralAir"])

plt.figure()
sns.heatmap(ct,annot=True,fmt="d",cmap="Blues")
plt.title("Heatmap of Building vs Central Air")
plt.xlabel("Central Air")
plt.ylabel('Building Type')
plt.show()

#example method 2 

ct_percent=pd.crosstab(df["BldgType'], df["CentralAir"], normalize="index")*100

sns.heatmap(ct_precent, annot=true, fmt="0.1f", cmap="Blues")
plt.title("Heatmap of Bulding Type vs. Central Air")
plt.xlabel("Central Air")
plt.ylabel("Building Type")
plt.show()

Crosstabulation

• basically indicating overlap

• rows=categories of variable A

• columns=categories of variable B

• cell value= count/percentage of observations w/that combo of both variables A and B

• cell color= the value mapped onto the color scale

Population

All items of interest

• Frequently very large; sometimes infinite

• Described by parameters

Sample

A set of data drawn from the population

• Potentially very large but less than the population

• Described by statistics

Statistical Inference

Process of making an estimate, prediction or decision about a population based on a sample

Sampling

Process of selecting a subset of units from a population

Representative Sample

A selection that reflects the characteristics of the population

Sampling is important for both…

data collection and analysis!

Non-probabilistic Sampling

When the selection process is not fully random

• (Ex.) Twitter Poll

Probabilistic Sampling

When the selection process is truly random

• (Ex.) Gallup poll

Methods of Non-Probabilistic Sampling

• Convenience Sampling

• Volunteer Sampling

• Judgmental Sampling

Sampling Bias

When some participants/samples are more likely to be chosen over others

Types of Sampling Bias

• Undercoverage/Exclusion Bias

• Self-selection/Voluntary response Bias

• Non-response/Participation Bias

• Survivorship Bias

Methods of Probabilistic Sampling

• Simple Random Sampling

• Stratified Random Sampling

Arguments FOR Sampling

• Resource Constraints

• Diminishing Returns

• Faster Iteration and Prototyping

• Algorithm Scaling Constraints

• Spurious Correlation

Arguments AGAINST Sampling

• Lost signal/bias risk

• Selection bias

• Overconfidence/false precision

• Artifacts of data heterogeneity (full data might reveal patterns/whatnot unseen by sample)

Statistical Inference Process

1. Population determined with a mean mew

2. A simple random sample of n elements is selected from the population

3. The sample data provide a value for the sample mean x-bar

4. The value of x-bar is used to make inferences about the value of mew

Sampling Distribution

The distribution of the sample statistic

Sampling Error

Error resulting from using a sample statistic to estimate a popultion parameter

Sampling Distribution of the Mean (X-bar)

• The distribution of the means for samples of the same sample size and variable

• As the sample sizes increase, the closer they are to the population mean and the lower the sampling error

• Standard Deviation= (Stan. Dev. of X) / (Sq. Root of Sample Size)

Central Limit Theorem

If X is a normally distributed, random variable →

Sample Mean = Pop. Mean

Sample Stan. Dev. = (Pop. Stan. Dev.) / (Sq. Root of Sample Size)

Hypothesis Testing

An inference used to make decisions/judgements about the value of a parameter based on a statistic

Null Hypothesis (Ho)

What we currently believe to be true

• Historically accepted/most recent values

• Status Quo

Alternative Hypothesis (Ha)

What we’re trying to evaluate based on the sample data

• Whether or not the data shows a certain thing?

• Challenges the null hypothesis

Hypothesis Testing Process

• State the null and alternative hypotheses

• Define a test statistic

• Compute the value of the test statistic

• Assess the evidence (using p-values)

• State your conclusion (in non-technical terms)

P-Value

Probability of obtaining a value more extreme than the value of the test statistic (where “more extreme” is determined by the form of the alternative hypothesis)

• Establish a significance level a

• Reject the null hypothesis if the p-value is less that a

• Fail to reject the null hypothesis if the p-value is greater than a

Finding p-value with Python

# Perform a one-sample t-test 
#The null hypothesis is that the mean is equal to 90000
#The alternative hypothesis is that the mean is less than 90000
t-statistic, p-value = ttest_1samp(df_admin['Salary'}, popmean=90000, alternative='less')
#Report the p-value 
print(f"The p-value for the one-sample t-test is: {p_value:.4f}")

A/B Testing

• Two-Sample Hypothesis Tests

• Evidence-based/Data-Driven decision making

• Reduces risk

• Continuous improvement

• Measures real user behavior

How does A/B Testing work?

• A/B testing is comparing two scenarios:

  • Option A (often a control version)

  • Option B (alternative version)

• Two separate samples of subjects are selected

• One sample gets Option A and the other gets Option B

• Outcome of interest is measure for each group

The function of A/B testing….

to understand if the observable differences between two groups is really significant and just due to random chance.

• Ho: no diff. in outcome between the groups

• Ha: diff. in outcome between the groups

Artificial Intelligence (AI)

Applies logic and analysis to interpret, support, and automate resulting decisions and actions

Generative AI

Creates new content by learning from patterns in data

Predictive AI

Systems based on machine learning models made for specific tasks based on structured data

• Designed to leverage historical data to perform specific prediction tasks efficiently

• Uses supervised and unsupervised machine learning algorithms

• Relies primarily on structured data

Supervised MLM

Trained on historical data that includes a known (or labelled) outcomes

Classification (A part of Supervised MLM)

Outcomes are binary or categorical

Regression (A part of Supervised MLM)

Outcomes are numeric

Unsupervised MLM

Trained on historical data that does NOT include known (or labelled) outcomes

Reinforcement (A part of Unsupervised MLM)

System learns by trial and error, receiving feedback, and adjusting its actions to maximize success

The goal of supervised learning…

Estimating a function f(x)

Prediction Error

f(x) - f_hat(x)

Components of Error

• Bias Error

• Variance Error

• Irreducible (Random) Error

Total Error Equation

Bias² + Variance + Irreducible Error

Bias

When an algorithm is hindered from learning the true signal from a dataset; may result in underfitting

Low Bias Model

• Fewer assumptions about the functional form of the model

• (Ex.) classification trees, support vector machines, k-nearest neighbor

High Bias Model

• More assumptions about the functional form of the model

• (Ex.) linear regression, logistic regression

Variance

Refers to an algorithm’s sensitivity to a specific set of data; may result in overfitting

Low Variance

• Small changes to the fitted model when the training set changes

• (Ex.) Linear regression, logistic regression

High Variance

• Large changes to the fitted model when the training set changes

• (Ex.) Classification Trees, Support Vector Machines, K-Nearest Neighbor

Overfitting

The model being too complex