Intro to Data Analytics

Analytics Overview

Descriptive, predictive, and prescriptive analysis of what happened, what will happen, and what should be done, using data-driven metrics rather than pure inference

Descriptive Analysis

Analytics that summarizes and describes data through wrangling, summary measures, and visualizations without determining causation or next steps

Predictive Analysis

Analytics that uses statistical models to identify associations between variables and predict future outcomes or event likelihood

Prescriptive Analysis

Analytics that uses optimization and simulation algorithms to recommend a course of action among several possible options

Data Wrangling

The process of cleaning and organizing raw data as part of descriptive analysis

Unsupervised Data Mining

A descriptive analytics technique for finding patterns in data without predefined labels

Supervised Data Mining

A predictive analytics technique using labeled data, including regressions, to make predictions

Business Forecasting

A predictive analytics technique for estimating future business outcomes such as sales

Risk and Simulation

A prescriptive analytics technique for modeling uncertainty and exploring possible outcomes

Optimization

A prescriptive analytics technique for selecting the best course of action given constraints

COUNT (Excel)

Excel function that counts only cells containing numerical values

COUNTA (Excel)

Excel function that counts all non-blank cells, numeric or not

COUNTBLANK (Excel)

Excel function that counts empty cells

COUNTIF (Excel)

Excel function that counts cells meeting a specific condition

COUNTIFS (Excel)

Excel function that counts cells meeting one or more conditions

PERCENTILE.INC (Excel)

Excel function used to find quartiles in a dataset using the full range and a percent value

AVERAGE (Excel)

Excel function that finds the mean by adding values and dividing by count

MEDIAN (Excel)

Excel function that finds the exact middle value in a sorted list

STDEV.S (Excel)

Excel function for standard deviation of a sample

STDEV.P (Excel)

Excel function for standard deviation of an entire population

VAR.S (Excel)

Excel function for variance of a sample

VAR.P (Excel)

Excel function for variance of an entire population

AVEDEV (Excel)

Excel function for average (mean absolute) deviation

CORREL (Excel)

Excel function that measures the strength and direction of a linear relationship between two variables on a scale of -1 to 1

YEARFRAC (Excel)

Excel function that calculates the fraction of a year between two dates

Coefficient of Correlation

A measure from -1 to 1 where the sign indicates direction and the magnitude indicates strength of a linear relationship

Strong Negative Correlation

Correlation coefficient between -0.70 and -1.00

Moderate Negative Correlation

Correlation coefficient between -0.30 and -0.69

Weak Negative Correlation

Correlation coefficient between -0.01 and -0.29

No Correlation

Correlation coefficient of 0.00

Empirical Rule

Also called the 68-95-99.7 rule, describing how data distributes within 1, 2, and 3 standard deviations of the mean in a bell-shaped distribution

68-95-99.7 Rule Detail

68% of data falls within 1 SD, 95% within 2 SD, and 99.7% within 3 SD of the mean for a normal distribution

Outlier Rule (Z-score)

A data point is considered an outlier if its z-score falls outside the range of -3 to +3

Mean Absolute Deviation (Sample)

The average of the absolute differences between each value and the sample mean, calculated by taking the difference, its absolute value, then the mean

Interquartile Range (IQR)

A measure of dispersion equal to Q3 minus Q1, representing the range of the middle 50% of data, used to reduce the impact of outliers

Variance vs Standard Deviation

Variance squares the differences from the mean instead of using absolute value

Data Quality

The completeness and correctness of a dataset, checked by looking for missing observations through counting and sorting

Counting and Sorting

Data inspection technique used to verify a dataset is complete, check for missing or wrong values, and confirm the range makes sense

Nominal Data

Categorical data where observations are grouped by label only, with no meaningful order or mathematical operation possible

Ordinal Data

Categorical data with a stronger level of measurement than nominal, where categories follow a meaningful order (e.g., star ratings) but differences between ranks aren't precisely measurable

Interval Data

Numerical data with meaningful, equal differences between values but no true zero, e.g., temperature in Celsius/Fahrenheit, IQ, SAT scores

Ratio Data

Numerical data with a true zero, meaning zero indicates the complete absence of the measured quantity, and negative numbers are not possible

Levels of Measurement Hierarchy

Nominal, then Ordinal, then Interval, then Ratio, with each level allowing more mathematical operations than the last

True Zero

A zero value that means the complete absence of the quantity being measured, present in ratio data but not interval data

Cross Sectional Data

Data recording a characteristic of many subjects at the same point in time

Time Series Data

Data collected over different periods of time (hourly, daily, weekly, annual) focusing on certain people, groups, events, or objects

Unstructured Data

Data that does not fit into defined rows/columns or database structures, such as text or multimedia

Structured Human-Generated Data

Human-created data with defined structure, such as prices, income, or retail sales

Structured Machine-Generated Data

Machine-created data with defined structure, such as sensor data, speed cameras, or web server logs

Unstructured Human-Generated Data

Human-created data without defined structure, such as email, text, or social media

Unstructured Machine-Generated Data

Machine-created data without defined structure, such as satellite images or video data

Big Data (3 V's)

Big data characterized by Volume (amount), Velocity (speed of arrival), and Variety (number of kinds of data)

Histogram

A chart visualizing the frequency distribution of one numerical variable, with bar height representing frequency and no gaps between bars

Bar Chart

A chart used to visualize a categorical variable's frequency distribution

Contingency Table

A table displaying the frequency distribution of two categorical (qualitative or grouping) variables

Stacked Column Chart

A chart displaying the relationship between two categorical variables and one continuous numerical variable

Scatterplot

A chart visualizing the relationship between two numerical variables by plotting points on a Cartesian coordinate system

Relative Frequency

Frequency of a category divided by total number of observations, expressed as a single value or percentage for one category

Relative Frequency Distribution

A complete chart showing the relative frequencies for all categories, not just one

Response Variable

The primary outcome a study aims to measure, explain, or predict

Deterministic Relationship

A relationship where the response variable's value is uniquely determined by predictor variables with no other influencing factors

Stochastic Relationship

A relationship containing randomness or uncertainty, represented as Y = f(X) + epsilon

Y = f(X) + epsilon

Regression model equation where Y is the response variable, f(X) is what the model explains, and epsilon is everything the model misses

Sample Space (S)

The set of all possible outcomes of an experiment, denoted S

Event (Probability)

Any subset of outcomes from a sample space

Exhaustive Events

All possible outcomes of an experiment collectively

Mutually Exclusive Events

Occurrences that cannot happen at the same time

Conditional Probability P(A|B)

The probability of event A occurring given that event B has occurred

Joint Probability P(A∩B)

The probability of both events A and B occurring together

Statistic (Sampling)

A variable whose value depends on the sample drawn

Estimator

A statistic used to estimate a population parameter

Estimate

A particular value produced by an estimator

Sampling Distribution

The probability distribution of a statistic (like the sample mean) derived from all possible samples of a given size

Mean of the Sampling Distribution

The average of all possible sample means, which equals the true population mean

Confidence Interval Formula

CI = x-bar plus or minus z* times (sigma divided by the square root of n)

SQL SELECT Clause

The SQL keyword used to specify which variable names/columns to retrieve

SQL FROM Clause

The SQL keyword used to specify which table(s) to retrieve data from

SQL WHERE Clause

The SQL keyword used to specify selection criteria for filtering records

CRM and Relational Databases

A Customer Relationship Management tool is a finished interface built on top of a relational database, storing linked tables of names, emails, and deals

Data Confidentiality

A data privacy principle ensuring customer data and identity remain private, especially sensitive medical and financial data

Data Transparency

A data privacy principle ensuring data processing and automated decisions are clear, with social and ethical risks understood

Data Accountability

A data privacy principle involving reflective, reasonable, and systematic use and protection of data against unauthorized processing or loss

Common Data Sources

Bureau of Economic Analysis, Bureau of Labor Statistics, Federal Reserve Economic Data (FRED), US Census Bureau, National Climate Data Center, Yahoo Finance, and Zillow

Business Analytics Definition

The process of analyzing data to gain valuable insights and inform business decisions, using data and statistical techniques to improve performance and decision making

Data Science

A field focused on advanced modeling and computer algorithms, distinct from business analytics' focus on business decision-making

Business Analytics Process

Understand the business context, ask the right question, identify the appropriate statistical analysis and computational model, then communicate information

Descriptive Analytics as Business Intelligence

Descriptive analytics, also called Business Intelligence, involves gathering, organizing, tabulating, visualizing, and summarizing data through dashboards and visualization tools

Analytics Value Curve

Prescriptive analytics offers the highest business value and requires the most organizational sophistication, followed by predictive, then descriptive analytics

Data Privacy Definition

A branch of data security concerned with the proper collection, usage, and transmission of data, also called information privacy

Data Privacy Concerns

How data are legally collected and stored, whether and how data are shared with third parties, and how data practices meet regulations

Marriott Data Breach

A 2018 data breach affecting up to 500 million guests, exposing names, addresses, phone numbers, passport numbers, and encrypted credit card data

US Privacy Act

A US law containing important rights and restrictions on data maintained by the government

HIPAA

The US Health Insurance Portability and Accountability Act, which outlines regulations for medical data

Data Ethics Definition

A branch of ethics that studies moral problems related to data, evaluating whether data are used for the right purposes for people and society

Data Ethics Key Considerations

Human first (human interests outweigh institutional/commercial interests) and no biases (algorithms should not absorb or amplify bias)

Artificial Intelligence (AI) Definition

Technology aiming to create machines capable of performing tasks that typically require human intelligence, such as reasoning, problem-solving, natural language understanding, and creativity

Deep Learning

A subset of AI techniques designed for solving complex analytics problems such as image/speech recognition, natural language processing, and multimedia analytics

Generative AI

A subset of AI techniques focused on creating new content such as text, images, programs, and music, exemplified by ChatGPT

Population (Statistics)

All items of interest in an analytics application

Sample (Statistics)

A representative subset of the population used to compute a sample statistic that estimates an unknown population parameter and make inferences