Data Analytics

Data Analytics

Using data to find useful patterns, relationships, and information to help make decisions

Data mining

Searching large amounts of data for useful information or patterns

CRISP-DM + name them

Cross-industry standard process of data mining. There’s 6.
1. Business understanding
2. Data understanding
3. Data preparation
4. Modelling
5. Evaluation
6. Deployment

Business understanding

what problems are we trying to accomplish and what would success look like

Data understanding

What information do I need to answer the question and how will I obtain it?

Data preparation

Cleaning up the data, making sure there’s no missing values, outliers, or inconsistent values

Modelling

Method for using existing information to describe or predict an outcome

Evaluation

Is the model actually good enough for the purpose we built for it? Evaluation needs to go back to what the original objectives were and whether the model meets those expectations.

Deployment

Using the analysis in the real world

Supervised vs. unsupervised learning

Supervised = you have a specific outcome you want the model to predict.

Unsupervised = there’s no predetermined correct outcome

Value estimation

Predict a number

Classification

Put something into a category

Clustering

Find natural groups

Co-occurrence grouping

Find things that happen together

Similarity matching

Find similar cases to other cases

Profiling

Describing typical characteristics of a group or category based on data analysis.

Casual modelling

Investigating what influences an outcome

Data reduction

reduce the amount of data while preserving useful information

Feature selecting

Deciding which variables are actually useful

Population

Everybody/everything you actually care about

Sample

The smaller group you actually care about

Observation

One individual case in your data. I.e. a student

Variable/attribute

Characteristics recorded in the observation. i.e. age, gender, program, GPA, etc.

Bias

A systemic problem that causes your sample or results to lean in a particular direction

Selection bias

A type of bias that occurs when the sample is not representative of the population, often due to non-random selection methods.

Observation study

You observe what happened in. You do not control who receives what.

Controlled experiment

You control a variable. Treatment groups are formed by random assignment to assess the effect of the treatment.

A/B Testing

Group A sees one ad, group B sees the other, compare behaviour.

Administrative data

Information an organization already collects while doing normal business. Customer records, purchase history, student records, employee date, etc.

Surveys

Methods for gathering data from respondents, often using questionnaires or interviews to collect opinions, behaviors, or characteristics.

Nominal vs. ordinal

Nominal = categories with no mathematical order. i.e. art, business, science
Ordinal = categories with a defined order, such as rankings or scales. i.e. poor, fair, good, excellent.

Interval vs. ratio

Interval = numerical data with equal intervals but no true zero, e.g., temperature in Celsius.
Ratio = numerical data with equal intervals and a true zero, allowing for comparison of magnitude, e.g., weight or height.

Data quality

Is this data actually what I think it is?

Unreasonable values

Asking if the largest or smallest value are plausible. Figure out if the data is possible or not.

Data dictionary

The instruction manual for your dataset. Should explain what each variable means and what its code mean.

What does VLOOKUP do in excel?

VLOOKUP is a function in Excel that allows users to search for a value in the first column of a table and return a value in the same row from a specified column. VLOOKUP(value, table, column, match).

What does FALSE mean in Excel?

Exact match

What does TRUE mean in Excel?

Approximate match

What does the IF function do in Excel?

The IF function in Excel evaluates a specified condition and returns one value if the condition is true and another value if it is false.

Sorting vs. filtering

Sorting = arranges data in an order. i.e. smallest to largest, A-Z, etc.

Filtering = Temporarily showing observations under specific conditions. i.e. business students only, female students only, etc.

Frequency disribution

is a summary of how often each value occurs in a dataset.

Frequency. relative frequency + what’s the formula?

Frequency = Number of observations of a particular value in a dataset.

Relative frequency = proportion/percentage of observation.

Formula = frequency/total observations.

Normal shape, right-skewed, and left skewed

Normal shape/bell curve = Observations in the middle with balances tails on each side.

Right-skewed = long tail points right. Caused by unusually high values.

Left-skewed - long tail points left. Caused by unusually low values.

Mean

Average

Median

Middle value

Mode

Most frequently occurring value

Percentiles

What % of observations fall below a particular value? i.e. 90th percentile salary = 90% of salaries are below that amount.

Range

The difference between the highest and lowest values in a dataset, indicating the spread of the data. Range = Maximum - Minimum.

Variability

How spread out are the values?

Standard deviation

A measure of the amount of variation or spread in a set of values. How much do the values deviate from the mean.

What is a PivotTable in Excel?

A PivotTable in Excel allows users to summarize, analyze, and present large datasets in a table.

What is COUNT in excel?

The number of cells that contain numeric values in a specified range.

What is AVERAGE in Excel?

Average total

SUM?

Total

Rows % vs. Column % in Excel?

Rows % represents the percentage of each row's total relative to the grand total, while Column % calculates the percentage of each column's total against the grand total.

Aggression

Combining observations into larger groups. i.e. all canadian provinces → canada

Disaggression

Breaking groups into smaller subgroups. i.e gender → gender + program → gender + program + province.

Simpson’s paradox

A pattern in the overall data changes or even reverses when you break the data into meaningful groups.