L 2: Big Data and Machine Learning Stages and Techniques

to collect data

data collection and preparation

-stored in the company (surveys, interviews (qualitative))

-option button (check constraint, dropdown list)

data preparation

data collection and preparation

-resolve formatting issues

-smoothing inconsistent data

determine equivalent value in a dataset

-normalization

-binning

basic data manipulation and techniques

SQL statements to be used- for converting relational data to tabular data

pivot command

basic data manipulation and techniques

-command to tabular data/summarized

-values must be aggregated

data mining

-determine patterns

-extract hidden patterns from potential useful data in a dataset

supervised machine learning techniques

machine learning

-we know what is to be extracted

-we have target/class/dependent variable

unsupervised machine learning techniques

machine learning

-we don't know what will be the result

-market basket and clustering

composition

data visualization

pie chart

comparison

data visualization

bar chart

distribution

data visualization

scatter plot

relationship

data visualization

one variable connected to another variable

Statistics/AI

Provides methods to analyze data and find patterns (e.g., averages, probabilities).

Machine Learning/Pattern Recognition

Uses algorithms to automatically learn and make predictions or group data.

Database Systems

Stores and organizes large amounts of data for easy access.

Data Mining

the middle ground that combines these fields to discover useful insights from large datasets

statistics and AI

Data Mining is at the intersection of these fields:

It uses ___________ and ___________ for theoretical frameworks and data analysis.

machine learning

Data Mining is at the intersection of these fields:

t incorporates ___________________ for automating pattern discovery and predictions.

database systems

Data Mining is at the intersection of these fields:

It depends on ___________________ for efficient data management and retrieval.

Descriptive Analytics

When an aggregate level of understanding of what is going on in a business is required.

Predictive Analytics

When something about the future needs to be predicted or some missing information needs to be approximated

Diagnostic Analytics

When the reason behind a certain observed phenomenon or characteristic needs to be determined

Prescriptive Analytics

When advice is needed regarding what action to take for the best results