Lecture 1 - Intro to Data Mining

Association Rule

If someone purchases product X as well as product Y

E.g. putting these products close to each other (store-floor planning)

Customer segmentation

Grouping people based on attributes

• Income

• Gender

• Likes

• Height

• Job

Clustering

Used for customer segmentation based on various factors

• Clusters are used for targeted marketing

Data flood

Large amount of data are generated in:

• retail industry

• bank

• clickstream

• social media

• e-commerce

• healthcare

• scientific data

Every click is sent to the server and people can see what you have been browsing

Structured data

Store it into easy to search structures that shows WHAT is happening

• Transaction info (dates, phone numbers, customer names)

Unstructured data

Dumps data somewhere then provides a keyword

• Difficult to search

• Shows the why

• Documents, emails, messages

Semi-structured data

• loosely organized

• meta-level structure that can contain unstructured data

• server logs

• tweets

Searching and Querying

• Keyword based search

• Pattern matching

  • SQL & NOSQL

Aggregation Reports

• Visualization

• Allows us to VISUALLY analyze data

  • Tableau & PowerBI

  • Excel

Data mining

Data mining is a MULTI-DISCIPLINARY field of science and analysis, analyzing large amounts of data to discover meaningful patterns and rules

• finds HIDDEN patterns in a database

Knowledge Discovery

The non-trivial process of identifying

• valid

• novel

• potentially useful

• and understandable patterns in data

Difference between stats and data mining?

Stats - Exploring data

Data Mining - Using statistics to learn and build from data provided

New instances of data (unseen data)

Unseen data → turned into a model → uses training data → customer into classes

Available data(training data)

• Turned into a model → Probability

Supervised learning

• Data is labeled and pre-defined groups (e.g. approved or not approved)

• You can use it for predications( e.g. predicts price)

• Categorical or numerical

What are the data mining models?

Predictive(supervised) and Descriptive(unsupervised)

Predictive(supervised model)

• Uses labeled data

  • classification

  • regression

  • learns from data and predicts what to do next

  • uses training data and use it to predict unseen data

Descriptive(unsupervised model)

• Uses unlabeled date

  • Clustering

  • Association Rule