1/200
Looks like no tags are added yet.
What is data mining?
The nontrivial process of identifying valid, novel, potentially useful, and ultimately understandable patterns in data stored in structured databases.
What does “valid” mean in data mining?
The pattern holds up when checked and applies beyond the examples used to discover it.
What does “novel” mean in data mining?
The pattern is new or previously unknown.
What does “potentially useful” mean?
The discovered pattern could help someone make a decision or take action.
What does “understandable” mean?
People can interpret the pattern and its meaning.
What does “nontrivial” mean?
Discovering the pattern requires meaningful analysis.
Why has data mining become more popular?
Stronger competition, recognition of data’s value, more available quality data, integrated data warehouses, and greater processing and storage capabilities at lower costs.
Which fields contribute to data mining?
Statistics, artificial intelligence, machine learning, pattern recognition, databases, mathematical modeling, management science, and information systems.
What is the most critical ingredient in data mining?
Data.
Does data mining always require a data warehouse?
A data warehouse is often a source, but data mining can use other data sources.
Who can perform data mining?
Business users and other end users can use data mining tools. Creative thinking and usable software matter.
What is data?
A collection of facts usually obtained as the result of experiences, observations, or experiments. It can include numbers, words, and images.
What is structured data?
Data organized into defined fields, such as database rows and columns.
What are examples of unstructured or semi-structured data?
Text, multimedia, and HTML/XML.
What is nominal data?
Categories with no natural order, such as college majors or product brands.
What is ordinal data?
Categories with a meaningful order, such as low, medium, and high satisfaction.
What is interval data?
Numerical data with equal intervals but no true zero, such as temperature in Fahrenheit.
What is ratio data?
Numerical data with equal intervals and a true zero, such as number of purchases or weight.
What is a pattern?
A mathematical, numeric and/or symbolic, relationship among data items.
What is association?
Finding items or events that occur together, such as products frequently purchased together.
What is support?
How often the items occur together in the entire dataset.
What is the support formula for A → B?
Number of transactions containing BOTH A and B divided by the TOTAL number of transactions.
Do transactions containing neither item count in support’s denominator?
Yes. Support uses all transactions in the dataset.
What is confidence?
For A → B, the proportion of transactions containing A that also contain B.
What is the confidence formula for A → B?
Number of transactions containing BOTH A and B divided by the number containing A.
Which item determines confidence’s denominator?
The FIRST item in the rule. For coffee → muffin, divide by coffee transactions. For muffin → coffee, divide by muffin transactions.
What is lift?
The response rate in a selected group divided by the overall response rate. It measures how many times as high the selected group’s rate is.
What is the association-rule lift formula for A → B?
Confidence for A → B divided by the overall rate of buying B. The overall B rate equals B transactions divided by all transactions.
What does lift greater than 1 mean?
The selected group is more likely to buy the second item than shoppers overall.
What does lift equal to 1 mean?
The selected group buys the second item at the same rate as shoppers overall.
What does lift less than 1 mean?
The selected group is less likely to buy the second item than shoppers overall.
If target response is 20% and overall response is 5%, what is lift?
20% divided by 5% equals 4. The target group responds at four times the overall rate.
There are 100 transactions, 20 with coffee, 20 with muffins, and 10 with both. What is support?
10 divided by 100 equals 10%.
In that example, what is confidence for coffee → muffin?
10 divided by 20 equals 50%.
In that example, what is lift for coffee → muffin?
The overall muffin rate is 20%. Divide 50% confidence by 20% to get a lift of 2.5.
Why can’t you add the A count, B count, and BOTH count to get total transactions?
The groups overlap. Transactions containing both are already included in the A and B counts. Transactions containing neither may also exist.
What is classification?
Supervised learning that learns from known outcomes and predicts a category, such as renew/cancel or fraudulent/legitimate.
What is regression?
A statistical process for estimating relationships between one dependent variable and one or more independent variables. It is widely used to predict numerical outcomes.
What is clustering?
Unsupervised learning that discovers natural groups of similar records. It is also called segmentation.
What is time-series forecasting?
Using patterns in data collected over time to predict future values, such as next month’s sales.
What is supervised learning?
Learning from training data that includes known outcomes. Classification and regression are examples.
What is unsupervised learning?
Discovering patterns or groups without a predefined outcome variable to learn. Clustering is an example.
Classification versus clustering?
Classification predicts membership in known categories. Clustering discovers groups from similarities.
Classification versus regression?
Classification predicts a category, such as sunny/rainy/snowy. Regression predicts a number, such as tomorrow’s temperature.
What is hypothesis-driven data mining?
Beginning with an idea or proposed relationship and analyzing data to test it.
What is discovery-driven data mining?
Exploring data to discover patterns without starting with a specific hypothesis to test.
What is churn analysis?
Identifying customers likely to leave so a business can improve customer retention.
What is cross-selling?
Encouraging the purchase of an additional related product, such as a case with a phone.
What is up-selling?
Encouraging a higher-value version of a product, such as a more expensive phone model.
What are common business applications of data mining?
Customer retention, campaign optimization, fraud detection, loan evaluation, inventory planning, equipment-failure prediction, insurance-claim forecasting, treatment-outcome prediction, and pricing.
What does CRISP-DM stand for?
Cross-Industry Standard Process for Data Mining.
What are the six CRISP-DM steps in order?
Business Understanding → Data Understanding → Data Preparation → Model Building → Testing and Evaluation → Deployment.
What happens during Business Understanding?
Define business objectives, assess resources and constraints, establish data mining goals, and develop a project plan.
What happens during Data Understanding?
Collect and explore data, examine its properties, and check quality, completeness, and missing values.
What happens during Data Preparation?
Consolidate, clean, transform, and reduce data to make it suitable for analysis.
Which phases account for approximately 85% of project time in the slides?
Data Understanding and Data Preparation together.
Is CRISP-DM a strictly one-way process?
The process is repetitive and experimental. Findings may require returning to earlier phases.
What is Data Consolidation?
Collecting, selecting, and integrating data, such as combining online and in-store customer records.
What is Data Cleaning?
Imputing missing values, reducing noise, and eliminating inconsistencies.
What does impute mean?
Fill in a missing value with an estimated value.
Data Understanding versus Data Cleaning?
Finding missing values is part of understanding data quality. Filling in those values is cleaning during Data Preparation.
What is Data Transformation?
Normalizing data, discretizing or aggregating data, and constructing new attributes.
What does normalize mean in data preparation?
Rescale numerical values to make their scales comparable.
What does discretize mean?
Convert numerical values into categories or ranges, such as converting ages into age groups.
What does aggregate mean?
Combine detailed records into summaries, such as adding daily sales into monthly totals.
What does construct new attributes mean?
Create new variables from existing data, such as calculating total customer spending.
What is Data Reduction?
The slide lists reducing the number of variables, reducing the number of cases, and balancing skewed data.
What are variables and cases?
Variables are columns or attributes. Cases are rows or individual observations.
What happens during Modeling?
Select modeling techniques, establish test scenarios, build models using prepared data, and assess the models.
What happens during Evaluation?
Judge results against business objectives and make the go/no-go decision about deployment. New requirements may emerge.
What happens during Deployment?
Put findings or models into use, plan implementation and maintenance, monitor performance, and review lessons learned.
What can deployment look like?
A simple report or a repeatable data mining process across an organization.
What does KDD stand for?
Knowledge Discovery in Databases.
What is the KDD sequence?
Selection → Cleaning → Transformation → Data Mining → Interpretation/Evaluation → Knowledge.
How does data mining relate to KDD?
Data mining is one stage within the broader knowledge-discovery process.
What does SEMMA stand for, in order?
Sample → Explore → Modify → Model → Assess.
What happens in SEMMA’s Sample step?
Generate a representative sample of the data.
What happens in SEMMA’s Explore step?
Use visualizations and basic descriptions to examine the data.
What happens in SEMMA’s Modify step?
Select variables and transform their representations.
What happens in SEMMA’s Model step?
Apply statistical and machine-learning models.
What happens in SEMMA’s Assess step?
Evaluate the accuracy and usefulness of the models.
Explore versus Modify in SEMMA?
Explore means examining data with charts and descriptions. Modify means selecting or changing variables.
What does slide 31 call the most frequently used data mining method?
Classification.
What is predictive accuracy?
Hit rate, or how often a model’s predictions are correct.
What does speed measure?
How quickly a model can be built and make predictions.
What is robustness?
A model’s ability to make reasonably accurate predictions with noisy, missing, or erroneous data.
What is scalability?
The ability to construct a prediction model efficiently with a large dataset.
What is interpretability?
Transparency and ease of understanding the model.
Robustness versus scalability?
Robustness concerns messy data. Scalability concerns large amounts of data.
What is a Type I error?
A false positive, or false alarm. Example: flagging a legitimate transaction as fraud when fraud is the positive category.
What is a Type II error?
A false negative, or missed detection. Example: labeling a fraudulent transaction as legitimate when fraud is the positive category.
Which is worse, a false positive or false negative?
It depends on the consequences and costs in that situation.
What is the simple split or holdout method?
Split data into mutually exclusive training and testing sets. The slide gives approximately 70% training and 30% testing.
What does mutually exclusive mean for training and testing sets?
An individual record belongs to only one of the sets.
What is training data used for?
Teaching the model patterns and relationships.
What is testing data used for?
Assessing predictions on records the model did not train on.
What three-way split does the slide give for artificial neural networks?
Approximately 60% training, 20% validation, and 20% testing.
What is validation data used for?
Guiding model tuning and selection before the final test assessment.
Which classification techniques are listed?
Decision trees, statistical analysis, neural networks, support vector machines, and Bayesian classifiers.
What is a decision tree?
A model that uses divide and conquer, repeatedly splitting data based on attributes to reach predictions.