Big Data Exam 3

0.0(0)
Studied by 0 people
call kaiCall Kai
learnLearn
examPractice Test
spaced repetitionSpaced Repetition
heart puzzleMatch
flashcardsFlashcards
GameKnowt Play
Card Sorting

1/200

encourage image

There's no tags or description

Looks like no tags are added yet.

201 Terms

1
New cards

What is data mining?

The nontrivial process of identifying valid, novel, potentially useful, and ultimately understandable patterns in data stored in structured databases.

2
New cards

What does “valid” mean in data mining?

The pattern holds up when checked and applies beyond the examples used to discover it.

3
New cards

What does “novel” mean in data mining?

The pattern is new or previously unknown.

4
New cards

What does “potentially useful” mean?

The discovered pattern could help someone make a decision or take action.

5
New cards

What does “understandable” mean?

People can interpret the pattern and its meaning.

6
New cards

What does “nontrivial” mean?

Discovering the pattern requires meaningful analysis.

7
New cards

Why has data mining become more popular?

Stronger competition, recognition of data’s value, more available quality data, integrated data warehouses, and greater processing and storage capabilities at lower costs.

8
New cards

Which fields contribute to data mining?

Statistics, artificial intelligence, machine learning, pattern recognition, databases, mathematical modeling, management science, and information systems.

9
New cards

What is the most critical ingredient in data mining?

Data.

10
New cards

Does data mining always require a data warehouse?

A data warehouse is often a source, but data mining can use other data sources.

11
New cards

Who can perform data mining?

Business users and other end users can use data mining tools. Creative thinking and usable software matter.

12
New cards

What is data?

A collection of facts usually obtained as the result of experiences, observations, or experiments. It can include numbers, words, and images.

13
New cards

What is structured data?

Data organized into defined fields, such as database rows and columns.

14
New cards

What are examples of unstructured or semi-structured data?

Text, multimedia, and HTML/XML.

15
New cards

What is nominal data?

Categories with no natural order, such as college majors or product brands.

16
New cards

What is ordinal data?

Categories with a meaningful order, such as low, medium, and high satisfaction.

17
New cards

What is interval data?

Numerical data with equal intervals but no true zero, such as temperature in Fahrenheit.

18
New cards

What is ratio data?

Numerical data with equal intervals and a true zero, such as number of purchases or weight.

19
New cards

What is a pattern?

A mathematical, numeric and/or symbolic, relationship among data items.

20
New cards

What is association?

Finding items or events that occur together, such as products frequently purchased together.

21
New cards

What is support?

How often the items occur together in the entire dataset.

22
New cards

What is the support formula for A → B?

Number of transactions containing BOTH A and B divided by the TOTAL number of transactions.

23
New cards

Do transactions containing neither item count in support’s denominator?

Yes. Support uses all transactions in the dataset.

24
New cards

What is confidence?

For A → B, the proportion of transactions containing A that also contain B.

25
New cards

What is the confidence formula for A → B?

Number of transactions containing BOTH A and B divided by the number containing A.

26
New cards

Which item determines confidence’s denominator?

The FIRST item in the rule. For coffee → muffin, divide by coffee transactions. For muffin → coffee, divide by muffin transactions.

27
New cards

What is lift?

The response rate in a selected group divided by the overall response rate. It measures how many times as high the selected group’s rate is.

28
New cards

What is the association-rule lift formula for A → B?

Confidence for A → B divided by the overall rate of buying B. The overall B rate equals B transactions divided by all transactions.

29
New cards

What does lift greater than 1 mean?

The selected group is more likely to buy the second item than shoppers overall.

30
New cards

What does lift equal to 1 mean?

The selected group buys the second item at the same rate as shoppers overall.

31
New cards

What does lift less than 1 mean?

The selected group is less likely to buy the second item than shoppers overall.

32
New cards

If target response is 20% and overall response is 5%, what is lift?

20% divided by 5% equals 4. The target group responds at four times the overall rate.

33
New cards

There are 100 transactions, 20 with coffee, 20 with muffins, and 10 with both. What is support?

10 divided by 100 equals 10%.

34
New cards

In that example, what is confidence for coffee → muffin?

10 divided by 20 equals 50%.

35
New cards

In that example, what is lift for coffee → muffin?

The overall muffin rate is 20%. Divide 50% confidence by 20% to get a lift of 2.5.

36
New cards

Why can’t you add the A count, B count, and BOTH count to get total transactions?

The groups overlap. Transactions containing both are already included in the A and B counts. Transactions containing neither may also exist.

37
New cards

What is classification?

Supervised learning that learns from known outcomes and predicts a category, such as renew/cancel or fraudulent/legitimate.

38
New cards

What is regression?

A statistical process for estimating relationships between one dependent variable and one or more independent variables. It is widely used to predict numerical outcomes.

39
New cards

What is clustering?

Unsupervised learning that discovers natural groups of similar records. It is also called segmentation.

40
New cards

What is time-series forecasting?

Using patterns in data collected over time to predict future values, such as next month’s sales.

41
New cards

What is supervised learning?

Learning from training data that includes known outcomes. Classification and regression are examples.

42
New cards

What is unsupervised learning?

Discovering patterns or groups without a predefined outcome variable to learn. Clustering is an example.

43
New cards

Classification versus clustering?

Classification predicts membership in known categories. Clustering discovers groups from similarities.

44
New cards

Classification versus regression?

Classification predicts a category, such as sunny/rainy/snowy. Regression predicts a number, such as tomorrow’s temperature.

45
New cards

What is hypothesis-driven data mining?

Beginning with an idea or proposed relationship and analyzing data to test it.

46
New cards

What is discovery-driven data mining?

Exploring data to discover patterns without starting with a specific hypothesis to test.

47
New cards

What is churn analysis?

Identifying customers likely to leave so a business can improve customer retention.

48
New cards

What is cross-selling?

Encouraging the purchase of an additional related product, such as a case with a phone.

49
New cards

What is up-selling?

Encouraging a higher-value version of a product, such as a more expensive phone model.

50
New cards

What are common business applications of data mining?

Customer retention, campaign optimization, fraud detection, loan evaluation, inventory planning, equipment-failure prediction, insurance-claim forecasting, treatment-outcome prediction, and pricing.

51
New cards

What does CRISP-DM stand for?

Cross-Industry Standard Process for Data Mining.

52
New cards

What are the six CRISP-DM steps in order?

Business Understanding → Data Understanding → Data Preparation → Model Building → Testing and Evaluation → Deployment.

53
New cards

What happens during Business Understanding?

Define business objectives, assess resources and constraints, establish data mining goals, and develop a project plan.

54
New cards

What happens during Data Understanding?

Collect and explore data, examine its properties, and check quality, completeness, and missing values.

55
New cards

What happens during Data Preparation?

Consolidate, clean, transform, and reduce data to make it suitable for analysis.

56
New cards

Which phases account for approximately 85% of project time in the slides?

Data Understanding and Data Preparation together.

57
New cards

Is CRISP-DM a strictly one-way process?

The process is repetitive and experimental. Findings may require returning to earlier phases.

58
New cards

What is Data Consolidation?

Collecting, selecting, and integrating data, such as combining online and in-store customer records.

59
New cards

What is Data Cleaning?

Imputing missing values, reducing noise, and eliminating inconsistencies.

60
New cards

What does impute mean?

Fill in a missing value with an estimated value.

61
New cards

Data Understanding versus Data Cleaning?

Finding missing values is part of understanding data quality. Filling in those values is cleaning during Data Preparation.

62
New cards

What is Data Transformation?

Normalizing data, discretizing or aggregating data, and constructing new attributes.

63
New cards

What does normalize mean in data preparation?

Rescale numerical values to make their scales comparable.

64
New cards

What does discretize mean?

Convert numerical values into categories or ranges, such as converting ages into age groups.

65
New cards

What does aggregate mean?

Combine detailed records into summaries, such as adding daily sales into monthly totals.

66
New cards

What does construct new attributes mean?

Create new variables from existing data, such as calculating total customer spending.

67
New cards

What is Data Reduction?

The slide lists reducing the number of variables, reducing the number of cases, and balancing skewed data.

68
New cards

What are variables and cases?

Variables are columns or attributes. Cases are rows or individual observations.

69
New cards

What happens during Modeling?

Select modeling techniques, establish test scenarios, build models using prepared data, and assess the models.

70
New cards

What happens during Evaluation?

Judge results against business objectives and make the go/no-go decision about deployment. New requirements may emerge.

71
New cards

What happens during Deployment?

Put findings or models into use, plan implementation and maintenance, monitor performance, and review lessons learned.

72
New cards

What can deployment look like?

A simple report or a repeatable data mining process across an organization.

73
New cards

What does KDD stand for?

Knowledge Discovery in Databases.

74
New cards

What is the KDD sequence?

Selection → Cleaning → Transformation → Data Mining → Interpretation/Evaluation → Knowledge.

75
New cards

How does data mining relate to KDD?

Data mining is one stage within the broader knowledge-discovery process.

76
New cards

What does SEMMA stand for, in order?

Sample → Explore → Modify → Model → Assess.

77
New cards

What happens in SEMMA’s Sample step?

Generate a representative sample of the data.

78
New cards

What happens in SEMMA’s Explore step?

Use visualizations and basic descriptions to examine the data.

79
New cards

What happens in SEMMA’s Modify step?

Select variables and transform their representations.

80
New cards

What happens in SEMMA’s Model step?

Apply statistical and machine-learning models.

81
New cards

What happens in SEMMA’s Assess step?

Evaluate the accuracy and usefulness of the models.

82
New cards

Explore versus Modify in SEMMA?

Explore means examining data with charts and descriptions. Modify means selecting or changing variables.

83
New cards

What does slide 31 call the most frequently used data mining method?

Classification.

84
New cards

What is predictive accuracy?

Hit rate, or how often a model’s predictions are correct.

85
New cards

What does speed measure?

How quickly a model can be built and make predictions.

86
New cards

What is robustness?

A model’s ability to make reasonably accurate predictions with noisy, missing, or erroneous data.

87
New cards

What is scalability?

The ability to construct a prediction model efficiently with a large dataset.

88
New cards

What is interpretability?

Transparency and ease of understanding the model.

89
New cards

Robustness versus scalability?

Robustness concerns messy data. Scalability concerns large amounts of data.

90
New cards

What is a Type I error?

A false positive, or false alarm. Example: flagging a legitimate transaction as fraud when fraud is the positive category.

91
New cards

What is a Type II error?

A false negative, or missed detection. Example: labeling a fraudulent transaction as legitimate when fraud is the positive category.

92
New cards

Which is worse, a false positive or false negative?

It depends on the consequences and costs in that situation.

93
New cards

What is the simple split or holdout method?

Split data into mutually exclusive training and testing sets. The slide gives approximately 70% training and 30% testing.

94
New cards

What does mutually exclusive mean for training and testing sets?

An individual record belongs to only one of the sets.

95
New cards

What is training data used for?

Teaching the model patterns and relationships.

96
New cards

What is testing data used for?

Assessing predictions on records the model did not train on.

97
New cards

What three-way split does the slide give for artificial neural networks?

Approximately 60% training, 20% validation, and 20% testing.

98
New cards

What is validation data used for?

Guiding model tuning and selection before the final test assessment.

99
New cards

Which classification techniques are listed?

Decision trees, statistical analysis, neural networks, support vector machines, and Bayesian classifiers.

100
New cards

What is a decision tree?

A model that uses divide and conquer, repeatedly splitting data based on attributes to reach predictions.