ACCY 609 Systems Test 1

data driven decision making (DDD)

the practice of basing decisions on the analysis of data, rather than purely intuition

data science

involves principles, processes, and techniques for understanding phenomena via the (automated) analysis of data in order to improve decision making

(1) decision for which discoveries need to be made within data and (2) decisions that repeat especially at massive scale (so decision making can benefit from even small increases in decision-making accuracy based on data analysis

two types of decisions:

CRISP-DM

codification of the data mining process

data mining

the extraction of knowledge from data, via technologies that incorporate these principles

information technology

used to find informative descriptive attributes of entities of interest from a large mass of data

context

formulating data mining solutions and evaluating the results involves thinking carefully about the ____ in which they will be used

business understanding, data understanding, data preparation, modeling, evaluation, deployment

CRISP-DM steps: (6)

classification and class probability estimation

attempt to predict, for each individual in a population, which of a (small) set of classes this individual belongs to

scoring

model applied to an individual that produces a score (instead of a class prediction) representing the probability (or some other quantification of likelihood) that the individual belongs to each class

classification

scoring and ____ are closely related

regression (value estimation)

attempts to estimate or predict, for each individual, the numerical value of some variable for that individual

regression

type of model that could be generated by looking at other, similar individuals in the population

regression procedure

produces a model that, given an individual, estimates the value of the particular variable specific to that individual

whether; how much

classification predicts _____ something will happen, whereas regression predicts ____ something will happen

similarity matching

identify similar individuals based on data known about them; can be used directly to find similar entities

similarity matching

the basis for one of the most popular methods for making product recommendations

clustering

attempts to group individuals in a population together by their similarity, but not drive any specific purpose

clustering

useful in preliminary domain exploration to see which natural groups exist because these groups in turn may suggest other data mining tasks or approaches

questions

clustering is used as an input to decision-making processes focusing on ___

co-occurrence grouping

useful to find associations between entities based on transactions involving them

aka: frequent itemset mining, association rule discovery, and market-basket analysis

market-basket analysis

co-occurrence of products in purchases is a common type of grouping known as ____

co-occurrence grouping

considers the similarity of objects based on their appearing together in transactions

associations

co-occurrence is useful in finding _____ between entities based on transactions involving them

frequency of the co-occurrence and an estimate of how surprising it is

the result of co-occurrence is a description of items that occur together and the description includes statistics on _____ and _____

profiling (behavior description)

attempts to characterize the typical behavior of an individual, group, or population

profiling

often used to establish behavioral norms for anomaly detection applications such as fraud detection and monitoring for intrusions to computer systems

use the degree of mismatch as a suspicion score and issue an alarm if it is too high

link prediction

attempts to predict connections between data items, usually by suggesting that a link should exist, and possibly also estimating the strength of the link

link prediction

common in social networking systems

data reduction

attempts to take a large set of data and replace it with a small set of data that contains much of the important information in the larger set

data reduction

usually involves loss of information

causal modeling

attempts to help us understand what events or actions actually influence others

counterfactual analysis

both experimental and observational methods for casual modeling generally can be viewed as ___ ___ - they attempt to understand what would be the difference between the situations - which cannot both happen - where the "treatment" event were to happen and were not to happen

unsupervised

no target information

supervised

specific target can be provided

technique is given a specific purpose for the grouping - predicting the target

label

the value of the target variable for an individual

clustering, co-occurrence grouping, and profiling

unsupervised data mining tasks (3):

classification, regression, causal modeling

supervised data mining tasks (3):

similarity matching, link prediction, data reduction

data mining tasks that can be either supervised or unsupervised (3):

type of target

two main sub-classes (classification and regression) of of SUPERVISED data mining are distinguished by the _____

classification and regression

two main sub-classes of supervised data mining:

binary

classification has a categorical often ___ target (the customer either purchases or does not)

numeric

regression has a ___ target

numerical prediction

for business applications we often want a ___ ___ over a categorical target

specific quantity

a supervised target variable must be a ___ ___ that will be the focus of the data mining (and for which we can obtain values for some example data)

class probability estimation

model that predicts the probability that something will happen but the underlying target is categorical

leak

situation where a variable collected in historical data gives information on the target variable - information that appears in historical data but it not actually available when the decision has to be made

data preparation

what phase of CRISP-DM should you beware of leaks

modeling

primary place where data mining techniques are applied to the data

evaluation

purpose of this phase is to assess the data mining results rigorously and to gain confidence that they are valid and reliable before moving on

this phase also serves to help ensure that the model satisfies the original business goals

analytics

the extensive use of data and advanced quantitative analysis to drive fact-based decision-making and actions

big data

a collection of data sets that are so large or complex that it is impossible to analyze them with traditional databases and tools

unstrucutred

the variety of big data is ___ (text, voice, video)

Gartner

who created the hype cycle of emerging technologies?

sector specialties

driven by deep industry expertise and tailored solutions

issue driven

start with the business issues and decisions that matter

technology enabled

leverage best-of-breed tools and analytics "assets"

central coordination

network individuals from around the world to field the right team

value approach

bring a commercial sophistication oriented toward creating value

alignment gaps

companies need to avoid ___ ___ between technical capabilities and behavior

technical capability

analytics production:

- data quality
- infrastructure and tools
- data science

behavioral alignment

analytics consumption:

- culture and mental models
- organization & process design
- learning and development
- incentives/rewards

behavioral alignment gap

high technical capability and low behavioral alignment

technical capability gap

high behavioral alignment and low technical capability

data science engineers

software engineers who have particular expertise both in production systems and in data science

exploration

the CRISP-DM cycle is based around ____

business understanding

phase of CRISP-DM:
understand the problem to be solved, design a solution, create sub-problems

data understanding

phase of CRISP-DM:
understand strength and limitations of data, estimate cost/benefit of data source

data preparation

phase of CRISP-DM:
manipulation for better results, removing missing, convert data types

modeling

phase of CRISP-DM:
apply data mining tasks/techniques

evaluation

phase of CRSIP-DM:
assess results, validity, reliability, business goal

deployment

phase of CRISP-DM:
use of models, automatic model building

query

a specific request for a subset of data or for statistics about data, formulated in a technical language and posted to a database system

data warehouse

collects and combines data from across an enterprise, often from multiple transaction-processing systems, each with its own database

-1

MATCH type that requires numbers in lookup range be in descending order

0

MATCH type that returns the row location of the first exact match found

1

MATCH type that requires numbers in lookup range be in ascending order

1; 0

default match type = ___
however, most MATCH function applications use match type = ___

solver

excel tool that optimizes values based on your objectives
- target cell
- changing cells
- constraints

simplex, GRG nonlinear engine, evolutionary solver

solver methods: (3)

simplex

solver method that assumes a linear model

supervised learning

model creation where the model describes a relationship between a set of selected variables (attributes or features) and a predefined variable called the target variable

supervised segmentation

segment the population into subgroups that have different values for the target variable

supervised data mining

Model describes a relationship between a set of selected variables (attributes or features) and a predefined variable (target variable) and estimates the value of the target variable as a function of the attributes

information

a quantity that reduces uncertainty about something

predictive model

a formula for estimating the unknown value of interest: the target

formula could be mathematical or a logical statement like a rule

instance (or example)

represents a fact or a data point

attributes (or features)

fields, columns, variables, or features

feature vector

an instance is sometimes called this because it can be represented as a fixed-length ordered collection (vector) of feature values

row; case

an instance is also referred to as a ___ for a database table or sometimes a ___ in statistics

attributes (or features)

aka:
independent variables
predictors
explanatory variables

model induction

the creation of models from data

induction

term that refers to generalizing from specific cases to general rules (or laws, or truth)

deduction

starts with general rules and specific facts and creates other specific facts from them

training data

input data for the induction algorithm used for inducing the model

labeled data

training data is also called ___ ___ because the value of the target variable is known

pure

homogeneous with respect to the target variable

purity measure

formula created to evaluate how well each attribute splits a set of examples into segments with respect to a chosen target variable

information gain

most common splitting criterion that is based on a purity measure called entropy

measures the change in entropy due to any amount of new information being added

how much entropy has been reduced

entropy

a measure of disorder that can be applied to a set, such as one of our individual segments
(how impure/mixed are the variables within each other)