KMS Big Data midterm

6vs model

volume, velocity, veracity, variety, value, variability

Challenges of Big Data:
High ___, ____, and ____

storage, transmission, and computation

data density

volume

data speed

velocity

data complexity

variety

data validity

veracity

transformation to meaningful results

value

explosive nature of data

variability

One of three major challenges due to limited memory

storage

One of three major challenges due to data analysis

computation

One of three major challenges due to network bandwidth

transmission

the scientific study of the creation, validation and transformation of data to create meaning.

data science

data is used to validate the hypothesis

deductive

____________________ analytics determine the cause of a phenomenon that occurred in the past using queries that focus on the reason behind the event.

Diagnostic

Collecting data without physical contact using sensors that measure electromagnetic radiation reflected by objects. The data collected comes in a grid of pixels with layers (known as bands, where each band is a different slice of the electromagnetic spectrum)

Remote sensing

The more features or dimensions in a dataset, the exponentially more data is required to be generalized accurately

• since to train a good classifier, it has to learn an exhaustive list of possible combinations of features to learn the patterns in the data (instead of purely memorizing the small dataset and overfitting)

• more dimensions/features = more combinations —> requires more data samples

Curse of dimensionality/Hughes phenomenon

A type of research question that is data driven; data brings a conclusion that should hold more broadly

Inductive

Theory driven research question where data validates the hypothesis

Deductive

Mixed model research question that combines inductive and deductive research

abductive

Types of data generated: ____-generated and ____-generated

Human and machine

Categories of data generated

Structures, unstructured, semi-structured

used to capture relationships between different entities and is therefore most often stored in a relational database

structured data

does not conform to a data model or data schema

• makes up 80% of data within any given enterprise

unstructured data

defined level of structure and consistency, but is not relational in nature

• data is hierarchical or graph-based

semi-structured data

data analysis algorithms that “learns” the structure of data to predict categorizations of new data points

machine learning

Algorithm is trained based on some “known/labeled” data and then tested on the unknown database.

Supervised learning

There is no training (prior learning) involved. Algorithm learns data based on data structure/statistics.

Unsupervised learning

The algorithm is trained on a small set of “known/labeled” data and then tested on the larger unlabeled data.

semi-supervised learning

discipline that includes the management of the complete data lifecycle, which encompasses collecting, cleansing, organizing, storing, analyzing and governing data

data analytics

Category of data analytics that enables to answer questions about events that have occurred via contextualized data to generate information; what happened?

descriptive analytics

Category of data analytics that aims to determine the cause of a phenomenon that occurred in the past using queries that focus on the reason behind the event; why?

Diagnostic analytics

category of data analytics that determines the outcome of an event that might occur in the future

predictive analytics

category of data analytics that builds upon the results of data-driven analysis by recommended actions that could be taken

prescriptive analytics

Used by organization to gain insight into its performance by analyzing data generated by its business processes and information systems.

Business intelligence (BI)

metric used to gauge success within a particular business context

Key performance indicators (KPI)

Development and utilization of frameworks that are capable of analyzing large volumes of data from different sources to derive insights from patterns

data analysis

data analysis based on mathematical or statistical criteria

Quantitative

data analysis based on improved human visual perception

Qualitative

A high dimensional data is projected to its low dimensional subspace based on an objective function such that important info in the data is preserved (unsupervised learning method)

Dimensionality reduction

Learning technique for capturing a signal using far fewer measurements than the Nyquist rate, and then reconstructing the full signal accurately. Works when signal is sparse (few important components) and measurements are incoherent (random spread out sampling)

Compressed sensing

Nonlinear learning technique that takes high dimensional data living on a curved, lower dimensional surface (cinnamon roll) and unrolls it into fewer dimensions so the output has fewer features than the input

Manifold learning

A low dimensional data is projected to a high dimensional subspace based on a transformation function (kernels) such that the data that was previously nonlinearly separable are now linearly separable in higher dimensions

Kernel learning

Learning method where the environment is dynamic and constantly learning to maximize the notion of “reward”

Reinforcement learning

Learning method that uses artificial neural networks (ANN) with more than one hidden layer to perform complex learning tasks

Deep learning

• big data analytics can reveal confidential informational about orgs/individuals

• can cause intentional/inadvertent breach

• requires understanding of the nature of data and awareness of data regulations

• can employ special techniques for data tagging and anonymization

privacy

• involves ensuring that the data networks and repositories are sufficiently secured via authentication and authorization mechanisms.

• can involve establishing data access level for different categories of users

security

• refers to information about the source of the data and how it has been processed.

• helps determine the authenticity, quality of data, and is used for auditing purposes.

• stored as metadata corresponding to stages in the data analytics lifecycle

• essential to realize value/confidence of analytic result

provenance

ensures that data and its solution environment are regulated, standardized and evolved in a controlled manner

• how data/metadata is tagged

• policies for management of data privacy and anonymization

• guidelines for data cleansing, filtering, archiving of data sources and analysis results

distinct governance requirements

provide remote environments that can host IT infrastructure for large-scale data storage and processing needs of organizations. May be required due to:

• inadequate in-house hardware resources

• limitations such as computing power, storage resources, and funds that prevent an in-house solution

cloud-based big data solution

Consists of data acquiring, processing, analyzing and repurposing datal; divided into nine stages

big data analytics lifecycle

Stage 1: the creation of a well-defined business case that presents a clear justification, motivation and data analysis goals, which are assessed and approved. Determines resources required and business challenges that will be tackled, and identifies KPIs or SMART goals.

Business case evaluation

Stage 2: identify a wide variety of datasets required for analysis and their sources, which could be internal or external to the enterprise

data identification

Stage 3: data acquired is subjected to automated filtering for removal of corrupted, irrelevant, or valueless data/noise

data acquisition and filtering

Stage 4: data identified as input for analysis may require transformation into a consistent, compatible format for the solution

data extraction

Stage 5: Since invalid, unstructured data can skew and falsify analysis results (especially external data), complex validation rules must be established to remove this data

Data validation and cleansing

Stage 6: methods for information aggregation (combination) across multiple datasets, which involve challenges such as differing semantics and data models as well as large volumes

Data aggregation and representation

Stage 7: employ one or more types of analytics such as data mining, complex statistical analysis techniques, statistical, or mathematical models to depict relationships between variables to discover patterns and anomalies. This stage can be iterative until the appropriate data pattern or correlation is discovered

data analysis

a deductive approach to data analysis where the cause (assumption/hypothesis) is proposed beforehand (data sampling techniques)

confirmatory data analysis

an inductive approach to data analysis where no hypothesis/assumption is made. The data is explored through analysis to develop an understanding of the cause (closely associated with data mining)

exploratory data analysis

Stage 8: techniques that analyze massive amounts of data to graphically communicate the results for effective visual interpretation

Data visualization

Stage 9: Analysis results with application/domain-specific value containing new insights about the patterns and relationships within the data, used to support business decision making

Utilization of analysis results

it refers to automated, software-based techniques that sift through massive datasets to

identify hidden or unknown patterns and trends.

• forms the basis for predictive analytics and BI b/c it discovers patterns to apply to new cases

• also known as pattern recognition or knowledge discovery

Data mining

Statistical analysis method also known as split or bucket testing; compares two versions of an element to determine which is superior based on a pre-defined metric

• current version is called the control version

• modified version is the treatment version

• both are subjected to experiments to determine which is more successful

A/B testing

Statistical analysis method used to determine whether two variable are related and what their relationship is to explain a phenomenon; commonly used in data mining to discover patterns and anomalies

• uses coefficients to measure degree of similarity between two variables from -1 (weak) to 1 (strong)

Correlation

measures the strength and direction of a linear relationship between two numeric variables where:

• r = +1: perfect positive linear relationship (one rises, the other rises).

• r = 0: no linear relationship.

• r = -1: perfect negative linear relationship (one rises, the other falls).

Pearson’s correlation coefficient

Statistical analysis technique that explores how a dependent variable is related to an independent variable within a dataset. Can be used to make predictions about the values of the dependent variable

• linear represents a constant rate of change

• nonlinear represents a variable rate of change

Regression

Assumptions made for ____:

• relationship between x and y is linear

• observations are independent

• y is distributed normally at each x (y should clusters when more samples are taken)

• variance of y at every value x is the same (spread of y is same no matter which x)

Linear regression

Linear regression function

Y = a + Bx

Least-squares function that makes the gap between the actual and predicted value (error) as small as possible by computing a SSE score for choosing the line of best-fit

sum[(actual y - predicted y)²]

Sample y-intercept formula: where the regression line crosses the vertical y-axis/average value of Y when X = 0

a = mean of y - B(mean of x)

Sample slope formula: gives the estimated Y change (B) for each 1 unit increase in X

B = sum(x-meanOfAllX)(y-meanOfAllY)/sum(x-meanOfAllX)

An unsupervised learning technique that linearly projects a high dimensional data to lower dimensions, preserving only the important patterns/trends

• rotation of existing axis to new locations in space to match the shape of the data. Aligns axes with the directions of maximum variance

  • rotation of axes via the eigenvectors (mutually orthogonal) represent directions with maximum variability = Principal components (where PC #1 points to direction of highest variance)

  • each direction/PC is a independent pattern

  • drop minor PC axes/ coordinate description

• fights the curse of dimensionality

Principal component analysis (PCA)

a square grid that summarizes how every variable in a dataset relates to every other variable

• the diagonal elements represent the variance of each individual variable by itself

• Off diagonal elements represent covariance (how two variable change together)

Covariance matrix

Matrix factorization technique that breaks complex matrices into three simpler, fundamental component matrices where:
X = USV^T

• U (Left Singular Vectors): An orthogonal matrix that captures the patterns, structures, and relationships across your samples (rows).

• S (Singular Values): A diagonal matrix containing scaling factors that quantify the importance, weight, or variance captured by each component (eigenvalues)

• V^T (Right Singular Vectors): An orthogonal matrix that captures the relationships and structure across your features/dimensions (columns).

Is used to calculate PCA, and is proven to be the best possible way to compress data in a linear fashion

Singular Vector Decomposition (SVD)

computational complexity of SVD
- N: dimensions, M: samples

O(MN²)

In dimensionality reduction, must consider factors for choosing how many PCs to keep:

• dropping dimensions means losing data, but if remaining components capture the variance, then loss is negligible

• Eigenvalues measure how much variance each PC/eigenvector holds

• final dataset only has __ dimensions/eigenvectors (with the highest variances)

choosing the value of K

Does not distinguish between class labels (since unsupervised), cannot capture nonlinear structures, and high computational complexity of O(min(N³, M³))

PCA cons

Provides data compaction while preserving info at lower dimensions, provides denoising, and improves data separability for better classification

PCA pros

Implicitly assumes data is gaussian (data follows normal distribution, average sits in the middle), finds orthonormal basis for data (axes that form a new coordinate system), sorts PCs/dimensions in order of variance, and discards low significant dimensions/PCs

PCA

Rule used by Naive Bayes to make its final classification

• which class has the highest posterior probability (score) for a given datapoint

• Probability of evidence can be dropped from Bayes Theorem since they are all being affected by the same amount → does not affect the final classification

Maximum A Posterior (MAP) classification rule

Given a training set with features and classes, calculate and store their prior and conditional probabilities in a lookup table. Then take an unseen data instance, look up pre-calculated probabilities, and apply MAP decision rule to assign its final label

• assumes independence across features

Naive Bayes Classification

a set of data examples used for learning, where the target value is known

Training set

a set of data examples used to tune the architecture of a classifier and estimate the error

Validation set

set of data used only to assess the performances of a classifier. Is not used during the training process so the error on the test set provides an accurate estimate of the generalization error

test set

True positive, false negative, false positive, true negative

classification categories

Root mean square error: measures the average magnitude of errors between predicted values and actual observed values

Root mean square error (RSME)

Evaluates the performance of a binary classification model or diagnostic test across all possible decision thresholds using sensitivity and specificity

Receiver operating characteristics (ROC) analysis

Sensitivity: the likelihood of spotting a positive case when presented with one

TP/TP+FN

specificity: the likelihood of spotting a negative case when presented with one

TN/TN+FP

a technique used to evaluate how well a machine learning model will perform on unseen data. Splits data into training and testing set multiple times

cross validation

Split data into k subsets of equal size and use a subset for testing and remainder for training. Rotate out the subsets for testing and training so every subset is the testing set at least once

k-fold cross-validation

A specialized version of K-fold cross-validation where the number of folds equal the total number of data points

• one data point acts as the test set, and the rest act as training data

• Shift the next data point and repeat the process until every single data point has been the test set once

• Average performance score across the rounds

• makes best use of data but very computationally expensive

Leave-one-out cross-validation

Consists of choice of kernel and modularity (any kernel can be used with any kernel algorithm).

• Examples: Kernel PCA, Fisher discriminant analysis, Support vector machines

Kernel methods

Nonlinear mapping using a nonlinear mapping function to project input data into a higher dimensional space for greater separability. Maps inputs to a new multidimensional mathematical space where features of dataset serve as the axes.

• Serves as the foundation behind kernel methods like SVMs

Data in a new feature space

central data structures used by all kernel-based ML algorithms. Converts the training set into a matrix of pairwise vectors (x₁,x₂) similarity scores (not the vectors themselves) calculated by the kernel function and tosses the original raw data

• if the kernel is valid, then K is symmetric positive-definite

Kernel gram matrix

Based on the linear model, but after the data is projected into higher dimensional space

• g(x) is the final score for a single data point used for classification into one class or the other

• phi(x): the non-linear mapping function

• w is the weight vector that determines the orientation and importance of each axis in the feature space (direction that best separates the classes)

• w^T*phi(x): dot product that measures how far the point projects along the direction of the weight vector

• w₀: bias, a scalar value that shifts the decision boundary (hyperplane) from origin

Support vector machines (SVM)

Finds the linear classifier among an infinite number of classifiers with the maximum margin (the orthogonal distance in feature space between the decision boundary and the closest training vector)

• minimizes the norm which maximizes the margin

• weight vector is known as the support vector

SVM

For nonlinearly separable data, use parameter C to introduce flexibility (soft margin) that allows a controlled number of margin violations. Data points with a non-zero multiplier is the boundary/support vector

• Uses Lagrangian multipliers to pick a hyperplane

• need to balance high dimensions for linear separability but also the computational complexity

• Uses Mercer’s theorem: calculate similarities in massive feature spaces without actually computing the transformation by computing lower dimension data with Kernel functions

SVM

3vs model

volume, velocity, variety

imaging data that captures light across hundreds of continuous, narrow spectral bands (wavelengths).

hyperspectral