big data midterm

issues with using google to track flu

correlation not causation, omitted variables, google trends is updated and changed

data as oil

drives growth and change, important resource that comapnies can use

ways to estimate big data impacts

case studies on its uses, macro regression comparing growth with/out it, micro regression comparing companies who do and dont use it

solow productivty paradox

computers are everywhere but productivity statstics.

general purpose technology

technolgy used in many places, improving over time and faciltate innovation

big data hubris

beleving that big data is a substitute rather than compliment to traditional data colelction and analysis

Deep learning

neural network with many layers

choi varian google trends study

added google trends results to a regression to see if searches correlated to increases in purchases

reasons for lack of immediate growth from big data

predictions were false, mismeasurement of impacts, concentrated gains in specifc areas, implementation causing lags

presenting data results

use storytelling and implications of data to make the message stick in the mind

5 v’s of big data

Volume, Velocity, Variety, Veracity, Value

times big data is a complement to small data

small data explains specific reasons for many customers behaviour, showing whether small data behaviour generalises across a population

bigrams

all the words in a text are paired with the word next to it to see the most common pairs of words

how to track sentiments of text

using a senitment analyser, finding the positive, negative and neutral senitment of the text

in sample vs out of sample accuracy

in sample predicts the cases that were used to train the model, out of sample predicts new data

k folds training in cross validation

the sample is folded into k number of sub samples, the model is trained on k-1 sub samples and the unused sample is predicted. this is repeated, predicting all of the sub samples

algorithmic discrimination

algorithims unintentionally discriminating based on biased training data, or due to unforseen correlations in the data

what is the score for binary variables in cross validation

average accuracy of predictions

classifcation tree

predicts the classifcation or value of a variable by dividing it based on value of its variables

advantage of classifcation trees

captures non linear relationships by splitting the model into intervals, automatically sets up the right functional forms

gini impurity

measures the chance that a variable is labelled incorrectly by our model. 0 is perfect predictions, 0.5 is random picking

how are categorical variables treated in a regression

should always be split into multiple dummies, even if they are numerical

when making predictions, which model should the importanve and shap values be run on,

the highest cv model, to ensure the predictions are closest to accurate as possible

bag of trees

creating trees using bootstrapped samples and aggregating the outcomes for all of the trees

random forest

creates more trees from bootstrapping but randomly excludes some variables

bag of trees vs random forest

bag of trees uses the same variables every time, random forest randomises some of the variables

bootstrapping

creating more samples from a sample by resampling with replacement

hard voting in bagging trees

the prediction is decided by a majority vote of categories

soft voting in bagging trees

averages the predicted probablity that a class is chosen by each of the models, then chooses the class with the highest average

ensemble models

many models are applied to a dataset and the results are averaged

gradient boosting

a tree is created, its residual errors are calculated and then new trees are created, targeting the residual errors of the first model, all the trees are used as an ensemble

support vector machine

finds the optimal line that divides two objects into different classes

surrogate model

running a regression on the estimations given by a tree as the regression is easier to interpret.

shap values

when a variable is added to the model, what is the change in the predictions? shows the value in knowing a given variable

permutation importance

change in correct predictions when a variable is ranodmised, having a larger change means it is more important

Interpreting shap values

how much a specific variable caused the prediction to move off the mean, uses the same values as the output

how to decide where the splits are for a tree

split whever the gini impurity or the squared error is the lowest, so to minimise variation within the groups

what is the difference in interpreting shap values and permuation importance

shap are calculated for an individual, permutation importance is calculated for the whole group

how can shap values be used to show the average effect of a variable

taking the average shap value shows the average magnitude of effects from a variable

what does a larger average shap value indicate

a larger proportion of variation is explained by the variable

what is the cross validation score for continous variables

mean r squared created by the differnt models

what is the rule of thumb for starting a grid search

start broad with larger steps and magnify. Expand the serach if it is near the boundries and use smaller steps next to the value.