1/35
Looks like no tags are added yet.
Threshold
determines when we convert p(x) to either true or false
impacts confusion matrix
Profit Curves
compare models by computing the profit/utility as we vary test set size
What information do you need to make proper use of profit curves?
appropriate prior on class proportions
accurate information on your costs and benefits
Receiver Operating Characteristics (ROC) Curve
X-axis = (FP) / (FP + TN)
Y-axis = (TP) / (TP + FN) → recall
common way to visualize performance of different models
ROC Properties
top-left corner is ideal perfection
FPR=TPR diagonal → performance of random classifier
classifiers on this line are no better than a coinflip
any classifier below this line is worthless
(0,0) and (1,1) mean never issuing positive/negative classifications
How To Vary The Threshold?
get the raw predictions p(x) from the model
define a range of threshold values
loop over threshold range
How can we convert an ROC plot into a single scalar metric?
AUC (area under curve)
allows for further comparison of different models where we can vary p(x) threshold
higher AUC is better
ROC/AUC Weaknesses
doesn’t reflect imbalanced data well
even if one model has low AUC, it may be superior for a given task
not good for multi-class scenarios
ROC/AUC Pros
measures ability to rank positives higher than negatives
shows performance across many thresholds, not optimal threshold
X and Y-axis usually [0.0, 1.0]
Precision Recall Curve Pros
measures the quality of positive predictions
more useful in imbalanced scenarios
Y-axis minima usually well above 0.0
Cumulative Response Curve (CRC)
plots the TPR (y-axis) as a function of the test instance judged to be positive
Lift Curve
modified CRC where the model y-axis is divided by the x-axis value
gives us a more direct advantage over the baseline random classifier
Lift Curve Pros
threshold free
better at helping identify what the optimal threshold should be
visualization based on ranking predictions according to p(x)
better for disseminating model results to stakeholders
derived by CR Curve by normalizing the FPR
ROC/PrRc Curves
visualization based on confusion matrix primarily
better for disseminating model results amongst other data scientists
plot FPN on one axis
TSS
total sum of sqaures (target lavels vs their mean)
measures variance in the target variable itself regardless of our predictions
RSS
residual sum of squares (predictions vs targets)
measures the amount of variability left unexplained by our regression
should be close to 1
R² Key Takeaway
informs you how well a model explains variance
if negative, model is really bad
Correlation Metrics Key Takeaway
how two variables move together, same direction = 1, opposite = -1
Unsupervised Learning
descriptive statistics
What are the 3 categories of unsupervised learning?
dimension reduction
clustering
density estimation
Dimension Reduction
allows us to better visualize dimensional data
Clustering
extracting representative samples from a large corpus of data
Density Estimation
feature distributions
What are the use-cases of unsupervised learning?
utilities and subscription models, customer churn
visualizing large dimension data
feature manipulation for supervised learning
Principal Component Analysis (PCA)
dimension reduction algorithm
1. standardize the data
2. compute feature variance
3. eigenvalue decomposition
4. sort/rank eigenvalues/vectors
eigenvectors are like coefficients for separate linear regression predictors
Noise Filtering
inverse of PCA
reconstructs the original d features
What does the whiten parameter do in PCA?
re-scales answer by number of samples sqrt of n and eigenvalues
useful if PCA features are then used in a supervised predictor
What is the weakness of PCA?
doesn’t work well when there are non-linear relationships within the data itself
Manifold Learning
a class of unsupervised estimators that seeks to describe datasets as low-dimensional manifolds embedded in high-dimensional spaces
think of folding paper example
Multidimensional Scaling (MDS)
unsupervised learning algorithm for recovering the original data
What are MDS use cases?
used for reducing high-dimensional data into a lower-dimension
sifting through marketing or social media data
biometrics and DNA sequences
Why can’t MDS do nonlinear embedding?
because we’re projecting from a 3D space into a 2D space using the distances between every point
t-SNE
t-distributed Stochastic Neighbor Embedding
approximates features using student’s ‘t’ distribution
optimization using the Kullback-Leibler (KL) divergence
doesn’t learn any weights, non-parameteric
Universal Manifold Approximation and Projection (UMAP)
newest and fastest manifold learning technique
Manifold Weaknesses
aren’t as rigorous and can’t explain the variance. vulnerable to noise
less meaning to the reduced dimensions
slower than PCA
Manifold Strengths
pretty straightforward and leads to usually one visualization
many more options or settings to play around to get a different visualization