Data Science Quiz 6

0.0(0)
Studied by 0 people
call kaiCall Kai
learnLearn
examPractice Test
spaced repetitionSpaced Repetition
heart puzzleMatch
flashcardsFlashcards
GameKnowt Play
Card Sorting

1/35

encourage image

There's no tags or description

Looks like no tags are added yet.

36 Terms

1
New cards

Threshold

  • determines when we convert p(x) to either true or false

  • impacts confusion matrix


2
New cards

Profit Curves

compare models by computing the profit/utility as we vary test set size

3
New cards

What information do you need to make proper use of profit curves?

  • appropriate prior on class proportions

  • accurate information on your costs and benefits


4
New cards

Receiver Operating Characteristics (ROC) Curve

  • X-axis = (FP) / (FP + TN)

  • Y-axis = (TP) / (TP + FN) → recall

  • common way to visualize performance of different models


5
New cards

ROC Properties

  • top-left corner is ideal perfection

  • FPR=TPR diagonal → performance of random classifier

    • classifiers on this line are no better than a coinflip

    • any classifier below this line is worthless

    • (0,0) and (1,1) mean never issuing positive/negative classifications


6
New cards

How To Vary The Threshold?

  1. get the raw predictions p(x) from the model

  2. define a range of threshold values

  3. loop over threshold range


7
New cards

How can we convert an ROC plot into a single scalar metric?

  • AUC (area under curve)

  • allows for further comparison of different models where we can vary p(x) threshold

  • higher AUC is better


8
New cards

ROC/AUC Weaknesses

  • doesn’t reflect imbalanced data well

  • even if one model has low AUC, it may be superior for a given task

  • not good for multi-class scenarios


9
New cards

ROC/AUC Pros

  • measures ability to rank positives higher than negatives

  • shows performance across many thresholds, not optimal threshold

  • X and Y-axis usually [0.0, 1.0]


10
New cards

Precision Recall Curve Pros

  • measures the quality of positive predictions

  • more useful in imbalanced scenarios

  • Y-axis minima usually well above 0.0


11
New cards

Cumulative Response Curve (CRC)

  • plots the TPR (y-axis) as a function of the test instance judged to be positive


12
New cards

Lift Curve

  • modified CRC where the model y-axis is divided by the x-axis value

  • gives us a more direct advantage over the baseline random classifier


13
New cards

Lift Curve Pros

  • threshold free

  • better at helping identify what the optimal threshold should be

  • visualization based on ranking predictions according to p(x)

  • better for disseminating model results to stakeholders

  • derived by CR Curve by normalizing the FPR


14
New cards

ROC/PrRc Curves

  • visualization based on confusion matrix primarily

  • better for disseminating model results amongst other data scientists

  • plot FPN on one axis


15
New cards

TSS

  • total sum of sqaures (target lavels vs their mean)

  • measures variance in the target variable itself regardless of our predictions


16
New cards

RSS

  • residual sum of squares (predictions vs targets)

  • measures the amount of variability left unexplained by our regression

  • should be close to 1


17
New cards

R² Key Takeaway

  • informs you how well a model explains variance

  • if negative, model is really bad


18
New cards

Correlation Metrics Key Takeaway

how two variables move together, same direction = 1, opposite = -1

19
New cards

Unsupervised Learning

descriptive statistics

20
New cards

What are the 3 categories of unsupervised learning?

  1. dimension reduction

  2. clustering

  3. density estimation


21
New cards

Dimension Reduction

allows us to better visualize dimensional data

22
New cards

Clustering

extracting representative samples from a large corpus of data

23
New cards

Density Estimation

feature distributions

24
New cards

What are the use-cases of unsupervised learning?

  1. utilities and subscription models, customer churn

  2. visualizing large dimension data

  3. feature manipulation for supervised learning


25
New cards

Principal Component Analysis (PCA)

  • dimension reduction algorithm

  • 1. standardize the data

  • 2. compute feature variance

  • 3. eigenvalue decomposition

  • 4. sort/rank eigenvalues/vectors

  • eigenvectors are like coefficients for separate linear regression predictors


26
New cards

Noise Filtering

  • inverse of PCA

  • reconstructs the original d features


27
New cards

What does the whiten parameter do in PCA?

  • re-scales answer by number of samples sqrt of n and eigenvalues

  • useful if PCA features are then used in a supervised predictor


28
New cards

What is the weakness of PCA?

doesn’t work well when there are non-linear relationships within the data itself

29
New cards

Manifold Learning

a class of unsupervised estimators that seeks to describe datasets as low-dimensional manifolds embedded in high-dimensional spaces

  • think of folding paper example


30
New cards

Multidimensional Scaling (MDS)

unsupervised learning algorithm for recovering the original data

31
New cards

What are MDS use cases?

  • used for reducing high-dimensional data into a lower-dimension

  • sifting through marketing or social media data

  • biometrics and DNA sequences


32
New cards

Why can’t MDS do nonlinear embedding?

because we’re projecting from a 3D space into a 2D space using the distances between every point

33
New cards

t-SNE

  • t-distributed Stochastic Neighbor Embedding

  • approximates features using student’s ‘t’ distribution

  • optimization using the Kullback-Leibler (KL) divergence

  • doesn’t learn any weights, non-parameteric


34
New cards

Universal Manifold Approximation and Projection (UMAP)

newest and fastest manifold learning technique

35
New cards

Manifold Weaknesses

  • aren’t as rigorous and can’t explain the variance. vulnerable to noise

  • less meaning to the reduced dimensions

  • slower than PCA


36
New cards

Manifold Strengths

  • pretty straightforward and leads to usually one visualization

  • many more options or settings to play around to get a different visualization