1/45
Looks like no tags are added yet.
Naive benchmark
A simple baseline rule (what you start with) where you predict the overall average outcome for every single record. Any useful predictive model must beat this baseline to prove it adds value.
Mean Absolute Error (MAE)
The average size of your prediction errors, measured in the exact same units as the outcome (such as dollars).
Mean Squared Errors (MSE)
The average of all squared errors. Because errors are squared, big misses are penalized much more heavily than small ones.
Root Mean Squared Error (RMSE)
The square root of MSE. Converts the metric back to its original units, making it easier to read, but punishes large errors extra heavily.
(Model evaluation & interpretation) What metrics are used for numeric outcomes?
Naive benchmark, MAE, MSE, RMSE
Why might a model scored only on its training data look great?
Because it could have memorized the training data; overfitting.
What is a confusion matrix?
A summary table that counts correct and incorrect predictions for each category, serving as the foundation for all classification metrics.
What is accuracy? (provide the formula too)
The overall share of all predictions that the model got right. It can be misleading when one outcome is very rare (like fraud or a rare disease). (TP + TN / (TP + FN + TN + FP))
What is precision? (provide the formula too)
Out of all the cases the model flagged as positive, how many are actually positive? (TP / (TP + FP))
What is recall / sensitivity? (provide the formula too)
Out of all the positive cases in the data, how much did the model successfully catch? (TP / (TP + FN))
What is specificity? (provide the formula too)
Out of all the actual negative cases, how many did the model correctly leave alone? (TN / (TN + FP))
What is the F1 score? (provide the formula too)
A single combined metric that balances precision and recall. (F1 = (2 x precision x recall) / (precision + recall))
What is the AUC [area under ROC curve]?
Measures how well a model separates positive and negative classes across every possible cutoff threshold. A score of 0.5 means the model is no better than a coin flip, while 1.0 is perfect separation.
What is overfitting?
When a model memorizes the quirks and noise of training data, making it look great during training but fail on new, unseen data. Always evaluate models on separate validation or test sets to get an honest score.
What is classification cutoff?
The probability threshold used to decide when a prediction counts as “positive”. Lowering the cutoff flags more cases as positive (boosting recall, hurting precision and specificity), while raising it does the opposite. Choose a cutoff based on the real-world cost of false alarms versus missed cases.
What is a lift chart?
A visual chart showing how well a model ranks records from most likely to least likely (for example, putting the most likely buyers at the very top of a marketing list).
What is cross-validation?
Testing a model across multiple different splits of your dataset to ensure its performance ranking isn’t just the result of one lucky split.
What is feature importance?
Score showing which variables matter most to the model's predictions overall.
What is a Partial Dependence Plot (PDP)?
A graph showing how the predicted outcome changes, on average, as one variable changes.
What are SHAP values?
A method that calculates how much each variable pushed one specific prediction up or down.
What is LIME?
A tool that creates a simple, local explanation of one specific prediction.
What is sensitivity analysis?
What happens to the output when one input changes and the others are held constant.
Is the most statistically accurate model always the best choice? Why?
No. In regulated fields like banking or healthcare, a slightly less accurate model that is fast, scalable, and easy to explain is often preferred.
What is the first thing you should do when evaluating a model?
Benchmark first; a model that cannot beat “predict the average” adds no value, however sophisticated it is.
RMSE vs. MAE, how are they different?
Both are measurements in the outcome’s units, but RMSE punishes big misses more, so it is always at least as large as MAE.
What is the main difference between recall and specificity?
Recall is about catching the actual positives, while specificity is about correctly clearing the actual negatives.
What are the simple ways to summarize metrics like accuracy, precision, recall, and specificity? What types of variables use these metrics?
Accuracy is overall correctness, precision is positive prediction correctness, recall is finding all the positives, and specificity is finding all the negatives.
(Model evaluation and interpretation) What does a “lift of 4” mean in the case where you’re trying to predict and target potential shoppers?
By targeting the top 10% of buyers, your model captures 4x as many actual buyers as you would by targeting a random 10%.
Which methods use standardization (k-NN, Naive Bayes, SVM)?
k-NN and SVM
When are dummy variables used?
To convert categorical text labels into 0/1 numeric columns so distance-based algorithms like k-NN can measure similarity
How is the “k” for “k-NN” chosen?
Cross validation or validation data
In two class problems, how can k avoid ties? (k-NN)
By choosing an odd value for k
k-NN is slow on large datasets. True or false? Why?
True. It must keep all its training data in memory because it evaluates predictions on the spot (“lazy learner”).
When is k-NN useful? What use case could k-NN be for?
Tasks built on similarity. k-NN could be used for product recommendations (“customers like you also bought…”).
Where does the “naive” in Naive Bayes come from?
It assumes predictors are independent from one another within each class
How does Naive Bayes work?
It combines values of the prior (how common each class is overall) with the likelihoods (how common each clue is within each class) to get the posterior (the updated probability after seeing the clues), then predicts based off those values.
What is different about Gaussian Naive Bayes?
It assumes a bell curve for each predictor within each class.
Naive Bayes is better at predicting probabilities over ranking and classifying. True or false?
False
What is the “zero-frequency problem” of Naive Bayes? What can fix it?
It’s when a model assumes a probability is zero because it doesn’t spot a predictor value in its training set (doesn’t mean it is actually zero); the solution is laplace smoothing, which adds one to every probability, so that no value has a zero chance.
When is Naives Bayes used? What can Naive Bayes be used for (use case)?
Tasks that need speed and scale; Spam email classification
How does Support Vector Machines (SVM) work?
An SVM finds the boundary (a line, plane, or hyperplane) that separates the classes with the widest possible margin, meaning the boundary sits as far as possible from the closest records of each class.
What are the closest records (to a boundary) for SVM called?
Support vectors
What is a kernel in SVM? What are common kernels?
SVMs use a kernel to map the data into a higher-dimensional space where a separating boundary can be found; used to handle nonlinear patterns.
Common kernels are linear, polynomial, and RBF.
What does a soft-margin SVM do? How do you adjust it?
Lets some records fall within the boundary margin or cross the line, at a penalty set by C. A small C is more tolerant (risks underfit), a larger C is more strict (risks overfit).
How is C in SVM tuned?
Through cross-validation
When would you use SVM; what is a use case for SVM?
Tasks where accuracy and a clean decision boundary matter most ;credit scoring