Exam PA

A greedy feature selection technique that starts with all available features and iteratively removes the least significant ones until a stopping criterion is met

Pros:

• Considers multivariate interactions and multicollinearity better at the start than forward selection because it begins with the full context of all variables.

• Yields simpler, more interpretable models
  Cons:

• Computationally expensive for very high-dimensional datasets (p > n cannot be handled because the initial model requires more observations than features).

• Greedy nature means removing a feature early can occasionally miss optimal combinations found via exhaustive search. [1, 2]

Backward Selection

A greedy wrapper method that builds a machine learning model by starting with an empty feature set and iteratively adding the single best-performing feature until a stopping criterion is met

• Pros:

  • Improves model interpretability and accuracy by reducing noise.

  • Computationally faster than evaluating every possible feature combination (exhaustive search) or using backward elimination on massive feature sets. [1, 2, 3, 4]

• Cons:

  • Greedy nature means it can miss optimal combinations of features that only work well together.

  • Lacks the ability to drop a feature later if it becomes redundant after other variables are added.

  • Prone to overfitting and computationally expensive for very wide datasets

Forward Selection

Combines Lasso and Ridge.

Performs variable selection and keeps correlated variables together rather than arbitrarily picking one and dropping others.

Elastic Net Regression (ENR)

A way of determining how well a classification model performs. Graph has True positive rate(sense) on y axis, and false positive rate(1-spec) on x axis. Two points are always calculated on graph, those being (0,0) and (1,1). The other points show the TPR and FPR for the confustion matrix that the model generates for a variety of cutoff values. A point can be generated for each matrix if multiple cut offs are used.

Receiver Operator Curve (ROC)

TPR/Sense on y axis
FPR/(1-Spec) on x axis

Three points: (0,0) (FPR,TPR) (1,1)
Use these to calc area of AUC
First part is triangle, second is rectangle, third is triangle

AUC

Area Under Curve (AUC)

Lasso, Ridge, ENR

Adds penalty to OLS model to shrink coefficient estimates.

Prevents overfitting and handles collinearity

Regularized Regression

(True Positives)/(True Positives + False Negatives)

Sensitivity/TPR

(True Negatives)/(True Negatives + False Positives)

Specificity

(True Positive)/(True Positive + False Postive)

Precision

(True Positive + True Negative)/(Total)

Accuracy

Defines a minimum amount of impurity reduction needed for a split to be made. Any split that does not decrease the overall lack of impurity won’t be made.

Controls overall size and depth of the tree. Increasing the complexity generally makes trees smaller and less flexible.

Complexity Parameter

Two predictors are highly correlated with each other

The effect of individual variables is difficult to estimate, leading to higher standard errors of coefficient estimates and making it difficult to interpret the impact of individual variables on the target.

Collinearity

Answers a what if question

Ex: Data can be used to determine whether a single pump or multi pump should be deployed to deal with a fire.

Prescriptive Analytics

Focuses on what happened in the past

Location data can be used to determine what area had more calls

Descriptive Analytics

Asks what might happen next. Can be used to predict values for certain variables

Use data to predict what the time it will take to respond to a fire.

Predictive Analytics

Written data collected via survey or some other collection method

Advantage: Unstructured data includes information that cannot be stored in a tabular format. Using this unstructured data gives insights and qualitative information that cannot be included in a structured dataset, e.g. insights on the sellers and background around the type of sale.

Disadvantage: Unstructured data requires more complex methods to process for input into a predictive model. It can also be more time-consuming and resource-intensive to analyze unstructured data.

Unstructured Data

Uses Euclidean distance for clustering observations into K clusters. Initial cluster assignment is random, from there euclidean is used to reassign until clusters no longer change. Requires standardization for trustworthy results. Limited by having to choose K before process starts.

K-means Clustering

Length between two points in a flat space. Used to reassign clusters in K-means. Struggles with hot encoded categoricals, as adjacent values will always cluster together, regardless of actual patterns in data.

Euclidean distance

Generates new variables from a set of original variables, it is useful if the original variables describe a unifying concept, thereby enhancing the interpretability of the new variables.

Can be dimension reduction, as multiple variables can be contained in one PC

Most valuable where there are many correlated variables, and a large proportion of the total variance is explained using just a few of the components.

Principal Component Analysis

PCA results in numeric features called principal components. They are linear combinations of the analyzed variables, which means a principal component summarizes the variables by specifying how much each variable contributes to its calculation. Clustering identifies clusters or groupings based on the analyzed variables, meaning it results in a factor. Similar observations are grouped into the same cluster while dissimilar observations are grouped into different clusters

Cluster vs PCA

Use your response variable is continuous, normally distributed, and linearly related to your predictors

Interpretation: Represents the absolute change in the expected value of the response variable (y) for a one-unit increase in the predictor variable

Guassian with Identity

Use when a discrete, non-negative target and mean = variance

Interpretation: Exponentiate the coefficient (\(\exp(\beta_1)\)) to find the multiplicative factor change in the expected count or rate for a one-unit increase in the predictor variable. % decrease or increase, based on the value.

Poisson with Log

Use when your dependent variable represents binary outcomes (0/1, yes/no) or aggregated proportions of successes out of a known number of trials.

Interpretation: Represents the expected change in the log-odds of the outcome for a one-unit increase in the predictor variable. Exponentiate coefficient to get odds ratio. % decrease or increase based on value.

Binomial with Logit

Use when modeling continuous, strictly positive data where the variance is proportional to the square of the mean.

Interpretation: Look at how the predictor affects the reciprocal of the mean response. Predict = 1/(B0 + B1x1 + …). Increase x1 one unit and see how predict changes.

Gamma with Inverse

Use when modeling continuous, strictly positive, and right-skewed data

Inverse Guassian with Negative Inverse Square

A partial dependence plot shows the impact that a specific variable has on the final prediction, without considering the model structure.

Used to interpret “black box” models like random forests.

Partial Dependence Plot

Outliers that have an impact on the model coefficients. An observation has high leverage if it greatly affects the coefficient estimate.

Leverage

Plots residuals vs leverage. Observations with high leverage should be reviewed and determine whether they should be removed from the dataset.

Leverage plot

Compound Poisson-Gamma distribution. It models data as a Poisson number of events (such as the frequency of claims) where each event has a Gamma-distributed magnitude (such as the cost of each claim)

Gives special emphasis to value of 0

Tweedie Distribution

Does not work with raw categorical variables. Can use one hot encoding of 0’s and 1’s and then use euclidean distance after. Do not code as sequential numbers, e.g. 1, 2, 3, as this will not work.

Categorical vars with Euclidean Distance

Use weight and loadings to get value

Value*(Variance)^.5 + Mean

Unstandardize Principle component

Normal distribution around the mean

Errors are normally distributed with mean 0 and variance sigma²

Gaussian Assumption

Known covariate added to linear predictor whose value is 1. Used as a structural adjustment that shifts expected value of target.

Using a var as an offset means that var means that the impact of the var on the target is known in advance, and thus does not require to be estimated, in contrast to the impact of predictors.

Offset

Used to evaluate and compare the relative quality of different models. Balances how well a model fits the training data against the number of parameters it uses. Adding more variables improves fit but increases K, raising the AIC score. When comparing models with same number of predictors, lower AIC is better.

AIC

Used to select the best model from a finite set of options while penalizing extra complexity to prevent overfitting. Adds a heavier penalty for extra parameters compared to AIC, especially when the sample size is large. Lower is better.

BIC

Find cp with minimum xerror. Add the xstd of this cp to the xerror, and take the first xerror under this new value. The cp corresponding to this xerror is the cp chosen by one standard error rule

One Standard Error Rule

If minimizing cv error, choose cp with lowest x error, else use one standard error.

Number of splits by complexity parameter

Show the contribution of each feature in the model structure. They can be used to rank which features are most important in fitting the model structure

Used to interpret “black box” models like random forests.

Feature Importance Plot

Trees are built sequentially. After a tree is built, any information that’s left unexplained, also called residuals, becomes what the next tree will build on, and so on with all subsequent trees. Ensemble method with low interpretability

Boosting process and goal

Trees are built independently of one another. Specifically, many bootstrap samples are created, and a tree is built on each one. Because one bootstrap sample does not influence the creation of another, the same can be said of the trees. Ensemble method with low interpretability

Random Forest

Average the MSE of all OOB obs

OOB Observations

Means that there are different proportions of each level of the variable. Can be fixed with oversampling and undersampling

Unbalanced Variable

Predictor is a linear combination of the other predictors in the same model. Holiday purchases and non holiday purchases both being included as predictors is perfect collinearity

Perfect Collinearity

Both perform feature selection and require input from analyst.

Stepwise selection measures flexibility by the number of predictors, while regularization does this by the shrinkage parameter. Stepwise selection would use AIC or BIC to select an optimal model, but regularization would instead use a model accuracy metric calculated from cross-validation.

Stepwise vs Regularization

Process of converting continuous numerical or categorical data into binary values (typically 0 and 1) based on a specific threshold.

Only the useful dummy variables from the same factor can be retained in the optimal model, thus simplifying the model to only the predictors that are deemed necessary.

Resulting modification that a factor experiences might make little sense and complicate its interpretation

Binarization

Scale the raw residuals by the estimated standard errors, adjusting for each point’s leverage. This puts all residuals on a common, approximately t-distributed, scale, so you can directly compare extremeness across observations.

Studentized Residuals

AIC has a weaker penalty than BIC for each predictor in a model, hence making it easier to retain predictors.

AIC vs BIC

Backward selection would tend to discourage predictors from being excluded because if a local minimum of the information criterion is found early in the procedure, the algorithm would end while there are still many predictors.

Want more predictors? Use backward. Worried about overfitting? Use forward.

Forward vs Backward Selection

Backward Selection Process

1) Start with the Full Model: Fit a regression model with all available predictors included.

2) Evaluate Predictors: Assess the significance of each predictor using the chosen criterion: P-values: Check the statistical significance of each predictor's coefficient. AIC/BIC: Evaluate the overall model fit and complexity with each predictor removed. Remove the least significant predictor.

3) Remove this predictor from the model. Refit the regression model without the removed predictor. Recalculate the metric used in step 2 for the reduced model.

4) Repeat the Process: Continue removing the least significant predictor in each step. Stop when: All remaining predictors are significant (if using p-values). Removing any additional predictors increases AIC or decreases model fit.

5) Select the final model: The process ends when no further improvements can be made, and the final model contains only the most significant predictors.

Forward Selection Process

1. Start with the null model, which has no predictors.

2. Consider all models that add one of the remaining terms to the current model.

   (Let be the number of terms that the current model has. Thus, there are "remaining terms" in the pool of potential predictors, resulting in models with one added term.)

3. Fit these models and, together with the current model, compare using AIC or BIC.

4. If the model with the lowest AIC or BIC is the current model (i.e., no improvement), stop. Otherwise, select the model with the lowest AIC or BIC as the new current model and go back to step 2

Weight

An average is judged to be more credible if it has more of a variable. The GLM will estimate the coefficients in a way that places a higher priority on targets with more of the weighted variable.