Quant

P-Value for Slopes

use the p-value to evaluate the null hypothesis that a slope coefficient is equal to zero.

The p-value is the smallest level of significance for which the null hypothesis can be rejected. We test the significance of coefficients by comparing the p-value to the chosen significance level:

• If the p-value is less than the significance level, the null hypothesis can be rejected.

• If the p-value is greater than the significance level, the null hypothesis cannot be rejected.

Assumptions underlying a multiple regression model include

• A linear relationship exists between the dependent and independent variables.

• The residuals are normally distributed.

• The variance of the error terms is constant for all observations.

• The residual for one observation is not correlated with that of another observation.

• The independent variables are not random, and there is no exact linear relation between any two or more independent variables.

Goal of QQ plot

Check if residuals are normally distributed

Points deviating from the Q–Q line indicate that residuals are not normally distributed. The plot assesses normality, not correlation. Points below the line indicate that observed values are more negative than expected under normality, while points above the line indicate more positive values than expected.

Functional Form Model Misspecifications

Misspecification	Description	Effect
Omission of important independent variable(s)	Based on economic theory, one or more variables that should have been included are omitted.	Biased and inconsistent regression parameters May lead to serial correlation or heteroskedasticity in the residuals
Inappropriate variable form	The relationship between the dependent and independent variables may be non-linear.	May lead to heteroskedasticity in the residuals
Inappropriate variable scaling	Variables may need to be transformed before estimating the regression.	May lead to heteroskedasticity in the residuals or multicollinearity
Data improperly pooled	Sample has periods of dissimilar economic environments (that should not be pooled).	May lead to heteroskedasticity or serial correlation in the residuals

More on Misspecifications

Problem	What goes wrong	Why it matters
Omitted variable (correlated)	Bias + inconsistency (other variables take credit)	Worst case
Omitted variable (uncorrelated)	Intercept bias only	Less severe
Wrong transformation	Nonlinear → forced linear	Poor fit
Bad scaling	Units distort meaning	Misleading comparisons
Pooling data	Different regimes mixed	Wrong conclusions

robust standard errors

Correcting Heteroskedasticity

To correct for conditional heteroskedasticity of regression residuals, we can calculate robust standard errors (also called White-corrected standard errors or heteroskedasticity-consistent standard errors). These robust standard errors are then used to recalculate the t-statistics using the original regression coefficients for hypothesis testing.

Serialcorrelation

Serialcorrelation, also known as autocorrelation, refers to a situation in which regression residual terms are correlated with one another; that is, not independent. Serial correlation can pose serious problem with regressions using time series data.

Positiveserial correlation

Positiveserial correlation exists when a positive residual in one time period increases the probability of observing a positive residual in the next time period.

Negativeserial correlation

Negativeserial correlation occurs when a positive residual in one period increases the probability of observing a negative residual in the next period.

lagged values and positive serial correlation

lagged values
- if lagged values is inaccurate and mess up the whole model and lead to inconsistency

• if not lagged values and uses external variables then can never be inconssitent

postiive serial correlation

• Type I error - false positive

Durbin-Watson (DW) statistic

Residual serial correlation at a single lag can be detected using the Durbin-Watson (DW) statistic.

Breusch-Godfrey (BG) test

more general test (which can accommodate serial correlation at multiple lags) is the Breusch-Godfrey (BG) test.

robuststandard errors

To correct for serial correlation in regression residuals, we can calculate robuststandard errors (also called Newey--West corrected standard errors or heteroskedasticity-consistent standard errors). These robust standard errors are then used to recalculate the t-statistics using the original regression coefficients.

most common sign of multicollinearity

The most common sign of multicollinearity is when t-tests indicate that none of the individual coefficients are significantly different than zero, but the F-test indicates that at least one of the coefficients is statistically significant, and the R² is high. This suggests that the variables together explain much of the variation in the dependent variable, but the individual independent variables don't. This can happen when the independent variables are highly correlated with each other—so while their common source of variation is explaining the dependent variable, the high degree of correlation also "washes out" the individual effects.

What test to check Heteroskedasticity?

Breusch Pagan Chi²

• regress residuals² back into the formula

• BP = n * R² for residuals

  • df = k

• H0: Homo Ha: Hetero

How to fix Hetero?

White-Correlated/Robert Standard error

By applying White’s correction for heteroskedasticity, the coefficient standard errors will increase, thereby decreasing estimated t-statistics and increasing corresponding P-values for the independent variables.

What test for serial correlation

Durbin Watson - single

Breusch Godfrey - multi

• regress residuals but with lagged variables

  • test if lagged variable slopes is stat sig

• H0: no serial Corr Ha: serial corr

How to fix serial corr?

Newey-West/Robust SE

Multicollinearity

inflated SE

Type II

regress the variables with each other removing 1 at a time

VIF = 1/1-R²

low is good high is bad

high-leverage points

high-leverage points are the extreme observations of the independent (or 'X') variables.

The sum of the individual leverages for all observations is k + 1 (with intercept). If a variable's leverage is higher than three times the average, [3(k + 1) / n], it is considered potentially influential.

studentized residuals

USED TO SPOT OUTLIERS

We can identify outliers using the studentized residuals. The following steps outline the procedure:

1. Estimate the regression model using the original sample of size n. Delete one observation and re-estimate the regression using (n - 1) observations. Perform this sequentially, for all observations, deleting one observation at a time.

2. Compare the actual Y value of the deleted observation i to the predicted Y-values using the model parameters estimated with that observation deleted.

   e*i = Yi -∧Y*i

3. The studentized residual is the residual in Step 2 divided by its standard deviation.

   t*i = e*is*e

4. We can then compare this studentized residual to critical values from a t-distribution with n – k – 2 degrees of freedom (because we now only have n – 1 observations), to determine if the observation is influential.

qualitative dependent variable

Financial analysis often calls for the use of a model that has a qualitative dependent variable—a categorical variable, usually a binary variable, which takes on a value of either zero or one. An example of an application requiring the use of a qualitative dependent variable is a model that attempts to estimate the probability of default for a bond issuer. In this case, the dependent variable may take on a value of one in the event of default and zero in the event of no default.

covariance stationary

Statistical inferences based on ordinary least squares (OLS) estimates for an AR time series model may be invalid unless the time series being modeled is covariance stationary.

A time series is covariance stationary if it satisfies the following three conditions:

1.  Constant and finite expected value. The expected value of the time series is constant over time. (Later, we will refer to this value as the mean-reverting level.)

2.  Constant and finite variance. The time series' volatility around its mean (i.e., the distribution of the individual observations around the mean) does not change over time.

3.  Constant and finite covariance between values at any given lag. The covariance of the time series with leading or lagged values of itself is constant.

How to compare AR models

RMSE

• test for forecasting model

• out of sample accuracy

Random walk

Random walk. If a time series follows a random walk process, the predicted value of the series (i.e., the value of the dependent variable) in one period is equal to the value of the series in the previous period plus a random error term.

A time series that follows a simple random walk process is described in equation form as x_t = x_t–1 + ε_t, where the best forecast of x_t is x_t–1 and:

1. E(ε_t) = 0: The expected value of each error term is zero.

2. E(ε_t²) = σ²: The variance of the error terms is constant.

3. E(ε_iε_j) = 0; if i ≠ j: There is no serial correlation in the error terms.

AR vs Random Walk

An AR model should not be applied to a random walk in levels because the series is nonstationary; instead, the series should be first differenced to achieve stationarity before modeling.

ARCH

An ARCH model is used to test for autoregressive conditional heteroskedasticity. Within the ARCH framework, an ARCH(1) time series is one for which the variance of the residuals in one period is dependent on (i.e., a function of) the variance of the residuals in the preceding period. To test whether a time series is ARCH(1), the squared residuals from an estimated time-series model, ˆε2t, are regressed on the first lag of the squared residuals ˆε2t–1.

The ARCH(1) regression model is expressed as:

ˆε2t=a0+a1ˆε2t−1+μt

where a₀ is the constant and μ_t is an error term.

If the coefficient, a₁, is statistically different from zero, the time series is ARCH(1).

2 time series regression

Regression results are reliable only when both series are stationary or when nonstationary series are cointegrated; otherwise, the regression is spurious and invalid.

Cointegration

Cointegration means that two time series are economically linked (related to the same macro variables) or follow the same trend and that relationship is not expected to change. If two time series are cointegrated, the error term from regressing one on the other is covariance stationary and the t-tests are reliable.

Bias, Variance, Base Error

• Bias error. This is the in-sample error resulting from models with a poor fit.

• Variance error. This is the out-of-sample error resulting from overfitted models that do not generalize well.

• Base error. These are residual errors due to random noise.

Regularization

Regularization forces the beta coefficients of nonperforming features toward zero.

SVM

Binary decision ML model

Ensemble and Random Forest

Model	Main Idea
Ensemble learning	Combine many models to improve predictions
Random forest	Ensemble method using many randomized decision trees

Eigenvectors & Eigenvalues (PCA)

These uncorrelated factors, called eigenvectors, are linear combinations of the original features. Each eigenvector has an eigenvalue—the proportion of total variance in the data set explained by the eigenvector. The first factor in PCA would be the one with the highest eigenvalue, and would represent the most important factor. The second factor is the second-most important (i.e., has the second-highest eigenvalue) and so on, up to the number of uncorrelated factors specified by the researcher.

Scree Plots

Scree plots show the proportion of total variance explained by each of the principal components. In practice, the smallest number of principal components that collectively capture 85%–95% of the total variance are retained. Since the principal components are linear combinations of the original data set, they cannot be easily labeled or interpreted, resulting in a black-box approach.

Hierarchical Clustering (agglomerative & divisive)

In an agglomerative (or bottom-up) clustering, we start with one observation as its own cluster and add other similar observations to that group, or form another nonoverlapping cluster. A divisive (or top-down) clustering algorithm starts with one giant cluster, and then it partitions that cluster into smaller and smaller clusters.

Neural - Network (Neurons, Activation, backpropagation, forward propagation)

Neurons combine weighted inputs, apply an activation function, and pass information forward through the network, while backpropagation adjusts the weights based on prediction errors so the network learns over time.

Regex

regular expression (regex) is a text string used to identify characters in a particular order.

Data Wrangling

Data wrangling involves preprocessing data for model use. Preprocessing includes data transformation and scaling.

Data transformation types include:

• Extraction (e.g., extracting number of years employed based on dates provided).

• Aggregation, which involves consolidating two related variables into one, using appropriate weights.

• Filtration, which involves removing irrelevant observations.

• Selection, which involves removing features (i.e., data columns) not needed for processing.

• Conversion of data of diverse types (e.g., nominal, ordinal).

Text Wrangling

Cleansed text is then normalized using the following steps:

1. Lowercasing. So as to not discriminate between market and Market.

2. Removal of stop words. In some ML applications, stop words such as the, is, and so on do not carry any semantic meaning; hence, they are removed to reduce the number of tokens in the training data.

3. Stemming. This is a rules-based algorithm that converts all variations of a word into a common value. For example, integrate, integration, and integrating are all assigned a common value of integrat. While stemming makes the text confusing for human processing, it is ideally suited for machines.

4. Lemmatization. This involves the conversion of inflected forms of a word into its lemma (i.e., morphological root). Lemmatization is similar to stemming, but is more computationally advanced and resource intensive.

What is the difference between Bag of Words (BOW) and N-grams in text wrangling?

Bag of Words (BOW): Represents text by counting word frequencies while ignoring word order. Text is tokenized, and a document-term matrix stores token counts.

N-grams: Preserve sequences of words. A 2-word sequence is a bigram (e.g., “market_is”), a 3-word sequence is a trigram. N-grams capture context and meaning better because word order matters. Stop words are often retained since removing them may destroy phrase meaning (e.g., “not good”).

Feature Selection vs Engineering

1. Feature selection is a process to select only the needed attributes of the data for ML model training. The higher the number of features selected, the higher the model complexity and training time.

2. Feature engineering is the process of creating new features by transforming (e.g., taking the LN(Y), natural logarithm), decomposing, or combining multiple features. A related term is feature extraction, whereby a feature is created from the data set (e.g., creating a value for age using date-of-birth data).

Feature engineering (FE) involves optimizing and improving the selected features. Model training results depend on how the features are presented to the algorithm. Feature engineering involves either decomposing a feature into multiple features or converting an existing feature into a new feature. One-hot encoding (OHE) is a process used to convert a categorical feature into a binary (dummy) variable suitable for machine processing. Feature engineering seeks to make model training faster and easier.

Class Imbalance

Class imbalance occurs when one class has a large number of observations relative to other classes. For example, in a model for predicting bond default, if the data set has a large number of high-grade bonds (i.e., those that would be less likely to default), then the model would be more likely to predict nondefault for a new observation. The training data set should have a variety of high- and low-grade bonds so as to have enough diversity to make correct predictions. One way to overcome class imbalance is to undersample the overrepresented class and oversample the underrepresented class.

Hyperparameters

Hyperparameters (e.g., the number of hidden layers in a neural network, or the p-threshold in logistic regression) are specified by ML engineers, and are not dependent on the training sample.

Grid Search

An automated tuning method that tests many combinations of hyperparameters systematically.

Instead of trial-and-error, grid search searches across a predefined parameter grid to find the best combination.

Ceiling Analysis

Evaluates every step in the model-building pipeline to identify the weakest component (“weak link”).

Purpose:

• Determine which stage most limits performance

• Improve that component to increase overall model accuracy