3.4 Principal Components Analysis

Section 3.4 of Exam MAS-II

Principal Components:

The m^th principal component is a _____ ______ of all the features in the dataset.

Give m^th component equation and it’s constraint

The m^th principal component is a linear combination of all the features in the dataset.

Principal Components:

• The m^th principal component score for the i^th observation is defined as

• Each principal component is ___ _______ with the others.

• Each principal component is not correlated with the others.

Principal Components: The sum of the product of the principal component _____is ____. Give equation

• The variance explained by each subsequent principal component is always ____ than the variance explained by the previous principal component.

• The signs of the first principal component loadings indicate the direction of _______.

The sum of the product of the principal component loadings is zero.

• The variance explained by each subsequent principal component is always less than the variance explained by the previous principal component.

• The signs of the first principal component loadings indicate the direction of correlation.

Proportion of Variance Explained: The total variance present in a dataset is

This is just complicated notation saying that the amount of variance explained by one PC divided by the total amount of variance from all PCs is the proportion of variance explained by the first PC

Proportion of Variance Explained:

• The variance explained by the m^th principal component is (equation)

• The proportion of variance explained by a principal component is the ratio of the _____ ______ over the _____ _______ present.

• The proportion of variance explained by a principal component is the ratio of the variance explained over the total variance present.

Principal Components: Other Ideas

• Principal components are ___-dimensional surfaces in _-dimensional space that are closest to the observations.

• Scaling has a ________ effect on the result of PCA.

• A ____ plot can be utilized to determine the number of principal components needed to explain a good amount of _________.

• Each principal component loading vector is unique up to a ____ flip.

• PCA is most useful when _____________ is present in the features.

• Principal components are low-dimensional surfaces in -dimensional space that are closest to the observations.

• Scaling has a significant effect on the result of PCA.

• A scree plot can be utilized to determine the number of principal components needed to explain a good amount of variability.

• Each principal component loading vector is unique up to a sign flip.

• PCA is most useful when multicollinearity is present in the features.

Algorithm for Matrix Completion

• Initialize the missing cells with the observed mean of their variable:

  x̃ᵢ,ⱼ = xᵢ,ⱼ if observed
  x̃ᵢ,ⱼ = x̄ⱼ if missing

• Run PCA on the completed matrix and retain the first k components.

• Use the loadings and scores to reconstruct an estimate for every cell:

  x̂ᵢ,ⱼ = ∑ₘ₌₁ᵏ zᵢ,ₘϕⱼ,ₘ

  If PCA was performed on centered variables, add the variable mean back:

  x̂ᵢ,ⱼ = x̄ⱼ + ∑ₘ₌₁ᵏ zᵢ,ₘϕⱼ,ₘ

• Replace only the missing cells with their reconstructed values. The observed cells remain their actual observed values; they are not overwritten.

• Calculate the reconstruction error using only the observed cells:

  J = ∑₍ᵢ,ⱼ₎∈𝒪 (xᵢ,ⱼ − x̂ᵢ,ⱼ)²

  The missing cells cannot be included because their true values are unknown.

• Repeat PCA, reconstruction, missing-cell replacement, and error calculation until the objective stops decreasing meaningfully. This is the algorithm shown in the image.