Study Notes: Multiple and Partial Correlation Analysis

Mathematical Context and Specific Formulas

Fragmentary derivation relating movement to an angle $\alpha$ : - Substituting specific values yields: $\mu_{11}(\cos^2(\alpha) - \sin^2(\alpha)) - \sin(\alpha)\cos(\alpha)(\sigma_x^2 - \sigma_y^2) = 0$ - This simplifies to the relationship: $\mu_{11} \cos(2\alpha) = \frac{1}{2} \sin(2\alpha) (\sigma_x^2 - \sigma_y^2)$ - Resulting in the expression for the angle: $\tan(2\alpha) = \frac{2\mu_{11}}{\sigma_x^2 - \sigma_y^2}$

Correlation Ratio (Definition and Application)

The correlation ratio (denoted by $\eta$ ) is an appropriate measure for the relationship between variables when that relationship is curvilinear rather than linear.
Limitations of the Correlation Coefficient ( $r$ ): - $r$ is a measure of the degree to which a relationship approaches a straight line law. - In bivariate distributions where a strong curvilinear relationship exists, $r$ may be very low or even zero, making it a misleading indicator of association.
Conceptual Definition of $\eta$ : - While $r$ measures the concentration of points about a straight line of best fit, $\eta$ measures the concentration of points about the curve of best fit. - If the regression is linear, $\eta = r$ . Otherwise, the absolute value of the correlation ratio is greater than the absolute value of the correlation coefficient: |\eta| > |r|

Measurement of the Correlation Ratio (\eta)

Consider a scenario where for every value $x_i$ of variable $X$ , variable $Y$ takes values $y_{ij}$ with frequencies $f_{ij}$ (where $j = 1, 2, …, n$ ).
Notation and Variables: - Array frequency: $n_i = \sum_{j=1}^n f_{ij}$ - Total frequency: $N = \sum_{i=1}^m n_i = \sum_{i=1}^m \sum_{j=1}^n f_{ij}$ - Mean of the $i^{th}$ array: $\bar{y}i = \frac{\sum_j f{ij}y_{ij}}{n_i}$ - Overall mean: $\bar{y} = \frac{\sum_i \sum_j f_{ij}y_{ij}}{N} = \frac{\sum_i n_i \bar{y}_i}{\sum_i n_i}$
Formulas for Correlation Ratio of Y on X ( $\eta_{yx}$ ): - $\eta_{yx}^2 = 1 - \frac{\sigma_e^2}{ \sigma_y^2}$ where $\sigma_{eY}^2 = \frac{1}{N} \sum_i \sum_j f_{ij}(y_{ij} - \bar{y}i)^2$ and $\sigma_y^2 = \frac{1}{N} \sum_i \sum_j f{ij}(y_{ij} - \bar{y})^2$ - Convenient expression using the variance of array means ( $\sigma_{mY}^2$ ): - $N\sigma_y^2 = \sum_i \sum_j f_{ij}(y_{ij} - \bar{y})^2 = \sum_i \sum_j {(y_{ij} - \bar{y}i) + (\bar{y}_i - \bar{y})}^2$ - This decomposes into: $N\sigma_y^2 = \sum_i \sum_j f{ij}(y_{ij} - \bar{y}i)^2 + \sum_i n_i(\bar{y}_i - \bar{y})^2$ - Therefore: $\sigma_y^2 = \sigma{eY}^2 + \sigma_{mY}^2$ - Simplified Ratio: $\eta_{yx}^2 = \frac{\sigma_{mY}^2}{\sigma_y^2} = \frac{\sum_i n_i(\bar{y}i - \bar{y})^2}{\sum_i \sum_j f{ij}(y_{ij} - \bar{y})^2}$

Visual Relationship Between r and \eta

Random Scattering: Dots are scattered randomly; $r = 0$ , $\eta_{yx} = \eta_{xy} = 0$ .
Precise Line: Dots lie exactly on a line; $|r| = 1$ , $\eta_{yx} = \eta_{xy} = 1$ .
Precise Curve: Dots lie on a curve (one ordinate per point); $\eta_{yx} = 1$ . If the dots are symmetrically placed about the Y-axis, then $\eta_{xy} = 0$ and $r = 0$ .
Scattered around a Curve: If \eta_{yx} > r, dots are scattered around a definitely curved trend line.

Intra-Class Correlation

Definition: Intra-class correlation refers to correlation within a class, specifically used when both variables measure the same characteristic (e.g., heights of brothers, yields of plots in the same block).
This is distinguishable from product-moment correlation because there is no organic way to distinguish one variable as X and the other as Y; they are interchangeable.
Data Setup: - $n$ families ( $A_1, A_2, …, A_n$ ) with $k_1, k_2, …, k_k$ members. - Measurements denoted as $x_{ij}$ (the $j^{th}$ member in the $i^{th}$ family). - For the $i^{th}$ family, there are $k_i(k_i - 1)$ pairs of measurements. - Total pairs in the intra-class correlation table: $N = \sum k_i(k_i - 1)$ .
Mathematical Expression: - In a symmetrical table, marginal means $\bar{x}$ and $\bar{y}$ are equal. - Total Covariance: $\text{Cov}(X, Y) = \frac{1}{N} \sum_i [k_i^2(\bar{x}i - \bar{x})^2 - \sum_j (x{ij} - \bar{x})^2]$ - If all families have an equal number of members ( $k$ ): - $r = \frac{1}{k - 1} \left( \frac{k \sigma_m^2}{\sigma^2} - 1 \right)$ where $\sigma_m^2$ is the variance of the family means and $\sigma^2$ is individual variance.
Limits and Interpretation: - $\frac{-1}{k - 1} \leq r \leq 1$ - It is a skew coefficient. Negative values do not carry the same significance as equivalent positive values because the lower limit depends on class size.

Introduction to Multiple and Partial Correlation

In social, biological, and physical sciences, phenomena are rarely isolated to two variables.
Multiple Correlation/Regression: Studies the joint effect of a group of independent variables (regressors) on a single dependent variable (response variable). - Example: Crop yield ( $X_1$ ) depends on seed quality ( $X_2$ ), soil fertility ( $X_3$ ), fertilizer ( $X_4$ ), etc.
Partial Correlation/Regression: Studies the relationship between two variables only, after mathematically eliminating the linear effect of other variables.
Objectives of Multiple Linear Regression: 1. Fit a plane of regression using the principle of least squares to estimate values. 2. Estimate the error involved (standard error of estimate). 3. Determine variation accounted for via the multiple coefficient of determination.

Yule's Notation and the Plane of Regression

Notation Structure: - Equation for $X_1$ on $X_2$ and $X_3$ : $X_1 = a + b_{12.3} X_2 + b_{13.2} X_3$ - When variables are measured from their means: $E(X_1) = E(X_2) = E(X_3) = 0$ , thus $a = 0$ . - Regression plane: $X_1 = b_{12.3} X_2 + b_{13.2} X_3$
Subscripts: - Primary subscripts (before the dot): Indicate the variables being directly related. - Secondary subscripts (after the dot): Indicate variables whose effects are kept constant/eliminated. - Order: The number of secondary subscripts determines the "order" of the coefficient or residual (e.g., $12.3$ is 1st order; $12.34$ is 2nd order).
Primary Subscript Importance: In $b_{12.34…n}$ , $X_1$ is the dependent variable and $X_2$ is the independent variable within the relationship.

Derivation of the Regression Plane using Least Squares

Objective: Minimize the sum of the squares of the residuals ( $S = \sum X_{1.23}^2 = \sum (X_1 - b_{12.3} X_2 - b_{13.2} X_3)^2$ ).
Normal Equations for Trivariate Case: 1. $\sum X_2 (X_1 - b_{12.3} X_2 - b_{13.2} X_3) = 0$ 2. $\sum X_3 (X_1 - b_{12.3} X_2 - b_{13.2} X_3) = 0$
Solving for Coefficients ( $b$ ): - In terms of correlation coefficients ( $r_{ij}$ ) and standard deviations ( $\sigma_i$ ): - $b_{12.3} = \frac{\sigma_1}{\sigma_2} \left( \frac{r_{12} - r_{13}r_{23}}{1 - r_{23}^2} \right)$ - $b_{13.2} = \frac{\sigma_1}{\sigma_3} \left( \frac{r_{13} - r_{12}r_{23}}{1 - r_{23}^2} \right)$

General Plane of Regression Equation (Determinant Form)

Determinant of correlations ( $\omega$ ): - $\omega = \begin{pmatrix} 1 & r_{12} & r_{13} \ r_{21} & 1 & r_{23} \ r_{31} & r_{32} & 1 \end{pmatrix}$
Cofactors ( $\omega_{ij}$ ): - $\omega_{11} = 1 - r_{23}^2$ - $\omega_{12} = r_{13}r_{23} - r_{12}$ - $\omega_{13} = r_{12}r_{23} - r_{13}$
The Plane Equations: - Regression of $X_1$ on other variables: $\frac{X_1}{\sigma_1}\omega_{11} + \frac{X_2}{\sigma_2}\omega_{12} + … + \frac{X_n}{\sigma_n}\omega_{1n} = 0$ - General coefficient form: $b_{1j.23…j-1,j+1,…n} = -\frac{\sigma_1 \omega_{1j}}{\sigma_j \omega_{11}}$

Properties of Residuals

Property 1: The sum of the product of any residual of order zero ( $X_i$ ) with any residual of higher order is zero if the subscript of the zero-order residual is among the secondary subscripts of the higher-order residual (e.g., $\sum X_2 X_{1.23} = 0$ ).
Property 2: The product sum of two residuals is unaltered if secondary subscripts from the first are omitted from the product with the second, provided all the secondary subscripts of the first are included in the second (e.g., $\sum X_{1.2} X_{1.23} = \sum X_1 X_{1.23} = \sum X_{1.23}^2$ ).
Property 3: The product sum of two residuals is zero if all the subscripts (primary and secondary) of one appear among the secondary subscripts of the other (e.g., $\sum X_{1.2} X_{3.12} = 0$ ).

Variance of the Residuals

The residual is defined as: $X_{1.23…n} = X_1 - (b_{12.34…n} X_2 + … + b_{1n.23…(n-1)} X_n)$
Variance $\sigma_{1.23…n}^2 = \frac{1}{N} \sum X_{1.23…n}^2 = \frac{\sum X_1 X_{1.23…n}}{N}$
In terms of determinants: $\sigma_{1.23…n}^2 = \sigma_1^2 \frac{\omega}{\omega_{11}}$
For the tri-variate case ( $X_1$ on $X_2, X_3$ ): $\sigma_{1.23}^2 = \sigma_1^2 \frac{1 - r_{12}^2 - r_{13}^2 - r_{23}^2 + 2r_{12}r_{13}r_{23}}{1 - r_{23}^2}$

Coefficient of Multiple Correlation (R)

Definition: The multiple correlation coefficient ( $R_{1.23…n}$ ) is the simple correlation between the actual variable $X_1$ and its estimated value $\epsilon_{1.23…n}$ derived from the regression plane.
Formula: - $R_{1.23…n}^2 = 1 - \frac{\sigma_{1.23…n}^2}{\sigma_1^2} = 1 - \frac{\omega}{\omega_{11}}$
Properties of R: 1. Measures closeness of association between observed and expected values. 2. Least squares regression provides the maximum possible correlation compared to any other linear combination. 3. Range: $0 \leq R \leq 1$ . It can never be negative. 4. If $R_{1.23} = 1$ , the prediction is perfect and residuals are zero. 5. If $R_{1.23} = 0$ , $X_1$ is completely uncorrelated with the other variables. 6. R is never less than any of the total correlation coefficients indicating it ( $R_{1.23} \geq r_{12}, r_{13}, r_{23}$ ).

Coefficient of Multiple Determination

Definition: The square of the multiple correlation coefficient ( $R_{1.23}^2$ ).
Interpreted as: $\frac{\text{Explained Variation}}{\text{Total Variation}}$ .
Calculation: $R_{1.23}^2 = 1 - \frac{\text{Sum of squares due to error}}{\text{Total sum of squares}}$ .
It represents the proportion of total variation in the dependent variable explained by the regression plane.

Coefficient of Partial Correlation

Definition: Measures the correlation between two variables after the linear effect of another variable (or set of variables) has been removed from both.
Trivariate Formula ( $r_{12.3}$ ): - $r_{12.3} = \frac{r_{12} - r_{13} r_{23}}{\sqrt{(1 - r_{13}^2)(1 - r_{23}^2)}}$
Relation to Regression Coefficients: It is the geometric mean of partial regression coefficients: $r_{12.3}^2 = b_{12.3} \times b_{21.3}$ .
Alternative Form using Determinants: - $r_{ij.k} = -\frac{\omega_{ij}}{\sqrt{\omega_{ii} \omega_{jj}}}
Interpretation: If r_{12.3} = 0 $, then$ r_{12} = r_{13}r_{23} $. This implies any observed correlation between$ X_1 $and$ X_2 $is purely due to their common relationship with$ X_3.
Coefficient of Partial Determination: r_{12.3}^2 $. If$ r_{13.2} = 0.7 $, then$ 49\% $($ 0.7^2 $) of the variation in$ X_1 $is associated with$ X_3 $when$ X_2 is held constant.

Quantitative Examples and Solved Problems

Example 12-1: - Given: r_{12} = 0.7, r_{13} = 0.61, r_{23} = 0.4, \sigma_1 = 1.86, \sigma_2 = 2.15, \sigma_3 = 3.46. - Calculation of R_{1.23} $:$ \sqrt{0.6196} = 0.7871. - Calculation of R_{2.13} $:$ \sqrt{0.4912} = 0.7009. - Calculation of R_{3.21} $:$ \sqrt{0.3735} = 0.6111. - Standard Errors: \sigma_{1.23} = 1.1472, \sigma_{2.13} = 1.5336, \sigma_{3.21} = 2.7386.
Example 12-2: - If all correlation coefficients are equal (r_{12}=r_{13}=r_{23}=r $), then$ R_{1.23}^2 = \frac{2r^2}{1+r}.
Example 12-3 (Cinchona Plants): - Observations from 18 plants on Bark (X_1 $), Height ($ X_2 $), and Girth ($ X_3). - Data: r_{12} = 0.77, r_{13} = 0.72, r_{23} = 0.52. - Partial correlation r_{12.3} = 0.62. - Multiple correlation R_{1.23} = 0.8564.
Example 12-4: - Given: \sigma_1 = 2, \sigma_2 = \sigma_3 = 3, r_{12} = 0.7, r_{23} = r_{31} = 0.5. - Results: r_{23.1} = 0.2425, R_{1.23} = 0.7211, b_{12.3} = 0.4, \sigma_{1.23} = 1.3856.

Relationship Between Correlation Types

General linking formula for three variables: - 1 - R_{1.23}^2 = (1 - r_{12}^2)(1 - r_{13.2}^2)
Generalization to n variables: - 1 - R_{1.23…n}^2 = (1 - r_{12}^2)(1 - r_{13.2}^2)(1 - r_{14.23}^2)…(1 - r_{1n.23…(n-1)}^2)$$