PSYC 3070K - Quantitative Methods: Linear Regression

PSYC 3070K Quantitative Methods Linear Regression: Part 1

Learning Objectives

  • Define linear regression.

  • Identify errors of prediction in a scatter plot with a regression line.

Vocabulary

  • Regression: Predicting scores on one variable (criterion variable) from the values of others (predictor variables).

  • Linear Regression Models: Utilize the equation of a straight line to fit the data. Other types of regression do not assume linear relationships.

  • Simple Linear Regression: A linear regression model with only one predictor variable.

Linear Regression

  • Like Pearson’s $r$, linear regression fits a straight line through a set of points.

  • The equation of the line can be used to predict the value of the criterion variable (Y) from the predictor variable (X).

The Equation of a Line

  • The general form of the equation is:

Y′=bX+AY' = bX + A

  • In a specific example, the equation can look like:
    Y=−0.5x+4Y = -0.5x + 4

Components of the Regression Equation

  • Y': Predicted value of the criterion variable.

  • X: Predictor variable.

  • b: Represents the association between X and Y.

  • A: The y-intercept, a constant value.

  • Prediction errors (or residuals) are the differences between each actual point (Y) and the predicted point (Y').

  • Best-fitting Line: Most commonly determined by minimizing the sum of squared prediction errors:
    ext{Sum of Squared Errors} = extstyle egin{pmatrix} ext{Sum} \ (Y - Y')^2 ext{, where } Y' ext{ is the predicted value}
    ewline ext{and } Y ext{ is the actual value} ext{.} ightspace ext{ } ext{ } ext{ } ext{ } ext{ } ext{ } ext{ } ext{ } ext{ } ext{ } ext{ } ext{ } ext{ } ext{ } ext{below } ext{ } ext{ } ext{} ext{ } ext{} ext{ } ext{} ext{ } ext{} ext{ } ext{} ext{} ext{} \ ext{best-fit line} ext{ assumes some linear relationships. } ext{ This point should still be written in full.} ext{ } ext{ } ext{ } ext{ } ext{ . } ext{ } ext{ } ext{ } ext{ } ext{ } ext{ } ext{ } ext{ } ext{ } ext{ } ext{ } ext{ } ext{ } ext{ } ext{ } ext{ } ext{ } ext{ } ext{ } ext{} ext{ } ext{ } ext{ } ext{ } ext{ } ext{ } } ext{ ext{ }}}

Example of Computing the Regression Line

  • Given values:

    • b=0.627imesrac1.0721.581=0.425b = 0.627 imes rac{1.072}{1.581} = 0.425

    • A=2.06−0.425imes3=0.785A = 2.06 - 0.425 imes 3 = 0.785

    • Regression equation can be formulated as:
      Y=0.425X+0.785Y = 0.425X + 0.785

    • Where:

    • Mx=extMeanofXMx = ext{Mean of } X

    • My=extMeanofYMy = ext{Mean of } Y

    • sx=extStandarddeviationofXsx = ext{Standard deviation of } X

    • sy=extStandarddeviationofYsy = ext{Standard deviation of } Y

    • r=extCorrelationcoefficientr = ext{Correlation coefficient}

Participation Example

  • To predict a woman's shoe size from height, information gathered includes:

    • Mean height: 64 inches

    • Standard deviation of height: 2 inches

    • Mean shoe size: 8

    • Standard deviation of shoe size: 1

    • Correlation (height & shoe size): 0.5

  • Find the equation of the regression line.

Standardized Residuals

  • If variables X and Y are standardized (converted to z-scores), the regression equation changes to: Zy′=rZxZy' = rZx

    • Where there is no y-intercept.

    • The slope of the regression line equals the correlation coefficient (r).

Unstandardized b vs. Standardized Beta

  • Unstandardized Coefficient (b): Computed from the original measurement units of the predictor variable (e.g., dollars, centimeters).

  • Example in Ride-Share:
    Y′=bX+AY' = bX + A

    • Total cost equation:
      Y’=extRateimesextDistance+extBasefeeY’ = ext{Rate} imes ext{Distance} + ext{Base fee}

    • Units for the slope = dollars per mile.

  • Standardized Coefficient (β or Beta): Computed from units of standard deviation (when X and Y are converted to z-scores).

  • For ride-share, the equation becomes:
    ZY′=(β)(ZX)ZY' = (β)(ZX)

  • Implication: A standard deviation increase in distance relates to a β standard deviations increase in cost.

Reporting Unstandardized b vs. Standardized Beta

  • Unstandardized units are more intuitive to the public (e.g., dollars per mile).

  • In cases of multiple regression with more than one predictor, standardized slopes (β) are preferable for comparison across different measurement units, highlighting predictor importance.

Example Correlation

  • Given:

    • Pearson’s r for correlation = -0.134

    • Unstandardized regression equation:
      extAgreeableness=−0.104imesextNeuroticism+3.729ext{Agreeableness} = -0.104 imes ext{Neuroticism} + 3.729

Regression Table


  • Predictors with coefficients:

    Predictor

    b

    P-value


    Intercept

    3.729

    <0.001


    Neuroticism

    -0.104

    0.003

    • Regression equation can be presented as:

    extAgreeableness=−0.104imesextNeuroticism+3.729ext{Agreeableness} = -0.104 imes ext{Neuroticism} + 3.729

    Regression Table with Standardized Beta

    Predictor

    Unstandardized b

    Standardized Beta

    P-value

    Intercept

    3.729

    -

    <0.001

    Neuroticism

    -0.104

    -0.134

    0.003

    • The standardized Beta correlates to the correlation coefficient with one predictor variable.

    • Regression equation can be represented under standardized terms:
      extAgreeableness=−0.134imesextNeuroticismext{Agreeableness} = -0.134 imes ext{Neuroticism}

    Example for University GPA Prediction

    • To predict University GPA from High School GPA, an unstandardized regression example:

    • Example calculation given High School GPA = 3.0:
      extPredictedcollegeGPA=(0.675)imes(3.0)+1.097=3.12ext{Predicted college GPA} = (0.675) imes (3.0) + 1.097 = 3.12

    JASP Outputs for Linear Regression

    • When using linear regression software like JASP, expect:

    • Model Summary: Comparison of fit across different models.

    • ANOVA: General test for significant variance explanation by predictors.

    • Coefficients: Values of b, Beta, t-statistics, and p-values for each effect overview.

    Example Output from JASP Analysis

    • Linear Regression detailing elements such as the model fit:

    • Describing dependent variables, methods, and regression results.

    Model

    R

    R²

    Adjusted R²

    RMSE

    (Complete results here)

    APA Style Reporting of Results

    • Example reporting the simple linear regression result:

    • "A simple linear regression was calculated to predict college GPA from High School GPA. High School GPA explained a significant proportion of variance in College GPA, R² = .608, F(1, 103) = 159.57, p < .001. High School GPA was positively associated with College GPA, b = .675, t(103) = 12.632, p < .001."

    Learning Objectives for Next Section

    • Compute the sum of squares Y.

    • Convert raw scores to deviation scores.

    • Compute predicted scores from a regression equation.

    • Partition sum of squares Y into sum of squares predicted and sum of squares error.

    • Define R² and compute the standard error of the estimate based on prediction errors.

    • Compute standard error using Pearson's correlation.

    Partitioning the Sums of Squares

    • Regression analyses partition variability into different sources.

    • Total Variability: Includes Explained (from predictor variable) and Error (unexplained variability).

    • Relationship: Total = Explained + Error.

    Concept of Sum of Squared Deviations

    • SSY (Total Variability): The sum of squared deviation scores for the criterion variable Y.

    Notation for Variability

    • Table formatting to reveal the notation on variations:

    • Y = observed outcome

    • $y = ext{deviation score} (Y - M_y)$

    • $y^2 = ext{squared deviation scores}$

    Explained vs. Unexplained Variability

    • Y′=extpredictedscoreY' = ext{predicted score}

    • $y' = ext{deviance of predicted scores}$

    • Y−Y′=extpredictionerrorsY - Y' = ext{prediction errors}

    • Total variability formulated as:

    • SSY = extstyle egin{pmatrix} ext{Total Variability} \ ext{Explained (SSY')} \ ext{Unexplained (SSE)} \ \ ext{Measured as: } ext{ Expl = } ext{(predicted scores)}^2, ext{Unexpl = } ((Y - Y')^2) ext{ for total measure.}\ ext{ } ext{ ext{Sum of errors is such }}}

    R-Squared Calculation

    • R-squared (R²): The percentage of the criterion variable explained by predictors (0 to 1).

    • In simple linear regression with one predictor: R² is the square of the correlation coefficient.

    • R² derives from the ratio:
      R2=racextExplainedSSY′extTotalSSYR^2 = rac{ ext{Explained SSY'}}{ ext{Total SSY}}

    R-squared Calculation Example

    • For this calculation:
      R2=rac1.8064.597oR2=0.39R^2 = rac{1.806}{4.597} o R^2 = 0.39

    Standard Error of the Estimate

    • Indicates accuracy in regression predictions.

    • Example highlighting more prediction error observed in one dataset compared to another (e.g., Figure 1A vs. Figure 1B).

    Definition of Standard Error of the Estimate

    • Measures: The standard deviation of prediction errors closely linked with the sum of squares error (SSE).

    • Formulas: To compute standard error of the estimate:

    • For the population:%
      SE=rac1extoutputoftheformulabasedonpopulation.SE = rac{1}{ ext{output of the formula based on population.}}

    • For the sample: SESE

      • Where $Y$ = actual score, $Y'$ = predicted score, $N$ = number of score pairs.

    Participation Exercise

    • Analyze a provided graph displaying a regression line to quantify the standard error of the estimate based on the visual prediction errors for Y.

    Using Pearson’s Correlation for Standard Error of the Estimate

    • Equations that indicate the derivation of standard error estimates using Pearson’s correlation: $ρ$ (population correlation) and $r$ (sample correlation).

    • Variance measures addressed through calculations involving the sum of squares total (SSY) relative to pairs of scores (N).