Comprehensive Guide to Inferential Testing: T-Tests, ANOVA, Factorial Designs, and Correlation

Foundations of Inferential Testing and Null Hypothesis Significance Testing (NHST)

• Core Purpose of Inferential Statistics:

  • Inferential testing determines whether a claim or finding observed within a sample can be generalized to a broader population.
  • The sample mean is not assumed to be automatically identical to the population parameter; tests must formally account for sampling error.
  • The fundamental objective is determining whether an observed difference or relationship is due to random chance or represents a real, underlying phenomenon.

• Understanding Sampling Error:

  • Sampling error does not indicate human error, methodological mistakes, or procedural inaccuracies.
  • It represents natural variability and differences among participant responses.
  • Descriptive screening in SPSS (examining group means and standard deviations) provides the empirical grounding to determine whether sufficient differences exist between groups to proceed with inferential testing.

• Logic of Null Hypothesis Significance Testing (NHST):

  • The Null Hypothesis ($H_0$): Assumes there is no effect, no relationship between continuous variables, or no difference between group means in the population.
  • The Alternative Hypothesis ($H_1$): Predicts that an observed change, difference, or association reflects an actual population phenomenon.
  • NHST calculates the probability of obtaining the observed sample data (or results more extreme) assuming that the null hypothesis is true.
  • Phenomena falling along the center/average of the sampling distribution are expected under random variation; true effects occur in the extreme tail margins of the probability distribution.

• Alpha Levels and the p-Value:

  • Significance Threshold ($\alpha$): In psychological science, the standard significance cutoff is set at $\alpha = 0.05$.
  • p-Value Definition: Represents the exact probability of observing the sample data given that the null hypothesis is true ($P(\text{Data} \mid H_0)$), rather than the probability that a hypothesis is "true" in an absolute sense.
  • Decision Criteria:
    • If $p < 0.05$: The probability of obtaining the result by coincidence is extremely small (marginal). The null hypothesis is rejected ($\text{Reject } H_0$), indicating statistical significance.
    • If $p \ge 0.05$: The data falls within expected population variation. The decision is to fail to reject the null hypothesis ($\text{Fail to Reject } H_0$).
    • The phrase "accept the null hypothesis" is never used in scientific reporting; researchers only state that they fail to reject the null hypothesis.

• Statistical Power, Effect Size, and Sample Size Interaction:

  • A test statistic's significance is directly determined by the interplay between the magnitude of the effect size and the sample size ($n$).
  • A weak effect size (e.g., $r = 0.10$) combined with a very small sample size will lack statistical power, failing to cross the $\alpha = 0.05$ significance threshold even if a genuine phenomenon exists.
  • When an inferential test is non-significant, researchers must evaluate whether the sample size was sufficiently powered to detect the anticipated effect size before concluding the effect is absent.

Variable Classification and Study Design Requirements

• Independent Variables (IV):

  • Categorical Variables: Consist of distinct groups, bins, or discrete categories (e.g., day vs. night conditions, academic classes like freshmen vs. seniors).
  • Continuous Variables: Measured along a numerical scale with meaningful quantitative increments where values can increase or decrease continuously.
  • Covariates: When multiple independent variables are evaluated, models should ideally include at least one covariate to account for extraneous variance.

• Dependent Variables (DV):

  • Continuous Requirement: Research designs must utilize a continuous, scale-level dependent variable. Categorical dependent variables require advanced, non-elementary analytical procedures that fall outside standard general linear modeling.
  • Quantity: Studies should focus on a single, well-defined continuous dependent variable to maintain clarity and prevent analytical errors.

• Participant Sampling Explicit Specification:

  • Methodology sections must explicitly specify the precise population origin of sample participants (e.g., explicitly designated as San José State University [SJSU] students or SJSU community members).

Taxonomy of Statistical Tests

• Test Selection Principles:

  • One-Sample T-Test: Compares a single sample mean against an established, fixed, or known population parameter (such as census data, official institutional records, or archival norms).
  • Dependent Samples T-Test (Paired/Repeated Measures): Evaluates whether scores change within the same group of participants measured across two different time points or distinct conditions.
    • Example Application: Measuring student stress levels via an identical survey administered immediately before an intensive $60\,\text{minute}$ yoga session and re-administered immediately upon class completion.
  • Independent Samples T-Test: Compares the means of two mutually exclusive, independent groups on a continuous dependent variable (e.g., men vs. women, STEM vs. non-STEM majors, freshmen vs. seniors, or binary "yes/no" screening responses).
  • One-Way Analysis of Variance (ANOVA): Compares the means of three or more independent groups against a single continuous dependent variable (e.g., comparing multiple college majors, different years in school, or post-hoc groupings such as attachment styles, parenting styles, or personality profiles).
  • Bivariate Correlation (Pearson $r$): Evaluates the linear association between two continuous quantitative variables (designated by a ruler icon in SPSS). Regression models are only pursued if the initial correlation demonstrates statistical significance.

• Descriptive and Graphical Representations:

  • Categorical group differences are graphically screened at the descriptive stage using bar charts or box plots.
  • Relationships between two continuous variables are visualized through scatter plots.
  • Reference charts for test selection are located on lecture slides 13 and 14; comparative tables across the three t-test variations are detailed on slide 30.

One-Sample T-Tests and Degrees of Freedom

• Theoretical Rationale and Formula:

  • Evaluates whether the difference between an observed sample mean ($\bar{x}$) and a known population mean ($\mu$) exceeds what would be expected from standard sampling error ($SE$).
  • $t = \frac{\bar{x} - \mu}{\frac{s}{\sqrt{n}}}$
  • The standard error term ($\frac{s}{\sqrt{n}}$) normalizes the variance by accounting for the sample size ($n$), ensuring sample sizes (e.g., $n = 1000$) can be accurately balanced against massive census parameters (e.g., $N = 100000$).

• Sample Size Targets:

  • While hand calculations are used for very small groups ($n < 30$), project teams of two members should aim for a minimum of $20$ to $30$ participants, maximizing recruitment wherever possible.

• Degrees of Freedom ($df$):

  • Defined as the number of values in the final calculation of a statistic that are free to vary.
  • Calculated as the total sample size minus one:
  • $df = n - 1$

• Concrete Calorie Estimation Case Study (One-Sample):

  • Hypothesis: Evaluating whether individuals accurately estimate the caloric content of a standard food item (a cookie) compared to the verified true calorie value.
  • Known Population Parameter: True calorie value is $\mu = 250\,\text{calories}$.
  • Sample Data: A sample of $n = 10$ participants produced a mean estimate of $\bar{x} = 212\,\text{calories}$ with a standard deviation of $s = 39.17\,\text{calories}$.
  • Degrees of Freedom: $df = 10 - 1 = 9$.
  • Inference Results: Evaluated against a two-tailed distribution, the resulting test statistic yielded a significance of $p = 0.01$.
  • Conclusion: Because $p = 0.01 < 0.05$, the null hypothesis is rejected. The data demonstrates that individuals significantly underestimate caloric content.

• One-Tailed vs. Two-Tailed Hypotheses:

  • Two-Tailed Design: Non-directional; tests for any significant difference (either higher or lower) across both extremes/tails of the distribution. This is the mandatory standard for course analyses to avoid premature omission of unexpected effects.
  • One-Tailed Design: Directional; isolates the critical region to a single predicted tail (e.g., testing strictly for underestimation). While it lowers the critical threshold required on that specific side, it eliminates the ability to detect significant findings in the opposing direction.

• Critical Values and Decision Boundaries:

  • Obtained test statistics ($t$) are benchmarked against critical value tables ($t_{crit}$) or historical distribution metrics (such as Balasci scores).
  • Significance Check Example: At $\alpha = 0.05$ with a critical threshold of $t_{crit} = 2.064$:
    • An obtained statistic of $t = 4.06$ surpasses the threshold ($4.06 > 2.064$), leading to the rejection of the null hypothesis.
    • An obtained statistic of $t = 0.0521$ falls well below the cutoff ($0.0521 < 2.064$), failing to reject the null hypothesis due to lack of statistical strength.

• APA Reporting Standards:

  • Formal write-ups must report the degrees of freedom in parentheses directly after the test statistic, provide the exact p-value, preserve directional signs (negative signs must not be omitted), and contextualize findings with sample size and effect size.

Dependent and Independent Samples T-Tests

• Dependent Samples T-Test (Within-Subjects Design):

  • Employs identical participants measured across two distinct intervals or operational conditions.
  • Concrete Scenario: Administering an emotional regulation inventory to the same cohort of students at $8:00\,\text{AM}$ (upon early university arrival) and re-testing them at $8:00\,\text{PM}$ (late departure).
  • Error Reduction: Substantially diminishes standard error variance by holding subject-level individual differences constant across conditions.

• Independent Samples T-Test (Between-Subjects Design):

  • Compares two isolated, non-overlapping groups on a continuous metric.
  • Can be derived naturally from demographic splits (freshmen vs. seniors) or generated via binary screening questions in survey designs ("yes/no").
  • Mathematical Concept: Replaces the population parameter $\mu$ with a second empirical group mean ($\bar{x}_2$), evaluating the difference between group means ($\bar{x}_1 - \bar{x}_2$) divided by the pooled standard error of both groups.
  • Concrete Calorie Estimation Case Study (Two Groups):
    • Hypothesis: Evaluating whether dietary habits (habitual junk food eaters vs. non-junk food eaters) influence calorie estimation accuracy for a single food item.
    • Test: Two-tailed independent samples t-test.
    • Result: Analysis yielded an obtained significance of $p = 0.02$.
    • Decision: Since $p = 0.02 < 0.05$, the null hypothesis is rejected, demonstrating a statistically significant difference in calorie estimation accuracy between dietary groups.

One-Way Analysis of Variance (ANOVA)

• Methodological Purpose:

  • Employed when comparing means across three or more independent groups against a single continuous dependent variable.
  • Protects against the inflation of familywise Type I errors that occurs when running multiple separate pairwise t-tests.

• Understanding Statistical Errors:

  • Type I Error (False Positive): Rejecting the null hypothesis when it is actually true (equivalent to the boy who cried wolf when no wolf was present; a false alarm).
  • Type II Error (False Negative): Failing to reject the null hypothesis when an actual effect exists (failing to detect the wolf when it is present).

• Partitioning of Variance and the F-Ratio:

  • ANOVA mathematically decomposes total variance into two distinct sources:
    • Between-Groups Variance: Measures the dispersion of each individual group mean around the grand mean. Reflects the experimental manipulation or treatment effect.
    • Within-Groups Variance: Measures the dispersion of individual participant scores around their respective group mean. Reflects baseline random error and individual differences.
  • The F-Ratio Statistic:
  • $F = \frac{\text{Between-Groups Variance}}{\text{Within-Groups Variance}} = \frac{\text{Effect}}{\text{Error}}$
  • Interpretation: An F-ratio yielding large numbers (double or triple digits) signifies substantial group separation over error variance. Small fractional values (decimals close to zero) indicate group differences are indistinguishable from random noise.

• Concrete Calorie Estimation Case Study (One-Way ANOVA):

  • Independent Variable: Academic training background with three levels (Psychology Majors, Nutrition Majors, Dietitians).
  • Dependent Variable: Estimated calorie content of a test food item.
  • Observed Group Means:
    • Psychology Majors: Mean guess = $187\,\text{calories}$.
    • Nutrition Majors: Mean guess = $195\,\text{calories}$ (exhibited the greatest variability and included one additional participant).
    • Registered Dietitians: Mean guess = $238\,\text{calories}$.
  • Inference Results: Obtained statistic of $F = 9.92$ with a significance value of $p < 0.001$.
  • Decision: Because $p < 0.001 < 0.05$ and the F-ratio is high ($9.92$), the null hypothesis is rejected. There is a statistically significant overall difference between professional backgrounds.

• Distribution Geometry and Post-Hoc Comparisons:

  • F-Distribution Characteristics: Unlike the symmetric t-distribution, the F-distribution is non-negative and right-skewed (positively skewed).
  • The Post-Hoc Requirement: The omnibus F-test only indicates that at least one group differs from the others; it cannot identify which specific pairs are statistically different. Pairwise post-hoc comparisons must be conducted:
    • Bonferroni Correction: Highly conservative; strictly controls Type I error rate, but reduces statistical power; optimal for small samples.
    • Tukey's Honestly Significant Difference (HSD): Balanced approach offering an optimal compromise between error control and power; the primary recommended post-hoc test in psychology.
    • Fisher's Least Significant Difference (LSD): Highly liberal and least strict; does not adequately control for familywise Type I error inflation; least recommended.

Factorial Designs

• Structural Architecture:

  • Incorporate two or more categorical independent variables, with each variable containing two or more distinct levels (e.g., a $2 \times 2$ factorial design).
  • Allows researchers to examine conditional boundaries ("for whom or under what conditions does an effect hold true?").

• Concrete Factorial Scenario ($2 \times 2$ Design):

  • Independent Variable 1: Academic Major (2 levels: Psychology vs. Nutrition).
  • Independent Variable 2: Food Type (2 levels: Cookie vs. Burger).
  • Dependent Variable: Continuous calorie estimates.
  • Observed Patterns:
    • Psychology majors estimated both cookies and burgers at similarly low caloric values.
    • Nutrition majors provided higher, more accurate estimates, averaging between $210\,\text{calories}$ and $220\,\text{calories}$ across items.

• Decomposition of Effects in Factorial Models:

  • Main Effects: The independent, isolated effect of one independent variable on the dependent variable, averaged across all levels of the other independent variable (evaluated separately for Main Effect 1 and Main Effect 2).
  • Interaction Effect: Evaluates whether the effect of one independent variable changes as a function of the level of the second independent variable.
  • Reporting standards require isolating and reporting statistical tests for both individual main effects and the interaction effect.

Questions & Discussion

• Audience Inquiry and Transition:
  • Instructor Prompt: The floor was opened for questions regarding group comparisons, t-test models, and ANOVA analyses.
  • Audience Response: No immediate questions were raised by attendees.
  • Procedural Directives: Students were instructed to hold remaining questions for the dedicated practical laboratory session immediately following the lecture.

Correlation Analysis

• Foundational Principles of Linear Correlation:

  • Evaluates the bivariate linear association between two continuous (scale/quantitative) variables without manipulating experimental conditions or artificially assigning groups.
  • Quantifies the strength and direction of association using the Pearson correlation coefficient ($r$), testing how far the observed coefficient deviates from zero.
  • Null Hypothesis ($H_0$): Assumes the population slope is flat and no linear association exists ($\rho = 0$).
  • Alternative Hypothesis ($H_1$): Predicts a non-zero linear relationship ($\rho \neq 0$).

• Degrees of Freedom for Pearson Correlation:

  • Because the model estimates two continuous scale parameters simultaneously without designating fixed independent and dependent roles, two degrees of freedom are lost:
  • $df = n - 2$

• Properties of the Correlation Coefficient ($r$):

  • Bounded strictly between $-1.0$ (perfect negative correlation) and $+1.0$ (perfect positive correlation), with $0.0$ representing no linear relationship.
  • The statistic $r$ intrinsically standardizes variance and accounts for error, directly scaling with the underlying $t$-statistic (a large $r$ corresponds to a high $t$-value, indicating an effect unlikely to be driven by chance).

• Concrete Calorie and Body Weight Correlation Study:

  • Hypothesis: A health psychologist investigated whether individuals' calorie estimation accuracy is associated with their own body weight.
  • Design: Two-tailed correlation without a predicted direction.
  • Sample: $n = 22$ university students.
  • Results: Obtained correlation of $r = -0.21$, reflecting a weak negative correlation.
  • Significance: Inferential analysis yielded a p-value of $p = 0.348$.
  • Decision: Because $p = 0.348 > 0.05$, the researcher fails to reject the null hypothesis. There is no evidence of a linear association between calorie estimation accuracy and body weight.

• Analytical Synthesis Across Tests:

  • Every inferential test computes an Effect-to-Error ratio:
    • T-tests and ANOVA (F): Conceptualized as statistical siblings; both compare categorical groups against a continuous dependent variable.
    • Correlation ($r$): Conceptualized as a statistical cousin; evaluates associations across continuous metrics without partitioning participants into categorical groups.

Methodological Assumptions, Pitfalls, and Best Practices

• Misinterpretation of Failing to Reject the Null:

  • Failing to reject the null hypothesis does not prove that an effect is absent, nor does it prove that the null hypothesis is true.
  • It strictly denotes that the empirical evidence gathered from the given sample size was insufficient to surpass the random error threshold.

• Data Screening and Assumption Verifications (Lab Part A):

  • Prior to running inferential models, researchers must systematically check distributional assumptions:
    • Normality Checks: Confirming sample distributions conform to normal, bell-shaped distributions without extreme skewness.
    • Outlier Screening: Identifying and addressing influential outliers that distort group means and artificially inflate error variance.
    • Categorical Group Sizes: Ensuring sufficient, balanced participant counts within each categorical level before proceeding with inferential testing.

• Avoiding Rigid Dichotomous Thinking at Cutoff Boundaries:

  • Researchers must avoid hyper-rigid interpretations of marginal significance boundaries (e.g., viewing $p = 0.049$ and $p = 0.051$ as fundamentally opposite states of nature).
  • Full transparency is required when reporting results adjacent to the $\alpha = 0.05$ cutoff, acknowledging that minor increases in sample size and statistical power could shift marginal non-significant trends across the threshold.