Lecture 5: Hypothesis Testing with z Tests

Overview of the Research and Data Analysis Pipeline

• Analysis of behavioural data follows a structured flow from research question to dissemination.

• Research Design and Planning (218 Content):

  • Development of research questions (e.g., "Is CBT effective for depression in Canadian youth?").

  • Conducting literature reviews to identify gaps in existing research.

  • Hypothesis development: Defining hypotheses and operationalizing variables.

  • Planning the study: Using effect size and power to calculate necessary sample size ($0.8$ power typical).

• Data Collection (217 Content):

  • Choosing an experimental design, such as a randomized controlled trial (e.g., CBT Group vs. Control Group).

  • Choosing recruitment methods (e.g., social media advertising).

• Data Analysis (218 Content):

  • Choosing statistical tests (z-test, t-test, regression, ANOVA).

  • Running statistical analysis using software (e.g., SPSS).

  • Calculating descriptive and inferential statistics (e.g., change in depression scores).

• Interpretation (218 Content):

  • Using knowledge of confidence intervals and effect sizes.

  • Considering replicability and generalizability (e.g., applying results outside of a specific region or social media user base).

• Dissemination (217 Content):

  • Designing and presenting posters or writing papers.

  • Describing statistical results using APA format.

Criteria for Selecting Statistical Tests

• z-Test:

  • Used when both the population mean ($\mu$) and the population standard deviation ($\sigma$) are known.

  • Comparison is made between a sample mean ($M$) and a population mean ($\mu$) or a set value.

• Single-Sample t-Test:

  • Used when the population mean ($\mu$) is known, but the standard deviation is unknown.

• Independent-Samples t-Test:

  • Used when comparing different samples of participants (e.g., Treatment Group vs. Control Group).

• Paired-Samples t-Test:

  • Used when comparing paired observations from the same sample of participants.

• One-Way Between-Subjects ANOVA:

  • Used when comparing more than two samples of participants on one variable.

• Two-Way (or higher) Between-Subjects ANOVA:

  • Used when comparing different samples of participants across two or more variables.

• One-Way Within-Subjects ANOVA:

  • Used when comparing more than two sets of observations from the same sample.

• Two-Way (or higher) Within-Subjects ANOVA:

  • Used when assessing two or more variables within the same sample.

Null Hypothesis Significance Testing (NHST)

• NHST is a type of inferential statistical test used to test the Null Hypothesis ($H_0$).

• The Research Hypothesis ($H_1$):

  • The hypothesis formulated by the researcher based on theory.

  • Also referred to as the "Alternative Hypothesis."

  • Example: "People taking the treatment will have lower symptoms than people not taking the treatment."

• The Null Hypothesis ($H_0$):

  • The negation of the research hypothesis.

  • Symbolized as $H_0$.

  • $H_1$ and $H_0$ must be mutually exclusive and exhaustive.

  • Example: "People taking the treatment will NOT have lower symptoms than people not taking the treatment."

• Outcomes of NHST:

  • Reject the null hypothesis.

  • Fail to reject the null hypothesis.

• Rationale for testing $H_0$ (Falsification):

  • Observations consistent with a theory do not prove it (e.g., observing a million white swans does not prove "all swans are white").

  • One inconsistent observation (a black swan) allows for the rejection of a hypothesis.

  • Scientific advancement occurs by looking for observations that reject the negation of the research hypothesis (the null hypothesis).

The Process and Standard of Unlikelihood

• The NHST Process:

  • Assume the null hypothesis is true.

  • Record observations.

  • Compare the observations to the distribution implied by the null hypothesis.

  • If the result is "very unlikely" under the assumption that $H_0$ is true, reject $H_0$.

• Alpha ($\alpha$):

  • The cutoff value for what is considered "very unlikely."

  • Traditionally, $\alpha = 0.05$ ($5\%$).

  • If observations as extreme as the ones recorded would occur $5\%$ of the time or less by random chance (assuming $H_0$ is true), the null hypothesis is rejected.

Directionality: One-Tailed vs. Two-Tailed Tests

• One-Tailed Test:

  • Used for directional hypotheses (e.g., "fertilizer increases pumpkin weight").

  • The alpha cutoff ($5\%$) is placed entirely in one tail of the distribution.

• Two-Tailed Test:

  • Used for non-directional hypotheses (e.g., "light change makes pumpkins a different size").

  • The alpha is split between both tails: $2.5\%$ in the left tail and $2.5\%$ in the right tail (totaling $5\%$).

• Traditional Standard:

  • Psychology traditionally uses two-tailed tests to prevent researchers from "cheating" by claiming a directional hypothesis after seeing results (HARKING).

Error Types in Hypothesis Testing

• Type I Error:

  • Occurs when the null hypothesis is actually true, but the researcher erroneously rejects it.

  • Commonly called a "false positive."

  • If $\alpha = 0.05$, the expected rate of Type I error when the null hypothesis is true is exactly $5\%$.

• Type II Error:

  • Occurs when the null hypothesis is false, but the researcher fails to reject it.

  • Commonly called a "false negative."

  • If the null hypothesis is true, a Type II error cannot occur (the chance is $0\%$).

z-Scores: Calculations and Percentiles

• Characteristics of z-Score Distributions:

  • The mean of a z-score distribution is always $0$.

  • The standard deviation ($\sigma$) is always $1$.

  • The shape (skew, etc.) is the same as the original raw scores.

• Formula for Population z-Score:

  • $z = \frac{X - \mu}{\sigma}$

• Example: Pacific Spirit Park Tree Heights:

  • Given: $\mu = 21\text{ m}$, $\sigma = 8\text{ m}$, tree height $X = 29\text{ m}$.

  • Calculation: $z = \frac{29 - 21}{8} = \frac{8}{8} = 1$.

  • Percentile Calculation:

    • A z-score of $1$ corresponds to the $84.13$ percentile ($50\% + 34.13\%$).

    • Probability of a tree being taller than $29\text{ m}$ is $15.87\%$

    • Probability of a tree having a more extreme value (further from the mean in either direction) is $31.74\%$ ($2 \times 15.87\%$\text{)}.

The Distribution of Means

• Definition: The sampling distribution generated by taking many samples of size $N$ from a population and recording their means in a histogram.

• Key Properties:

  • If $N$ is large enough (usually $N \geq 30$), the distribution of means will be a normal distribution, regardless of the population's shape.

  • The mean of the distribution of means ($\mu_M$) is equal to the population mean ($\mu$).

  • The standard deviation of the distribution of means is called the Standard Error ($\sigma_M$).

• Standard Error Formula:

  • $\sigma_M = \frac{\sigma}{\sqrt{N}}$

  • The Standard Error is an indicator of how much error occurs when using a sample mean to estimate the population mean. It decreases as the sample size ($N$) increases.

The z-Statistic and the z-Test

• The z-statistic applies the logic of the z-score to a distribution of means.

• Formula for z-Statistic:

  • $z = \frac{M - \mu_M}{\sigma_M}$

• Assumptions of the z-Test:

  • The outcome variable is a continuous measure (otherwise use a nonparametric test).

  • Participants are randomly selected.

  • The population is normally distributed (or $N$ is large enough to rely on the central limit theorem).

• Parametric vs. Nonparametric Tests:

  • The z-test is a parametric test, meaning it requires population parameters.

  • Nonparametric tests are used for nominal/ordinal data or extreme violations of normality.

Six Steps of Hypothesis Testing: Pumpkin Growth Example

• Scenario: Fertilizer influence on pumpkin weight. Known population: $\mu = 6.3\,kg$, $\sigma = 2.1\,kg$. Sample: $N = 100$, $M = 6.77\,kg$.

1. Identify Populations and Assumptions:

   • Population 1: All pumpkins (no fertilizer).

   • Population 2: All pumpkins (fertilizer).

2. State Hypotheses (Two-tailed):

   • $H_1: \mu_2 \neq \mu_1$

   • $H_0: \mu_2 = \mu_1$

3. Determine Characteristics of Comparison Distribution:

   • $\mu_M = \mu = 6.3\,kg$

   • $\sigma_M = \frac{2.1}{\sqrt{100}} = \frac{2.1}{10} = 0.21$

4. Determine Critical Values:

   • For $\alpha = 0.05$ (two-tailed), the critical values are $\pm 1.96$.

5. Calculate Test Statistic (z-statistic):

   • $z = \frac{6.77 - 6.3}{0.21} = \frac{0.47}{0.21} = 2.24$

6. Make a Decision:

   • Compare observed $z (2.24)$ to critical $z (1.96)$.

   • Since 2.24 > 1.96, reject the null hypothesis.

Determining the p-Value: Caffeine and Extraversion Example

• Scenario: Caffeine effect on extraversion ($1-5$ scale). Population $\mu = 2.95$, $\sigma = 0.87$. Sample: $N = 40$, $M = 3.27$. Alpha $= 0.05$.

1. Hypotheses: $H_1: \mu_{caffeine} \neq \mu_{non-caffeine}$; $H_0: \mu_{caffeine} = \mu_{non-caffeine}$.

2. Standard Error: $\sigma_M = \frac{0.87}{\sqrt{40}} \approx 0.1376$.

3. Critical Value: For $\alpha = 0.05$, two-tailed, critical $z = \pm 1.96$.

4. z-Statistic: $z = \frac{3.27 - 2.95}{0.1376} = \frac{0.32}{0.1376} = 2.33$.

5. Decision: $2.33$ is more extreme than $1.96$, so reject $H_0$.

6. Calculate p-value:

   • The p-value is the decimal probability of observing a result as extreme or more extreme than observed, assuming $H_0$ is true.

   • Look up $z = 2.33$ in the z-table.

   • $\% \text{ in tail} = 0.99\%$.

   • To account for both tails (two-tailed test): $2 \times 0.99\% = 1.98\%$.

   • In decimal notation: $p = 0.0198$ (rounded to $0.02$).

• Conclusion: Since p < \alpha (0.02 < 0.05), the result is "statistically significant," and the null hypothesis is rejected.

Questions & Discussion

• Question: If the $\alpha = 0.05$ and the null hypothesis is true, what is the expected rate of Type I error?

  • Response: It is exactly $5\%$. Alpha defines the probability of rejecting the null hypothesis by chance when it is actually true.

• Question: If the $\alpha = 0.05$ and the null hypothesis is true, what is the expected rate of Type II error?

  • Response: It is $0\%$. Type II errors can only occur if the null hypothesis is false.

• Question: If the values in our results have a mean with a z-statistic of $3$, what percentage of means in the distribution are more extreme?

  • Response: Looking at the two tails, the value far beyond $3$ is very small. For $z=3$, the percentage more extreme is approximately $0.26\%$ ($0.13\%$ in each tail). Since 0.26\% < 5\%, we reject $H_0$.