Lecture 11: Statistical Inference

Probability is a measure of how likely an event is to occur, ranging from 0 (impossible) to 1 (certain).
Denoted as $p$ , it can be expressed as a fraction, decimal, or percentage.
Example: Dealing a card; the probability of getting a heart is 1 in 4.
$p = \frac{1}{4} = 0.25$ or 25% of the time.
$p$ ranges from 0 (never) to 1 (always).

Probability is used to determine if a particular score or observation belongs to a specific distribution.
This helps in making inferences about whether an effect or difference is statistically significant.
Example: Comparing a stroke patient's reading score to the distribution of reading scores of healthy controls to see if brain damage impaired their performance.

Standardized reading test: A test designed to measure reading ability, with scores following a normal distribution for the age-matched population of healthy people.
- Mean = 50 (average score in the healthy population)
- Standard Deviation = 10 (variability in scores within the healthy population)
- Patient’s Score = 28
Question: Did the brain damage cause a decrease in reading ability for this patient?

Null Hypothesis (H0): A statement of no effect or no difference.
- The patient’s reading ability does not differ from that of healthy people.
Experiment/Alternative Hypothesis (H1): A statement that contradicts the null hypothesis, suggesting an effect or difference.
- The patient’s reading ability is lower than that of healthy people.
- This is a directional hypothesis, specifying the direction of the effect.

Assume the null hypothesis (H0) is true (i.e., there is no difference) as a starting point for the analysis.
The score from the patient belongs to the distribution of control scores due to individual differences (random variation).
Calculate the probability of obtaining a score as extreme as or more extreme than the one obtained, under the assumption that H0 is true. This probability is known as the p-value.
If this probability is very small, reject the null hypothesis (H0) and accept the alternative hypothesis (H1). This implies the patient's reading ability is worse/better than healthy controls and the difference is not just due to individual differences.
If this probability is not very small, retain the H0. This suggests the patient's reading ability is no different from healthy controls.

Assuming H0, calculate how probable it is to get a score of 28.
Z-score: A measure of how many standard deviations an individual data point is from the mean.
Z-score formula: $z = \frac{(x - mean)}{SD}$
For the patient’s score: $z = \frac{(28 - 50)}{10} = -2.2$
From the z-table, the percentage of people with a score of 28 or lower is 0.0139.
Interpretation: If you randomly sample one person from the population of healthy people, 1.39% of the time you would get a score of 28 or lower.

The conventional threshold for rejecting the H0 is 5% or p = 0.05, also known as α (alpha), which represents the significance level.
If p < 0.05, reject H0.

Patient with brain damage has a probability of getting a lower score, $p = 0.0139$ .
Since this is a directional hypothesis and p < 0.05, we reject the null hypothesis.
We reject that the ability of the patient is not different from healthy controls.
The patient’s score did not come from the same population as scores for healthy people.
The difference is not just due to individual variance.
We accept the alternative/experimental hypothesis that the patient’s reading score was significantly lower than healthy controls.

Critical values are real-life scores that are at the threshold (cut-off point) for statistical significance.
Scores above one threshold are significantly higher than the population scores.
Scores below one threshold are significantly lower than the population scores.

Using the previous example, we are only interested in the lower critical score.
A z-score of -1.645 is associated with $p = 0.05$ . This is negative as it is below the mean (approximately 1.6 SDs below).

Critical z-score = -1.645
Normal distribution: Mean = 50, SD = 10
$z = \frac{(critical \ value – mean)}{SD}$
$-1.645 = \frac{(critical \ value – 50)}{10}$
$-16.45 = critical \ value – 50$
$critical \ value = 50 - 16.45 = 33.55$
Reading scores of less than 33.55 are significantly lower than that of controls.

Type I error occurs when we incorrectly reject the null hypothesis (H0).
This means deciding the score is significantly higher/lower when it is not.
Also known as a "false positive".
When the alpha level is 0.05, our inference is wrong 5% of the time, meaning we find significance wrongly 5 times out of 100.
The probability of a Type I error is equal to the alpha level.

Reducing the alpha level (e.g., to 0.01) to reduce Type I error increases the risk of a Type II error.
Type II error is failing to reject the null hypothesis when we should.
For example, finding a score to be not significantly different from the population when it is.
Also known as a "false negative".
Decreasing the likelihood of Type I error increases the likelihood of Type II error and vice versa, illustrating a trade-off between these two types of errors.

Related to predictions about the data.
Directional (one-tailed) alternative hypothesis: Specifies the direction of the effect.
- "The patient’s score will be lower than the scores of healthy controls."
- "Grades on your second essay will be better than your first."
Non-directional (two-tailed) alternative hypothesis: Simply states that there will be a difference, without specifying the direction.
- "The patient’s score is different from the scores of healthy controls."
- "Grades on your second essay will differ from those on your first."
The choice between directional and non-directional hypotheses is based upon prior research and theoretical expectations.

Alternative hypothesis (H1): States the direction of the effect.
- “The patient’s score is lower than the scores of healthy controls.”
- “The patient’s score is higher than the scores of healthy controls.”
Null hypothesis (H0): A statement of no effect or no difference.
- “The patient’s score does not differ from the scores of healthy controls.”

Alternative hypothesis (H1): States that there will be a difference.
- “The patient’s score is different from the scores of healthy controls.”
Null hypothesis (H0): A statement of no effect or no difference.
- “The patient’s score does not differ from the scores of healthy controls.”
H0 does not differ whether the hypothesis is directional or not.

Using probability theory to make inferences about a population from sample data.
This involves generalizing findings from a subset of the population to the entire population.
Example: Using data from a 20-participant sample.
Generalizing from the sample mean and standard deviation to the general population.
Estimating population parameters (e.g., the average weight of 18-year-olds in Britain) by taking a sample.

We cannot be 100% sure that the estimated mean/standard deviation (from the sample) equals the population mean/SD.
However, we can state the probability of our inference being wrong (or correct), providing a measure of confidence in our findings.