p-values explained (StatQuest)

Flashcards covering the concept of p-values, hypothesis testing, thresholds, false positives, and the drug A vs B example from the StatQuest lecture.

What is a p-value?

A number between 0 and 1 that quantifies how confident we should be that there is a difference between two groups (lower means more confidence).

What does a p-value approaching zero indicate?

Strong evidence that the groups are different.

What does a p-value near one indicate?

Little or no evidence of a difference; results may be due to random chance.

What threshold is commonly used to decide if there is a difference?

0.05 (5%).

What does it mean if there is no real difference and we repeat the experiment many times?

About 5% of experiments would yield a p-value below 0.05 (false positives).

What is the null hypothesis in the drug A vs drug B example?

The drugs are the same (no real difference).

What is the alternative hypothesis in this context?

The drugs are different.

What does the p-value help us decide?

Whether to reject the null hypothesis.

Does a small p-value tell us how big the difference is?

No; the p-value does not measure the size of the effect.

Can a small p-value occur with a tiny difference if the sample size is large?

Yes; large samples can detect small differences and produce small p-values.

Can a large p-value occur with a large difference?

Yes, if the sample size is small, the difference may look non-significant.

Why can't we conclude A is better with two groups of two people?

Because the sample is too small; random variation could mislead.

What were the big-number results for A vs B in the example?

A cured 1043 of 1046; B cured 2 of 1434; A looks vastly better.

What were the cure rates for A and B in that big example?

A about 99.7%; B about 0.14% (2/1434).

If the p-value is less than 0.05, what do we conclude?

There is likely a difference between the drugs.

What does a threshold of 0.00001 imply?

We would get a false positive only about once in 100,000 experiments.

What does a threshold of 0.2 imply?

We are willing to accept false positives about 2 out of 10 experiments.

What is 'hypothesis testing'?

The process of deciding whether to reject the null hypothesis.

What is the 'null hypothesis' in this context?

The drugs are the same.

What is the 'alternative hypothesis'?

The drugs are different.

What does a small p-value indicate about the null?

It suggests rejecting the null hypothesis.

Can a small p-value occur when there is no real difference?

Yes; that is a false positive.

What random factors can cause apparent differences in small experiments?

Rare allergies, drug interactions, mislabeling, placebo effects, missed doses.

What does the video say about placebo effects?

Placebo effects can produce apparent cures not due to the drug.

Why is replication important?

To determine whether the observed difference holds up with more data and isn't just due to chance.

What is the range of p-values?

Between 0 and 1.

What does a p-value of 0.9 mean in a run?

We fail to see a difference; results are consistent with no difference.

What does a p-value of 0.01 in a bad run illustrate?

A difference appears, but could be a false positive due to random variation.

What are some random reasons given for differences in results?

Placebo differences, rare allergies, mislabeling, not taking a dose.

What is the relationship between p-values and sample size?

Larger samples can produce smaller p-values for the same effect; smaller samples can give larger p-values.

What does a p-value measure?

Confidence that the observed difference is not due to random chance; evidence against the null.

Can the same data yield different p-values in different runs?

Yes, due to random variation in the sample.

What is a false positive in hypothesis testing?

A small p-value when there is no real difference.

What is the main practical takeaway when you see p<0.05?

There is evidence for a difference, but not necessarily about the size of the difference.

What is the effect of using a very small p-value threshold in critical contexts?

Reduces false positives but may miss true differences (lower power).

What is the effect of using a moderately high threshold like 0.2?

Increases false positives but may catch more true differences.

What does the video say about p-values and the size of the difference?

A small p-value does not imply a large difference.

What is the interpretation of a p<0.05 result in the two-drug example?

We infer that the drugs are different.

What does a p-value of 0.24 imply about drug A vs B?

We are not confident that there is a difference.

In the context of decision making, what do p-values help determine?

They help decide whether to reject the null, not the magnitude of the difference.

What does the cutoff concept do for experiments?

It converts p-values into a yes/no decision about whether there is a difference.

What final message does StatQuest give about p-values?

P-values help you decide about differences, but do not measure effect size; consider the broader context and options to support the channel.