Quality Control Lecture

null hypothesis

• default assumption that states there is no real relationship/effect between variables in an experiment

P-value / Statistical Significance

• probability of obtaining the results you got, if the null hypothesis is true

• small means it is unlikely to occur by chance ~ lower = stronger

p < .05

• statistically significant

• can reject null hypothesis and conclude that your result is real effect

p > .05

• not statistically significant

• not enough evidence to rule out chance results

null hypothesis significance testing

• how unlikely it would be for you to find the data you found if there were no effect of interest

• if the null hypothesis is true, how likely are the realists you got

Example Testing

• NH: dems and reps like red the same

• p < .01

• p value small, so NH is rejected and results were not chance

Sample Size

• necessary amount depends on the study

• all else equal, more data will allow you more confidence in findings

Between Subject: Sample Size Testing

• variable manipulated across different groups of people

• 50 sleep before exam, 50 don’t

• problem: data on either side is tested on different people, results could just be because the people are different regardless of sleep

Within Subject: Sample Size Testing

• test two different things on same person

• 50 sleep first, then don’t sleep; other 50 don’t sleep first, then sleep

• same person shows both results so it is more representative

Positive to Between Subject Testing

some things can only be tested once, so within subject is not possible

Negative to Within Subject Testing

• the subjects may begin to figure out what is being tested, and alter their behavior in accordance to what they think is being gathered

• capacity to skew results

If you do enough tests…

• you can discover almost anything you want

• manipulation of data to display results you intended, while still using the accurate data

• test many things, hide failures, and project only success ~ mislead readers

Researcher Degrees of Freedom

• undisclosed flexibility in experimental design and analysis that can lead to suspicious effects

• if you do enough tests, you can discover almost anything

  • at least one test will be bound to show an interesting result