Statistical Significance and Related Concepts

Definition: Inferential statistics involve using data collected from a sample to make inferences about a population.
Context: After collecting data and examining its characteristics (Descriptive Statistics), we can assess the meaningfulness of the sample data.
Tools for inference:
- Statistical significance
- Effect sizes
- Confidence intervals
Purpose: Inference allows us to go beyond descriptive statistics and derive conclusions about the broader population.

Definition: Statistical significance refers to the likelihood or probability that a sample observation reflects what is happening within the entire population.
Importance: Establishes two core points:
- The observed phenomenon in the sample is also occurring in the larger population.
- The observation is statistically rare—greater than chance—and thus notable and deserving of further inquiry.

Knowledge of Normal Distribution:
- Most common values in a distribution are near its center.
- Least frequent values are found at the distribution's tails.
- Statistically significant values are identified in these tail ends.

Relevance: Probability is crucial in inferential statistics.
Sampling Distribution of Means:
- Example context: Selecting an infinite number of random samples from a population (e.g., men’s shoe sizes).
- Enables calculation of the probability of obtaining specific values within the population.

Types of Hypotheses:
- Null Hypothesis: suggests no effect, indicating no statistically significant difference in means between the sample and population.
- Alternative Hypothesis: suggests that a difference does exist.
  - One-tailed hypothesis examples: Sample mean is greater than the population mean.
  - Two-tailed hypothesis examples: Sample mean differs from the population mean without specifying direction.

Distinction between Difference and Significance: A difference in means does not automatically imply statistical significance due to potential random sampling error.
Determining Significance:
- Important criterion: The difference must reach an alpha level ($ ext{p-value}$) of .05, meaning the chance of a result occurring by chance is less than 5%.
- Techniques: Statistical tests (e.g., z-score) help determine this probability.
  - Significant if p < .05.

Population Mean: 11 (men’s shoe sizes)
Sample Mean: 9
Questions posed: Is this difference significant?
- Evaluate probability:
  - Example probabilities might include p = .30 or p = .01.

Challenges:
- Forces researchers into binary yes-or-no questions regarding significance.
- Findings may lack generalizability, truth, meaning, and replicability.
- Heavily dependent on sample size: Minor increases in sample size might lead to statistical significance.
- Necessity of additional metrics like effect size and confidence intervals to fully understand results.

Definition: Effect size gauges the magnitude of difference between two groups; indicates the level of impact (small vs. large).
Illustrative Examples:
- Strong relationship: Studying and exam scores.
- Weak relationship: Text message frequency and relationship satisfaction.
Stability: Effect size remains consistent regardless of sample size.
Complementary Importance: Effect size, combined with statistical significance, provides crucial information to assess the reliability of results.
Common Metrics:
- Cohen’s d: Common formula associated with t-tests to measure effect size.
- Correlation coefficient (r): Effect size associated with correlation tests.

Definition: Practical significance assesses whether differences between groups hold real-world relevance.
Consideration: This is often subjective, influenced by the testing context and ultimate goals.
Example Contexts:
- A 3% change in preference for ice cream flavors may be statistically significant but not practically significant.
- A 3% mortality rate of a new clinical drug poses real-world implications.

Definition: Confidence Intervals (CIs) represent a range of values believed, to a certain confidence level, to contain the population parameter.
Accuracy Assessment: Even with unknown population parameters, they provide a better estimate of population representation.
Levels of Confidence:
- 95% confidence intervals are associated with p = .05.
- 99% confidence intervals are associated with p = .01.
Integral Components: For a holistic understanding, statistical significance, effect size, and confidence intervals should be considered together.