Statistics and Research Methodology in Sports Science

The context for this discussion is the 2026 conference, specifically addressing people interested in health mechanisms and sports science.
Sports science is divided into health-related studies and the sports science of performance.
Performance science focuses on methodology for training and daily sport-specific applications.
Science is defined fundamentally as a search for truth.
Historically, truth is sought through data (large numbers) to make inferences from a sample to a population.
Logic provides the framework for science, which is now integrated with Artificial Intelligence (AI) and Big Data Science.

Complex statistics are not always a necessity in science.
Significant scientific breakthroughs have occurred through observation rather than complex statistical modeling:
- Heliocentric sphere: The theory that the Earth revolves around the sun (noted since Aristotle).
- Theory of Evolution.
- Laws of Inheritance.
- Germ Theory.
Ivan Pavlov and Classical Conditioning:
- Pavlov originally used observation.
- He instrumented one dog and wrote his first paper in $1897$ .
- He later observed $30$ dogs using observation rather than statistics to establish classical conditioning principles.
Law of World (Three Laws):
- Only the third law involved any statistics at all.
- These include epidemiological findings.
Discovery of DNA structure.
Oxygen Uptake:
- The original paper by A. V. Hill was based on two people (including one of his "rabbit stings").
- Findings were developed through replication and repeated observation.
COVID-19 Vaccine development: Breakthroughs occurred through careful observation and replication rather than solely through sophisticated complex statistics.
In sports science, many concepts were first developed through observation and only confirmed later through statistical processes.

Scientific intuition involves informed judgments, hypothesis generation from experience, and pattern recognition using internalized scientific principles.
The relationship between signal and noise is described by the signal-to-knowledge ratio:
- Signal: The specific effect an investigator hopes will happen.
- Noise: Factors that prevent finding the signal (variability).
Group variance vs. Between-group variance influences this ratio.
Population Noise vs. Contextual Noise:
- Large populations have more variability, but small samples (like elite athletes) have unique noise.
- Sources of noise in athletes: Travel, unfamiliar environments, media attention, fatigue, micro-injuries, and human injuries.
- Bill Pranker noted the importance of context in this noise.
Small sample sizes in sports science ( $n$ ) increase the risk of Type II errors.
The Adaptation Window:
- Untrained people: Large window of adaptation ( $\text{Large Signal}$ ) despite high noise, making effects easier to find.
- Elite athletes: Decreased signal as athletes reach peak performance, with high biological and contextual noise, making effects harder to detect.

Null Hypothesis ( $H_0$ ): Assumes no real difference or relationship between variables; observed differences are attributed to random variation.
Limitations of NHST:
- It does not tell the practicality of the magnitude of change.
- It encourages a "black or white" view of research (it happened or didn't).
- It shifts focus away from the practicality of findings.
- P-values ( $p$ ) are substantially dependent on sample size.
- Large samples can produce significant $p$ values for effects that mean almost nothing.
- Arbitrary thresholds (e.g., $p < 0.05$ ) can lead to absurd conclusions if $0.051$ is considered failure compared to $0.05$ .
Effect Size:
- Provides a measure of the magnitude of change.
- Generally does not depend on sample size, though larger samples provide greater precision.
- Does not depend on statistical significance.
- Includes measures like Pearson's correlation, which conversely does depend on probability/significance.
The speaker questions if science has supplanted logic with a "statistical religion," comparing some academic arguments to debates over "how many angels can stand on the head of a pen."

Hypotheses are not always necessary; asking a question may be more appropriate for exploratory research.
Forcing a hypothesis can hold research back, whereas exploratory research guides future investigations.
Sports Science Context:
- Complex real-world contexts preclude tightly controlled studies.
- Ecological Validity: Paramount in sports science. Research should mimic how the sport actually operates.
Comparison Table (Exploratory vs. Confirmatory):
- Aim: Exploratory aims to ask what happened; confirmatory aims to test a specific hypothesis.
- Data: Exploratory uses small numbers and asks questions; confirmatory drives confirmation through a priori hypothesis.

Overfitting and Data Mining:
- Real-world systems are non-linear and highly variable.
- Complex statistics can create an "illusion of precision."
The Black Swan: A book by Taleb that addresses these concepts.
Cases where statistics favored findings later debunked by replication:
- Hormone replacement for women (previously thought to reduce cardiovascular risk).
- Vitamin add-ons (previously thought to reduce cardiovascular risk).
- Adult neurogenesis (the idea that adult brains create new neurons).
These original studies were in Q1 journals with good statistics but failed upon replication.
Murphy NL ( $2025$ ) Study:
- Evaluated data from $25$ studies in exercise science and medicine.
- Only $7$ out of $25$ could be replicated (a replication rate of $28\%$ ).
Recommendation for improvement:
- Educate and reform rather than "throwing rocks" at pioneers who used accepted practices of their time (e.g., Licht).
- Convince editors and reviewers that exploratory research and replication are as valuable as hypothesis testing.

Definitions of responders and non-responders in training may be flawed.
Evidence, particularly regarding strength and power, suggests true non-responders are rare or non-existent.
Multiverse Analysis/Analytical Variability:
- When the same data is sent to multiple teams of statisticians, they often come up with different or even opposite answers.
- Small analytical choices can reverse conclusions.
- A study mentioned by the speaker used three methods (paired differences, a two-by-two ANOVA, and linear mixed models) and got the same answer, but the effect size decreased as complexity increased.

Start with the simplest family of comparisons; only add complexity if it improves understanding.
Complex models are often unstable in sports science due to small $n$ .
Risks of complex models: Overfitting, correlated predictors, and estimated effects the data cannot support.
Recommendations:
- Label studies clearly as exploratory designs.
- Acknowledge limitations.
- Cultivate strong, prior mechanistic knowledge to aid interpretation.
- Utilize Bayesian reasoning where appropriate.
- Emphasize effect sizes.
- Apply appropriate adjustments to the "photon test" (as identified in the transcript).
- Repeat. Repeat. Repeat. One study should not be taken as absolute truth.