Numbers, Sampling, & Interpretation

Even accurate numbers can lead to inaccurate conclusions if they're misinterpreted, poorly sampled, or stripped of context.

Right-censoring

Wgen people haven’t lived long enough for outcomes to show (like musician age studies)

• The study ends before the outcome of interest has occurred for all participants

• Like the musician age studies where the artist is still alive so their total lifespan & career longevity are unknown at the time of the analysis

Sampling bias

Non-representative samples

Self-selection bias & attrition bias

Self-selection bias

When individuals choose whether to participate in a study, often attracting those with strong opintions or high motivation

Attrition bias

The systematic dropout of participants form a study

• If the people leaving differ in meaninful ways from those who remain, the results become biased

Overfitting

When a model fits one data set too perfectly but can’t generalize

• When a statistical model or machine learning algorithm fits the training too closely, capturing noise instead of the underlying pattern

• Fails to generalize new unseen data when the model performs exeptionally well

What are two logical missteps?

Prosecuter’s fallacy

Will Rogers effect

Prosecutor’s fallacy

Low probability of an event happens by chance is misinterpreted as a high probability that the defendant is guilty

• DNA match error: chance of a match is 1 in a million & the prosecutor argues that there is only a 1 in a million chance they are innocent

• This is false because in a city of 4 million people, there could be four inncoent people with that match

Will Rogers effect

When moving an item from one group to another raises the average for both groups despite not changing the overall population average

• Usually caused by shifting items that are below the average of one group but above average of the other

• A team moves a player who is bad for team A (but good for team B) from team A to team B

  • Both teams see an increase in their average performance simply with restructuing rather than hiring new talent

What is right-censoring, and how can it distort results?

• Right-censoring is where people have not lived long enough for outcomes to show.

• This can distort the results by underestimating time by ignoring cases or treating them as survivors.

• Additionally, if a patient drops out due to illness or because they are cured in a non-random way, the data are still important and can cause an over- or underestimation.

• Finally, a large number of patients censored at the same time can create artificial gaps.

How do different sampling methods affect generalizability?

• Different sampling methods can affect generalizability by impacting how well the results apply to a larger population.

• Random selection means that every member has an equal chance of being chosen, which can enhance the generalizability and create representative samples.

• Convenience sample where participants are chosen base don their ease of access is cheaper but has a higher risk of sampling bias.

• These studies are less generalizable due to the broader population.

• Finally, small sample sizes reduce the statistical power to detect true effects, also limiting the representation of population diversity.

What is the Prosecutor’s Fallacy, and why is it misleading?

• The prosecutor’s fallacy is a logical error where a low probability of an event happening by chance is misinterpreted as a higher probability that the defendant is guilty.

• This confuses the probability of evidence given innocence with the probability of innocence given the evidence.

• For example, if a DNA match is 1 in 1,000,000, the prosecutor may argue that there is only a 1 in 1,000, 000 chance that the defendant is innocent.

• However, in a city of 4,000,000 people, there could be four innocent people with that same match.