Chapter two: Seven Critical Components of Statistical Studies

Funding sources shape interpretation; common sources include governments, private companies, and universities.
If funding comes from groups with a vested interest, scrutinize procedures in the other components to ensure sound science.
Be aware that private sponsors can influence study design or reporting if not properly controlled.

Who interacted with participants can bias responses (interviewer effects, expectations).
Blind or double-blind procedures help reduce bias (e.g., inspectors/handlers unaware of group assignment).
Examples: biased lab interactions or non-blind taste tests can distort results.

Generalizability depends on the study population; often not representative of the broader group.
Volunteer/self-selected samples bias results toward those with strong opinions or interest.
Sampling issues include who was available, who attended, and how they were recruited.

Precise definitions matter (e.g., what counts as “eat breakfast”).
Measurement tools and question wording can bias answers; order effects can matter.
Ideally, the exact survey questions should be available for evaluation.

Location, timing, and mode of data collection affect results (phone, online, in-person, lab).
Timing effects: public events, holidays, or seasonal patterns can shift responses.
Lab vs real-world settings can limit external validity.

Extraneous differences can confound the observed effect (e.g., weight, motivation, background).
Researchers should discuss and, if possible, control for these factors.
Uncontrolled confounding can misattribute effects to the factor of interest.

Reports often omit the size of effects; practical significance matters.
Report actual rates, differences, or effect sizes (e.g., how much one group differs from another).
Example framing: a reduction from $17$ to $9.4$ per $1000$ participants indicates practical impact beyond a binary "effect present" claim.

Use the seven components as a checklist to identify missing or biased information.
If any component is weak or missing, treat conclusions with skepticism until clarified.
Compare the news story to the original source when possible.

Component 1: Funded by a senior-thesis project; minimal funding but potential bias from researcher motives.
Component 2: Who interacted with participants unclear; possible bias if participants knew the purpose.
Component 3: Sample limited to students present on the day, not necessarily representative; missing non-attendees biases results.
Component 4: GPA self-reported; could be inaccurate; GPA not a direct measure of intelligence.
Component 5: Timing around a holiday may affect participation; anonymous responses reduce accountability.
Component 6: Other differences between groups (e.g., course rigor, selection bias) could explain GPA differences.
Component 7: Reported GPAs (e.g., 3.05 vs 2.91) need sample size and variability to judge significance.

Component 1: Funding source not stated; independent group without transparency raises concern.
Component 3: Countries chosen and rationale not provided; selection bias possible.
Component 4: Measurements flawed; current exchange rates used with no inflation adjustments; comparability missing.
Component 5–7: No meaningful timing or size context given; inflation adjustment needed for valid interpretation.

Component 1: Funding not clearly disclosed; potential industry influence if funded by a drug company.
Component 2: Dog-handlers and their interactions not described; could bias outcomes.
Component 3: Dogs were volunteers from local area; generalizability limited.
Component 4: Group-level measurements (time barking) may be biased by group dynamics; individual measures would be better.
Component 5: Measured on weekends in a facility, not in dogs’ natural environment; external validity limited.
Component 6: Randomization helps but differences in handling and weekend effects could confound.
Component 7: Reported “barked half as much” lacks absolute time without context; need actual times.

Component 7: Major flaw: volunteer/self-selected sample; 5% response rate; nonresponse bias severe.
Component 1–3: Sample not representative of population; survey framed to provoke emotional response.
Takeaway: Volunteer samples often misrepresent broader opinions.

Source and funding: NIH-supported study by a university team; credible backing.
Contact with participants: enrolled in psychology courses; likely researcher involvement could influence responses.
Sample: 1,230 college students in intro courses; representativeness of all college drinkers questionable.
Measurements: 13 symptoms on a 5-point scale plus a hangover count; more detail in the article.
Setting: university context; questions may be administered in group settings or online; anonymity impacts honesty.
Group differences: gender comparison (Male/Female) and family history; weight and other factors could influence outcomes.
Size of effect: simple comparisons insufficient; results improved when accounting for frequency/quantity of drinking.

Component 1: Purpose and scope of the study; decide items to measure and stores to compare.
Component 2: Who records prices; training and consistency in recording.
Component 3: Items selected; representative vs. exhaustively many items; sampling decisions.
Component 4: How to measure price (sale vs regular, smallest vs largest size, brand choices).
Component 5: Timing of data collection; consider seasonal or promo-driven price changes.
Component 6: Differences between item groups or stores beyond price (time, location, convenience).
Component 7: Reported difference size; translate into annual savings or practical impact.

Brooks vs Suave: design bias, non-representative sample, track-meet setting, leading questions.
Amstar vs Domino’s Pizza: both surveys flawed; sample choice biased (home-based vs outlet-based), city coverage not representative.
Lesson: Surveys used in legal cases must have representative sampling and unbiased questions to be persuasive.