Stat slides chapter 1 part 3
Introduction to Experimental Design
What you'll learn in this section:
What a census is.
About simulations, observational studies, and experiments.
How to identify control groups, placebo effects, and different types of experiments (completely randomized, randomized block).
Common problems that can make data unreliable.
Census
A census collects information from everyone in a population. Example: The U.S. Census Bureau attempts to count every person living in the United States every ten years.
Sample
A sample collects information from only a part of a population. Example: Surveying 1,000 registered voters to predict the outcome of an election.
Observational Studies vs. Experiments
Observational study: You observe and collect data without influencing the subjects or conditions. You simply watch what's happening without intervention. Example: Watching how long students study for an exam and correlating it with their grades, without telling them how long to study.
Experiment: You actively impose a treatment on individuals to observe a response. You do something specific to individuals to see if it causes a change in what you're measuring. Example: Testing a new fertilizer by applying it to one group of plants and not another, then measuring plant growth.
Example 4: Hawaii Goats (An Experiment on Goats and Silver Sword Plants, 1778-related)
Background: Captain James Cook brought goats to Hawaii. The goat population grew dramatically, leading to a noticeable decrease in the number of native silver sword plants on Maui.
Problem: Biologists suspected that the goats were causing the decline of the silver sword plants and decided to conduct a study.
(a) How the experiment was designed:
They established several stations in remote areas.
At each station, two plots of land were selected, similar in soil composition, climate, and initial plant count.
One plot was fenced to exclude goats (this was the 'treatment' applied).
The other plot was left unfenced (the control).
Biologists regularly counted silver sword plants in both plots to track changes.
This qualifies as an experiment because the researchers changed something (added a fence) to one plot to observe its effect on plant growth, allowing for a causal inference.
(b) Control and comparison: The unfenced plot served as the control group. It was identical to the fenced plot in all aspects except for the fence (the treatment), which allowed direct comparison to determine the fence's effectiveness in protecting the plants.
Placebo Effect
Definition: The placebo effect occurs when a person experiences a perceived improvement or favorable response to a fake treatment (a placebo) simply because they believe they are receiving real treatment. Example: A patient with a headache feels better after taking a sugar pill, thinking it's a powerful painkiller.
Completely Randomized Experiment
Definition: In this type of experiment, all participants are assigned to different treatment groups entirely by chance (randomly).
Example: To test a new drug, 100 volunteers are randomly split into two groups: one receives the active drug, and the other receives a placebo. Each volunteer has an equal chance of being in either group.
Block and Randomized Block Experiment
Block: A group of experimental units (individuals or things) that are similar in some characteristic that might affect the outcome of the experiment. Example: In a study testing a new teaching method, students might be blocked by their current academic performance (e.g., high-achievers, average, low-achievers).
Randomized block experiment: Participants are first divided into blocks based on a shared characteristic (like age group, gender, or pre-existing conditions). Then, within each block, individuals are randomly assigned to different treatment groups. This helps ensure that the treatment effect isn't skewed by these blocking factors. Example: In a skin cream study, women and men are grouped into separate blocks. Within the 'women' block, half are randomly assigned the new cream and half the old; the same random assignment happens independently within the 'men' block.
Guided Exercise 5: Data-Collection Techniques (Solutions)
(a) Which technique is best for studying stopping a nuclear reactor's cooling process?
Solution: Simulation. Using a computer model allows for testing dangerous scenarios without real-world risk.
(b) Which technique is best for studying the time spent exercising by full-time college students?
Solution: Sampling and observational study. Surveying a random group of students allows data collection without influencing their exercise habits.
(c) Which technique is best for studying the effect of calcium on bone mass in young women?
Solution: Experimentation. This involves randomly assigning some women to a calcium supplement and others to a fake pill (placebo) to isolate the calcium's effect. An example from a Tom Lloyd JAMA study with 94 participants showed the calcium group gained about 1.3 ext{ extbf{%}} more bone mass per year compared to the placebo group.
(d) Which technique is best for determining the credit hour load of students after the drop/add period?
Solution: Census. Collecting records for every single student from the registrar's office provides complete and accurate data on all students' credit loads.
Some Potential Problems with Surveys (Part 1)
Nonresponse: Occurs when people can't be reached or refuse to answer. This can lead to certain demographic groups being underrepresented in the results, biasing the survey.
Truthfulness of response: Respondents might not tell the truth, especially on sensitive topics, or may not accurately remember events.
Faulty recall: People might not remember events or their specific timings exactly as they occurred, leading to inaccurate data.
Hidden bias: How questions are phrased, the order they're asked, or the range of answer choices can subtly push people towards certain answers. Example: A question like "Do you agree that our excellent public schools deserve more funding?" implies a desired answer, introducing bias.
Some Potential Problems with Surveys (Part 2)
Vague wording: Words like “often,” “seldom,” or “occasionally” mean different things to different people, leading to inconsistent responses. Example: "How often do you exercise?" is less precise than "How many hours per week do you engage in vigorous exercise?"
Interviewer influence: The interviewer's tone, body language, clothes, gender, perceived authority, or background can affect how people answer. Example: A researcher in a white lab coat asking about health habits might get different answers than a peer asking the same questions.
Voluntary response (or self-selection bias): Only people with strong opinions are more likely to respond, meaning the survey results might not accurately represent the general population. Example: Online polls where participants self-select often show extremely skewed results because only those deeply invested bother to respond.
Lurking Variables and Confounding Variables
Lurking variable: An unmeasured variable not included in the study but still affects the relationship between the measured variables. It 'hides in the background' and can influence the results.
Confounding: Occurs when the effects of two or more variables are mixed together, making it impossible to determine which variable is truly causing the observed effect. You can't tell if the effect is from one factor, another, or both.
Relationship: Confounding variables can be part of the actual study, or they can be external 'lurking' variables that were not accounted for.
Guided Exercise 6: How Useful is the Data-Collection Plan?
(a) Scenario: A uniformed police officer interviews 20 college freshmen about illegal drug use in the last month.
Potential issues: People might refuse to answer or lie, especially about sensitive or illegal topics, to avoid negative judgment (social desirability bias). The presence of an authority figure significantly influences responses.
(b) Scenario: Frankie's data shows cities with more low-income housing seem to have more homelessness.
Caution: Be careful! This correlation does not imply causation. Other factors (like overall city size, economic conditions, local job markets, or cost of living) might be causing this link, not just low-income housing itself. These are possible confounding or lurking variables that could explain the observed relationship.
(c) Scenario: A cafeteria survey is conducted using paper forms placed at the cash register with a drop box for submissions.
Potential issues: This method will likely only gather responses from people with very strong opinions (either very happy or very unhappy with the cafeteria service). People with neutral feelings are less likely to take the time to fill out the form. Therefore, the results will not represent the general population (voluntary response bias or self-selection bias).
(d) Scenario: Coronary studies are conducted only with male participants over 50.
Potential issues: The results from this study might only apply specifically to older men and not generalize to women or younger individuals. It's difficult and inappropriate to assume the same findings would apply to different sexes or age groups, limiting the study's external validity.
Important Features of a Data Collection Plan
A good data collection plan outlines exactly how data will be gathered to answer a research question effectively. It should clearly specify:
Population and Variables: Who or what is being studied (the population of interest), and what specific characteristics or measurements (the variable(s) of interest) are to be observed or measured.
Study Type: Whether the approach is an observational study (observing without intervention) or an experiment (intentionally imposing a treatment to see its effect).
Control Mechanisms: If applicable, details on using control groups, fake treatments (placebos), and 'blinding' (where participants, and sometimes researchers, are unaware of treatment assignments) to minimize bias and isolate treatment effects.
Sampling and Grouping: How the sample will be selected from the population, and whether 'blocks' (groups of similar individuals) will be used to manage variability and improve the study's precision.
Data Collection Method: The precise way data will be obtained (e.g., through surveys, direct measurements, counting observations, interviews, etc.).
Viewpoint Discussion Activity (Part 1 of 4): The Placebo Effect
The placebo effect is a documented phenomenon, yet it remains complex. Early studies suggested that people felt about 35 ext{ extbf{%}} better just by believing they received active treatment.
Some researchers propose that observed improvements attributed to placebos are sometimes due to 'regression to the mean'—meaning patients experiencing particularly bad symptoms often get better naturally over time, regardless of intervention.
Intriguing findings indicate that expensive fake treatments (placebos) can sometimes elicit a stronger response than cheaper ones (e.g., a branded placebo might work better than a generic-looking one), highlighting the psychological component.
Your mind plays a huge role in how you respond to treatment; this psychological influence significantly impacts outcomes, particularly for subjective measures like pain perception.
Viewpoint Discussion Activity (Part 2 of 4): Blinded Placebo-Control Groups
The use of 'blinded' studies with fake treatments (placebo-control groups) is crucial in clinical trials for several reasons:
They help researchers distinguish between the actual effect of a treatment and the psychological or spontaneous improvement known as the placebo effect.
They reduce bias from both participants (who don't know if they're receiving the real treatment) and researchers (in 'double-blind' studies, where researchers also don't know treatment assignments) when judging results.
Ultimately, they make the trial results more trustworthy, scientifically sound, and easier to accurately interpret.
Viewpoint Discussion Activity (Part 3 of 4): Ethics of Placebos
A significant question arises: Is it ethically acceptable for doctors to prescribe fake pills for long-term conditions like pain?
A survey of rheumatologists revealed that over half had prescribed a placebo at some point, and more than 80 ext{ extbf{%}} of them felt it was an acceptable practice in certain situations.
What about companies selling 'miracle cure' products (such as the Q-Ray Ionized Bracelet, which was legally found to be fraudulent) that seem to work only because of the placebo effect?
Viewpoint Discussion Activity (Part 4 of 4): Other Ethical Considerations
The use of kinesiology tape in sports is a prime example: It claims to reduce pain and aid recovery, but scientific consensus often suggests its observable effects are primarily due to the placebo effect.
This raises several ethical questions:
Does the cost of a fake treatment (placebo) matter ethically?
Should misleading claims for these products be regulated with the same strictness as real drugs or medical devices?
Summary
This chapter, 'Getting Started,' reviewed fundamental concepts in experimental design and data collection.
Key ideas covered include: census, sample, observational study, experiment, placebo effect, completely randomized experiment, randomized block designs, and the challenges posed by lurking and confounding variables.
We also explored how to plan data collection effectively, identified common problems encountered in surveys, and discussed important ethical considerations when conducting studies.
Real-world examples and guided exercises were used to illustrate how to select the appropriate data collection method and critically evaluate potential weaknesses in various studies.