Ch.3 Research Methods and Study Design

Experimental Design

a “design” for an “experiment” that is used to rule out other possible explanations and determine causation, rather than correlation.

First step in experimental design: selecting the population

-selecting the population is the first step in experimental design

-low cost

-easy access

-not “all humans”

-the results of the experiment only apply to the population in the study, other populations are outside the scope of the experiment.

second step in experimental design: choose the independent and dependent variable

-independent variable: variable manipulated by researchers

-dependent variable: variable measured by researchers

the researchers manipulate the independent variable, and because the dependent variable depends on the independent variable, they measure the dependent variable.

-reproducibility: good experimental design requires experiments that can be reproduced by other researchers “reproducibility”

-operational definition: a specification of precisely what researchers mean by each variable. “how each variable operates”

An operational definition translates an abstract theoretical concept (like "intelligence" or "stress") into concrete, measurable variables. It specifies exactly how a researcher will observe, measure, or manipulate the concept within their specific study, ensuring the research remains objective and easily replicated.

Dependent variable more information

-the dependent variable measured must be equally well defined, and must be quantitative or numerical, not qualitative, because the dependent variable is used to conduct statistical analysis that will test the research hypothesis.

-if the dependent variable indirectly measures, then the dependent variable data indirectly suggests.

third step in experimental design: carefully select control and experimental groups

control group: group that acts as a point of reference and comparison

experimental group: group of participants that receive treatment

homogenous control group and confounding variables

-needed in studies

-”homogenous”: control groups that are the same throughout , as similar as possible to the experimental group, except for the variable of interest

-goal of homogenous control groups is to rule out extraneous (confounding) variables.

-confounding variables: variables on the outside that confuses or invalidates an observation

-to optimize study design: experimental and control groups must be as similar as possible to limit confounding variables.

Placebo effect

-patients symptoms improve after receiving fake treatment, just believing the treatment is being administered leads to a measurable result.

-the placebo effect is counted by “double blind”

-double blind: neither the person administering the treatment nor the participants “double” truly know “blind” if they are assigned to the treatment or control groups.

fourth step in experimental design: randomly sample a population

-sampling should be random

-it should be equally likely for any member of the population to be a participant.

population flaw: most common flaw in social research is most studies are done on on undergraduate students and then results are applied to the population.

-Sampling bias: some members of a target population have a lower or higher chance of being selected than others.

-Selection bias: a more general category in systemic flaws in a design that can compromise results.

-Meta-analysis: big-picture “meta” analysis of many studies look for trends in data.

-attrition: latin “attrio”: “a rubbing against”, attrition is when participants drop out systematically differ from those who remain. Attrition is another type of selection bias. attrition effects: participant fatigue; participants drops out of study.

fifth step in experimental design: randomly assign individuals to control groups

-after individuals have been incentivized, it’s time to randomly assign them to the experimental and control groups

-well designed experiments have it equally likely that people are assigned to either group.

-randomized block technique: used to ensure study groups have equal sample sizes at regular intervals. Participants are divided into small, ordered, blocks “blocks” randomly assigned “randomized” so that each treatment is equally represented within every block.

sixth step in experimental analysis: measure the results

-you measure the results of the experiment by measuring the dependent variable (because the researchers manipulate the independent variable and measure the dependent variable)

-instruments should be reliable. Reliability means that the instruments produce stable and consistent “reliable” results, measure what they are supposed to "construct validility”, and that repeated measurements should lead to similar results replicability.

-replicability: repeated measurements should lead to similar results.

-psychometrics: the study of how to measure psychological variables through testing.

-Another concern with surveys is response bias, the tendency for respondents to not have perfect insight into their state and provide inaccurate responses.

seventh step in experimental analysis: test the hypothesis

-type I error: false positive

-type II error: false negative

-null hypothesis: no effect, no relationship, no difference between variables “null”

-researchers see whether evidence from the experiment suggests that the null hypothesis is true or false.

-experimental hypothesis: the hypothesis that suggests that variations in the independent variable causes changes in the dependent variable.

-what needs to happen to reject the null hypothesis, it is not satisfactory to simply observe a difference between two groups.

-a significant difference is a measured difference between two groups that is large enough that is is probably not due to chance.

-scientists evaluating data from experiments have the problem that they can never be certain that a difference measured in an experiment actually reflects a fundamental difference between the groups. Scientists must arbitrarily pick a cutoff point at which it is reasonable to conclude “beyond reasonable doubt” that there was a difference. Conventionally, social scientists have decided that if the probability of an observed difference is found to be 5% (or 0.05) or less, this constitutes a significant difference. A p-value “p for probability” is a number from 0 to 1 that represents the “probability” that a difference observed in an experiment is due to chance.

If (and only if) the p<0.05, scientists reject the null hypothesis (because it’s less than 5% probability that chance is observed). Therefore, a lower p-value means a stronger relationship.

Other p-values such as 0.01 or 0.001 are also used as the threshold for significance in some cases.

-to make sure the experiment picks up an effect, it’s necessary to have a large enough sample size (number of participants). 30 or more participants are necessary to meet the mathematical criteria needed to conduct statistical tests. A larger sample size is usually preferred, because a large sample size increases the power of the experiment, or the ability to pick up an effect if one is actually present.

Summary of a good experimental design (page 44)

1. select the population of interest

2. determine the independent and dependent variables

3. Carefully select experimental and control groups, homogenize the groups

4. Random sampling from the population of interest: make sure all members of the population are equally represented AND each member has an equal chance of being selected. Meeting these criteria is often not possible for practical reasons.

5. Random assignment of individuals to groups: individuals who have been sampled are equally likely to be assigned to treatment or control

6. Measure the results: use the quantitative dependent variable, instruments are reliable and construct validility, repeated measurements lead to similar results (replicability, able to be “replicated”)

7. Test the hypothesis: type I error, type II error, null hypothesis, experimental hypothesis, p-value.

Validity

-external validility

-internal validity

-demand characteristics

-predictive validity

-some potential flaws are able to come up in experiments

-a flaw or limitation might make it difficult to apply our conclusion to the real world. This is known as a flaw in external validility. For example, the fact that only students of Healthy Living U can participate in the study is a threat to external validity. We cannot be absolutely sure that a result that rejects the null hypothesis applies to all healthy young adults in their twenties.
-External validity is the extent to which the results of a scientific study can be generalized to other situations, populations, and time periods.

-on the other hand, a limitation to this study might be such that the experiment is not “well done”, leaving doubts about the conclusion because of some inherent flaw in the design. This is known as internal validility.

Internal validity refers to how confidently you can establish that a cause-and-effect relationship observed in a study is real and not caused by other variables

Internal validity is high if confounding variables have been considered a minimized, and the casual relationship between independent and dependent variables can be established by the way the experiment was set up. If the researchers forgot to control for gender, used a diet that was not actually Mediterranean, or gave the control group unhealthy foods, for example, internal validity could be threatened.

-demand characteristics is the tendency of participants to consciously or subconsciously act in ways “characteristics” that match how they are expected “demanded” to behave, can also threaten internal validity.

-another consideration, especially when psychometric evaluations are used, is predictive validity. Predictive validity is the extent to which a score on an assessment or tool accurately forecasts a future behavior, performance, or outcome (the MCAT is an example of predictive validity).

Frequently tested common threats to validity in social science experiments:

Impression management: participants adapt their responses based on social norms or perceived researcher expectations; self fulfilling prophecy; methodology is not double-blind, Hawthorne Effect.

Hawthorne Effect: a psychological phenomenon where people alter their behavior, usually improving their performance or productivity, simply because they know they are being watched

Confounding variables: extraneous variables not accounted for in the study, another variable offers an alternative explanation for results, lack of a useful control

lack of reliability: measurement tools do not measure what they purport to, lack consistency

sampling bias: selection criteria is not random, Population used for sample does not meet conditions for statistical test (e.g. population is not normally distributed).

attrition effects: participant fatigue; participants drops out of study

demand characteristics: participants interpret what the experiment is about and subconsciously respond in ways that are consistent with the hypothesis.

experiment doesnt reflect real world: laboratory setups don’t translate to real world, lack of generalizibility.

selection criteria: too restrictive of inclusion/exclusion for participants (i.e., sample is not representative).

situational effects: presence of laboratory conditions changes outcomes (e.g., pre-test and post-test, presence of experimenter, claustrophobia in an MRI machine).

lack of statistical power: sample groups have high variability: sample size is too small.

Hawthorne Effect

The Hawthorne effect is a psychological phenomenon where people alter their behavior, usually improving their performance or productivity, simply because they know they are being watched.

background

The term originates from a series of studies conducted between 1924 and 1932 at the Western Electric company's Hawthorne Works factory in Cicero, Illinois. Researchers initially wanted to see if altering environmental factors, such as lighting levels and break times, would impact worker productivity.

Surprisingly, researchers found that worker productivity increased not just when lighting was improved, but even when lighting was made worse. They eventually concluded that the workers' boosted output was a response to the attention they received from the researchers and the novelty of being part of an experiment.

Ethical considerations

-All studies should consider ethical implications of the procedures they employ.

-Ethical problems tend to arise more frequently in experimental designs because researchers are directly manipulating variables, not just observing what they see in nature.

-Ethical problems arose in many social science experiments of the early 20th century (1901-2000) as researchers, institutions, and society gradually came to an agreement to the correct protocol for running experiments.

-some studies became infamous for their egregious breach in the correct protocol running experiments (Tuskegee Syphilis Experiment). African American males in Alabama who had contracted Syphilis were not told they had the disease after they tested positive, were not treated for the disease, even though treatments were available. The participants were recruited under false pretexts and treated with inhumane disregard for their health and well-being.

-any procedures that could lead to detrimental health consequences must be ruled out.

-If researchers become aware that patients have a condition or disease, they must be immediately notified and given treatment options.

-Over time, ethical standards and potentially traumatic experimental procedures must include protocols for dealing with the harm they might cause, such as counseling or other clinical treatment.

Ethical considerations (ensuring modern experiments meet ethical standards)

-To be sure ethical standards are met, modern experiments must be cleared by an independent internal commission.

-Modern experiments also need to contain some type of disclosure: an outline given to pariticipants before the experiment begins that clarifies incentives and expectations while reminding them or their right to terminate the experiment at any time.

-Modern experiments must also have debriefing: participants are told after the experiment exactly what was done and why the experiment was conducted. In cases where the experiment particularly triggered psychological vulnerability, participants may be offered access to treatment or counseling services as part of debriefing.

Non-experimental designs

When experiments are not feasible for practical or ethical reasons, researchers in social sciences have many other types of designs at their disposal. Each design offers its own benefits and potential drawbacks. In general, non-experimental designs tend to offer the benefit of observing phenomena in a more naturalistic setting, often improving external validility.

The trade off with non-experimental designs is reduced control of variables of interest, therefore reducing internal validity.

Non-experimental designs: Correlational studies.

-Correlational studies explore the relationship “correlation” between two quantitative (numerical) variables.

Pearson Correlation

-Pearson Correlation is the most common type of correlation.

Pearson Correlation assigns a number from -1 to +1 to a pair of variables.

If the value is negative, the two variables are negatively correlated. This means that if one variable increases the other variable decreases.

If the value is positive, the two variables are positively correlated. This means if one variable increases, the other variable also increases. If one variable decreases, the other will also decrease.

Note the in the title, it says “correlation”, which implies correlation. This does not imply that one variable causes another.

A value of zero indicates no correlation, that there is no linear relationship between the two variables, although a nonlinear relationship is still plausible.

Significance testing can be combined with Pearson correlations to see if the computed correlation is likely to have occurred by chance or not.

Non-experimental designs: Ethnographic studies

-Ethnographic studies are a qualitative method in which researchers immerse themselves completely in the lives, culture, or way of life of the people they are studying.

Ethnography translates literally to "writing about a people". The term originates from the Greek words ethnos (meaning "folk," "nation," or "people") and grapho (meaning "to write" or "to describe").

-Ethnographic studies are lengthy and involve as little interference and intervention by the researchers as possible.

Example: a researcher interested in the effect of the Miditerranean diet on health might go to a remote Sicilian village and study the lives of the participants, scrutinizing their everyday lives and recording everything they possibly can over the span of several years.

Non-experimental designs: Twin Studies

Twin studies compare identical (monozygotic) and fraternal (dizygotic) twins to untangle the "nature vs. nurture" debate.

-Twin studies are the best way to measure the heritability, the extend to which an observed trait is due to genetics (nature) vs environment (nurture).

Example: twin studies interested in the heritability of a trait (intelligence is the trait in this case), might look at correlations in IQ scores between monozygotic (identical) and dizyogtic (fraternal).

-it is reasonable to conclude that any differences between these two correlations are due to genetics, because both types of twins share the same environment.

Non-experimental designs: Longitudinal Studies

Researchers may be interested in how individuals develop overtime (longitudinally) along some research variable.

Longitudinal method: involves intervallic measurements of a dependent variable over long time frames.

-Longitudinal studies are costly, difficult to execute, time-and-resource intensive, and have high attrition rates.

-The benefit of longitudinal studies is high accuracy when observing for change because of the ability to detail how an effect or factor can develop over time.

-cross-sectional study: data collection of a population or sample at a specific time. Cross sectional studies are related to but are slightly different than longitudinal studies.

Non-experimental designs: Case Studies

“case studies” involves in-depth exploration of one individual or “case”.

Non-experimental designs: Phenomenological Studies

Phenomenological studies are interested in describing phenomena, using the introspective method to explore research questions.

Introspection is the process of examining your own internal thoughts, feelings, and motives

-Phenomenological Studies: involves researchers studying themselves

-Hermann Ebbinghaus made many groundbreaking discoveries in learning and forgetting by taking detailed data and notes on his own learning and memory performance. These investigations were “phenomenological”, they attempted to understand his own perceptions and understandings, rather than make a comparison between variables and draw a casual conclusion.

-not all phenomological studies are confined to self-observation.

-a phenomenological study attempts to understand people’s perceptions, perspectives and understandings of a phenomenon.

A phenomenon is an observable fact, event, or circumstance. It often refers to something that is out of the ordinary, impressive, or requires scientific explanation (e.g., a natural occurrence or a sudden cultural trend).

Non-experimental designs: Survey

Survey: method for collecting information or data as reported by individuals.

-participants answer a series of questions and self-report the information.

-surveys are used to get an idea of how a group or population feels about a number of things, such as political debates, new businesses, classes, and religious views.

-Additionally, surveys can be a way for people to measure how often or little people engage in different behaviors, such as smoking or drinking alcohol.

Non-experimental designs: Other studies

These are a few other studies that are worth mentioning since they may appear on the MCAT.

-Archival studies: analyze collected data from historical records and authentic original documents.

-Biographical studies: exhaustive accounts of an individual’s life experience.

-Finally, many of the study types we’ve looked at fall under the category of observational studies. An observational study is any study in which individuals are observed and outcomes measured with no attempt to control the outcome.