Sampling/Internal and External Validity

Goal of most clinical research

Develop theories that apply to all human behavior and the environment

population

larger group to which research results are generalized

sample

a subgroup of the population used for estimating characteristics of that population

efficacy trials

high control, limited bias; very strict inclusion/exclusion criteria

efficacy trial limitations

limits the generalizability and clinical relevance of results

effectiveness trials

limited control, greater bias; less strict inclusion/exclusion criteria

effectiveness trial limitations

increase risk for bias; less confidence in cause/effect conclusions

two types of sampling techniques

probability and non-probability

probability sampling

randomization involved at some point in the process

nonprobability sampling

randomization not involved at any point; suspect that the sample is biased in some way; far more common in clinical research

types of probability sampling

simple random sampling, systematic sampling, stratified sampling, cluster sampling

simple random sampling

Each member of the population has an equal chance of being selected

limitation of simple random sampling

It requires the entire population to be known; difficult for studies with broad inclusion/exclusion criteria

systematic sampling

participants chosen from a list (every Kth name)

limitations of systematic sampling

need a list of the population

stratified sampling

random sampling from subgroups

limitation of stratified sampling

possible to randomly over/undersample on some key variable

cluster sampling

divide population into clusters and randomly sample clusters

types of nonprobability sampling

convenience sampling, quota sampling,

convenience sampling

subjects chosen based on availability

limitations to convenience sampling

subjects typically from one clinical site, volunteers tend to have greater motivation

quota sampling

nonprobability equivalent of stratified sampling; subgroups of subjects that vary on some characteristic are recruited/enrolled until appropriate sample size is reached

the more subjects you analyze, the ___ confident you can be that your sample represents the population characteristics you wish to measure

more

internal validity

extent to which we can draw cause-and-effect inferences from a study

requirements for internal validity

temporal precedence, covariation of cause and effect, no plausible alternative explanations

temporal precedence

change in outcome must occur AFTER a change in the treatment

covariation of cause and effect

outcome only occurs in the presence of the intervention OR greater intervention, greater outcome

No plausible alternative explanations meaning

you can rule out competing reasons for the observed change in the outcome of interest besides the intervention

threats to internal validity

history, maturation, testing, instrumentation, mortality/attrition, regression to the mean

social threats to internal validity

interaction of subject with investigator, performance bias, attention bias

types of performance bias

contamination and cointervention

history threats to internal validity

events unrelated to the study that may influence the dependent variable

maturation threats to internal validity

changes occur in the outcome of interest as a result of the passing of time

testing threats to internal validity

repeated test taking may improve test scores

instrumentation threats to internal validity

a change in equipment that affects the outcome of interest

mortality/attrition threats to internal validity

subjects are "dying" with respect to your study (dropping out)

regression to the mean threats to internal validity

statistical phenomenon that affects studies where groups are selected based on extreme scores to a single test (tutoring students with low test scores)

interaction of subject with investigator is mitigated by

masking/blinding of the subjects and investigator

contamination (performance bias)

members of the control group receive the intervention; ex. member of the control group learns about the elements of the experimental condition and applies condition to self

cointervention (performance bias)

subjects seek/receive another form of treatment that influences the DV during the experiment

attention bias

peoples behavior and performance improve following any new or increased attention (Hawthorne effect); OR peoples behavior and performance decline if they perceive they are not receiving equal attention/care (Reverse Hawthorne effect)

recall bias

error associated with self-report tools (surveys) asking subjects to recall past events; especially relevant if participants know the hypothesis being tested

external validity

can you generalize your results to other patients, settings, or time?

threats to external validity

people, place, time

people threats to external validity

selection process yields sample that does not represent the population the researchers hope to apply the results (ex. using volunteers with increased enthusiasm and compliance-not going to see that in a clinical setting)

place threats to external validity

outcomes affected by the setting in which the data were collected (ex. data collected in a lab with crazy equipment)

time threats to external validity

results of an experiment specific to the era the data were collected (can "old" results be applied to today?)