2: Research Validity

scientific method/research process

1. statement of the problem

2. design of research study

3. measurement of variables

4. analysis of data

5. conclusions from research

research design

naturalness of the research setting

• lab or field

• “naturalness” vs. “artificiality” of the setting

• field research typically employs a real-life setting, whereas lab research is more contrived and artificial

degree of control

• control over confounding and extraneous variables

• manipulation: reflective of high degree of control

• research designs that permit manipulation are technically referred to as “experiments”

validity

the key criterion in the evaluation of any piece of research or test(measure)

• the appropriateness of inferences drawn from data

  • data = results of research study → research validity

  • data = test scores → test and measurement validity

low control, natural setting

• quasi-experiment

• correlational design

• survey (questionnaire)

• observation (non-expt)

inferential claims

1. frequency/amount

   1. there are 1400 psychology majors

2. association/relationship

   1. children who play video games have shorter attention spans

3. causal

   1. if football players wear gloves, they will catch more footballs

research validity

a conclusion based on research is valid when it corresponds to the actual or true state of the world

• internal

• external

• statistical-conclusion

• construct

internal research validity

extent to which causal inferences or inferences about observed relationships are sound (relationship is real/not artifactual)

external research validity

extent to which results can be generalized to and across alternate measures, participants/populations, settings, times

statistical-conclusion research validity

appropriateness of inferences made from data as a function of conclusions drawn from statistical analyses

construct research validity

extent to which study/operationalizations align with theory/conceptualizations (manipulation for experiments)

threats to internal validity

• history

• maturation

• testing

• mortality/attrition

• selection

• diffusion/imitation of treatment

• regression to the mean

• instrumentation

• noncomplicance

primary control → random assignment (randomization)

threats to external validity

• interaction between subjects and treatment: population validity

• interaction between setting and treatment: ecological validity

• interaction between history and treatment: temporal validity

primary control → random sampling (for representativeness)

random sampling

process of choosing a “representative” sample from an entire population such that every member of the population has an equal and independent chance of being selected into the sample

• ideally all research design/studies should use

random assignment

control technique that equates groups of participants by ensuring that every member (of the sample) has an equal chance of being assigned to any group

bias

systematically favoring certain outcomes

• sampling: systematically favoring the selection of certain individuals from the population

• sample lacks of representativeness of population

biased sampling methods

• convenience sample: select individuals from the population who are easiest to reach

• voluntary response sample: individuals from the population self-select into the sample based on general appeal

simple random sample (SRS)

every individual in population AND every possible sample has equal chance to be selected

stratified random sampling

divide the sampling frame into distinct groups of individuals, called strata

take a separate SRS into each stratum and combine these to make up the complete sample

statistical-conclusion validity

• appropriateness of inferences made from data as a function of conclusions drawn from statistical analyses

• is the difference between groups (or relationship between the IV and DV) statistically significant

• is the difference/relationship simply a function of chance?

threats to statistical-conclusion validity

• small sample size → low statistical power

• using incorrect statistical test

• violations of statistical assumptions

• poor reliability of observed scores

• power analysis

  • sample size (N)

  • effect size (magnitude of effort)

  • statistical power (1-B) → .80

    • B is the probability of committing a Type II error

  • alpha (p-value)

    • p is the probability of committing a Type I error

sampling variability/sampling error

statistics differ cross multiple samples from the same population

good sampling method → small bias and small sampling error

comparing 2 conditions

t-test (independent vs. dependent t-test)

comparing more than 2 conditions

ANOVA

do 2 variables covary?

correlation

correlation w/ prediction and/or more than !X

regression

frequency/counts; just groups

chi-square

good sampling methods

small bias and small sampling error

witihin-subjects tests

dependent t-test or repeated measures ANOVA

between-subjects tests

independent t-test or ANOVA

threats to statistical-conclusion validity

low statistical power

• control → do a power analysis; ensure adequate power (.80)

violations of statistical test assumptions

• control → meet test assumptions

poor reliability of measures

• control → use reliable measures

construct validity

• extent to which labels placed on what is being observed are theoretically relevant/appropriate

• “do the results support the theory underlying the research”

• distinct from internal validity

  • X may cause Y, but do X and Y speak to underlying theory?

  • threat → loose connection between theory and research study

• misalignment between conceptual and operational definition of construct

  • for experiments → does the manipulation adequately operationalize the independent variable

threats to construct validity

• threats associated w/ experimenter

  • experimenter effect - characteristics of experimenter/proctor

  • experimenter expectancy - (un)conscious expectations of experimenter/proctor

• threats associated w/ participants

  • demand characteristics - participant expectations/beliefs about study aims

    • “good subject response”

  • social desirability - participant respond/ behave in socially desirable way

  • Hawthorne effect - participants change behavior as a function of being observed

controlling threat to construct validity

• double-blind procedures

• single-blind procedures

• deception

• separate the IV and DV in time

• use unobtrusive methods/measures