HHS Exam 2

Validity

Will results remain truthful when applied to people, situations, or objects outside of the
original study (generalization)?

Reliability

the extent to which an experiment, test, or any measuring procedure yields the same result on repeated trials.

How well constructed is the study?

internal validity

How generalizable are the findings?

external validity

Internal validity

The structure of the study and whether it’s doing what it’s supposed to do

Is the study well constructed?
Allows a valid explanation or description and rules out
competing explanations or descriptions

For an experimental study (Internal V)

– sufficient control of extraneous variables and sources of error
– Must be able to attribute changes in the dependent variable to manipulation of the independent variable

For a descriptive study (Internal V)

– sufficient control and systematic data collection
– to present a convincing description of what occurred

Threats for internal validity

1. Ambiguous temporal procedures
2. Differential Participant (subject) selection p 164
3. History – events occurring between first and second
measurements (p 160)
4. Maturation – changes in subjects that cannot be
controlled (p 161)
5. Statistical regression to the mean – subject selected
based on high or low scores improve spontaneously
(p. 163-164)
6. Attrition – differential loss of subjects between
groups (p 165)

7. Multiple testing effects (Reactive measures) – raking
a test can affect scores on next administrations
(p161)
8. Instrumentation error – changes in calibration of a
measuring instrument (pp 162-163, 264)
9. Additive & interaction effects – interacion between
two factors (p 166)
10. Experimenter Bias (Rosenthal/Pygmalion
Effect/Self-fulfilling prophecy) (p. 167-168)
11. Participant Bias (Hawthorne Effect/Observer Effect)
– changes in behavior due to knowing you are being
observed experimentally (p 172)
12. Belief (Placebo Effect) – changes due to believing
you are receiving treatment

Threats for external validity

1. Participant selection pp 170-171
– Target population vs. accessible population
2. Treatment (Tx) fidelity - describing the independent
variable explicitly
3. (Interactive) Multiple Tx interference pp 172-172
– Pure txs may be rare
4. Novelty & Disruption
5. Experimenter effects p 172
6. Pretest and post test sensitization p. 171
7. Measurement precision especially of dependent variable

External validity

Generalizability to other subjects

• Degree to which results can be generalized from
the sample to the population
• Sometimes a trade-off with internal validity
– Highly controlled “perfect” study  real life applications
• Population validity:
– How representative is the sample of the population?
– Generalizing across people even those not represented
in the sample
• Ecological validity:
– Can result generalizes across settings (environments)

Internal validity

• Procedures designed to test a hypothesis
• Aim for a well-structured study with good internal
validity
• Choose the right design for the question
• Plan the procedures as well as you can given your
resources

Types of reliability

• test, re-test reliability

• interrater reliability

• intraratar reliability

Test, re-test reliability

the reliability of a test measured over time.

Interrater reliability

the extent to which two or more raters (or observers, coders, examiners) agree.

Intrarater reliability

a measure of how consistent an individual is at measuring a constant phenomenon

What is the best way to control for most threats to internal validity?

Increasing sample size

Randomization

reliable but not valid

valid but not reliable

neither reliable nor valid

both reliable and valid

Population

All of the persons of interest for a particular study

Sample

Smaller group of persons, from the population, who actually participate in a study

Parameter

Numerical summary based on an entire population

Measure

Statistic

Numerical summary based on a sample from a population

Mathematical formula

List and describe three sources of bias in selecting participants for a study

– Failing to identify all members (e.g., recruiting first
responders to the ad)
– Using samples of convenience
– Volunteerism
• Unavoidable source of bias ( i.e., volunteers may differ from the general population in some systematic way)

One criterion for establishing a population for a study is that all members have at least one characteristic in common. Explain.

This is important because it helps to define the scope and relevance of the study, ensuring that the results are meaningful and applicable to the group being studied.

Are populations always large? Can you think of an example when a population might
not be terribly large?

The size of the population often depends on the focus of the study, the rarity of the characteristic being studied, and the available resources.

Ex: Rare Diseases or Conditions: If a study is focused on a rare disease or condition, the population might be small. For example, research into a rare genetic disorder like Huntington's disease or a rare form of cancer might involve only a small number of participants because the condition is not widespread.

What is the main goal in recruiting a sample? Explain how data from a sample is
used to make an inference about the population.

The main goal in recruiting a sample is to select a subset of individuals from a larger population in a way that accurately represents that population. By using a sample, researchers can gather data and draw conclusions about the broader population without needing to collect data from every single member of the population, which may be impractical or impossible.

Once a sample is selected, data is collected and analyzed. Researchers then use the data from the sample to make inferences about the population as a whole.

Define inference

a conclusion or generalization drawn from sample data that is intended to apply to a broader population. It involves using the sample's findings to make estimates, predictions, or judgments about the population from which the sample was drawn.

True or false: Increasing sample size should increase how well a sample reflects a
population. Explain.

Depends on where you get the sample

A larger sample is more likely to include a diverse range of individuals from the population, which helps capture more of the variability and characteristics of the population.

True or false: Increasing sample size decreases systematic bias. Explain.

False

systematic bias means that it is built into the way we do things so by adding it we would just keep doing it over and over again.

If a measure is reliable, is still may not be valid, but if a measure is valid, then it must
be reliable. Explain.

You can have a measure that’s very reliable and you can get people to measure it the same way time and time again but it may not reflect what it’s supposed to be. If its valid we know that it does what it’s supposed to do so it’s reliable

What are two important characteristics of true experimental design?

There has be to randomization and reasonable. You have to be sure that the groups are equivalent start out with

Randomized pretest post test control group design

Factorial design

Why is experimental control important? How does it affect internal validity?

Unless you can document experimental control, you’re not assured that the effect that you see is the result of your treatment. You want to be able to demonstrate that your treatment makes a difference.