Week 10: Validity in Research Designs

Internal validity

In experimental and quasi-experimental research designs, the degree to which a change in the outcome can be attributed to the experimental intervention rather than to extraneous factors.

Research validity

The extent to which a study's methods can answer the research question posed.

Quantitative Research Validity (Believability)

Accuracy of study results — the extent to which the study measures what it claims to.

Quantitative research validity key question

Can alternative explanations for the results be ruled out? *If yes → strong validity; if no → threats are present.

External validity

The degree to which quantitative research may be applied to other individuals and circumstances outside of a study.

external validity aka

applicability or generalizability

Transferability

The degree to which qualitative research results may be applied to other individuals and circumstances outside of a study.

Selection

The process by which subjects are chosen from a potential candidate pool.

Assignment

The process by which subjects are placed into two or more groups in a study.

Inadequate assignment procedures

May threaten research (internal) validity by producing groups that are different from one another at the start of the study.

assignment aka

allocation

Attrition

A term that refers to subjects who stop participation in a study for any reason.

Loss of subjects may threaten research validity by

reducing the sample size, producing inequal groups.

attrition aka

dropout or mortality

Power

The probability that a statistical test will detect, if it is present, a relationship between two or more variables or a difference between two or more groups.

Maturation

A threat to research validity characterized by the natural processes of changes that occur over time in humans that may influence a study's results independent of any other factors.

Compensatory rivalry

A threat to research validity characterized by subjects in the control (comparison) group who, in response to knowledge about group assignment, change their behavior in an effort to achieve the same benefit as subjects in the experimental group.

Resentful demoralization

A threat to research validity characterized by subjects in the control group who, in response to knowledge of group assignment, limit their efforts in the study.

Placebo

A treatment that is inert and cannot influence a patient's physiology or function.

Diffusion or imitation of treatment

A threat to research validity (internal validity) characterized by a change in subject behavior that may occur as a result of communication among members of different study groups.

Statistical regression to the mean

A threat to research validity (internal validity) that may occur when subjects produce an extreme value on a single test administration; mathematically the next test scores for these individuals mostly will move toward the mean value for the measure.

Compensatory equalization of treatment

A threat to research validity (internal validity) characterized by the purposeful or inadvertent provision of additional encouragement or practice to subjects in the control (comparison) group in recognition that they are not receiving the experimental intervention.

History

A threat to research validity (internal validity) characterized by events that occur during a study that are unrelated to the project but may influence its results.

Instrumentation

A threat to research validity (internal validity) characterized by problems with the tools used to collect data that may influence the study's outcomes.

Testing

A threat to research validity (internal validity) characterized by a subject's change in performance due to growing familiarity with the testing or measurement procedure or to inconsistent implementation of the procedures by study personnel.

Intention-to-treat analysis

Statistical analysis of data from subjects according to the group to which they were assigned despite noncompliance with the study protocol

Construct underrepresentation

A threat to construct validity characterized by measures that do not fully define the variable or construct of interest.

Construct validity

The degree to which a measure matches the operational definition of the concept or construct it is said to represent.

Triangulation

A method to confirm a concept or perspective generated through the qualitative research process.

Research Validity

The extent to which conclusions of a research study are:

Believable

Useful

Whether the proposed explanation for the outcome of interest is really the right explanation...or can a competing explanation be identified?

research validity

Threats to research validity

subjects, investigators, study logistics

threats to subjects

Selection

Assignment

Attrition

Maturation

Compensatory Rivalry/Resentful Demoralization

Diffusion or Imitation of Treatment

Statistical Regression to the Mean

threats to investigators

compensatory equalization of treatments

threats to study logistics

-history

-instrumentation

-testing

threat of selection

Selection process leads to sample that is not representative of the population from which it is drawn

solutions to selection threat

Recruit subjects from Multiple study sites

Probabilistic (randomized) selection

threat asssignment

Group assignment process leads to unequal distribution of subject characteristics

solutions to assignment threat

Probabilistic (randomized) assignment

Perform statistical adjustment

threat of attrition

Loss of subjects during the course of the study resulting in reduction of sample size and inequality of baseline characteristics

solutions to attrition

Replacement of those lost

Statistical comparison w/ those lost

Statistical comparison of groups

Intention to treat analysis

Threat of maturation

Natural changes in human behavior over time that are internal to the subjects and may influence the outcome

solutions to maturation

Control or comparison group

Repeated baseline measures

Scheduling

Time the study to minimize the effect

Randomize the testing order

Compensatory Rivalry/Resentful Demoralization

Problem: Subjects change behavior in response to learning they are in the control group.

solutions to compensatory rivalry/resentful demoralization

Ask subjects to avoid behavior change

Masking

Instructions re: adherence to protocol

Separation of subjects

Threat: diffusion or imitation of treatment

Change in behavior that occur in the control or comparison group as a result of communication with subjects in the experimental group about interventions they are receiving

solutions for diffusion or imitation of treatment

1)Keep groups separated

2)Blind participants to group assignment

3)Ask participants to avoid behavior changes over the study period

Threat: statistical regression to the mean

Appearance of change due to an extreme score for the baseline measure on the outcome of interest

solutions for statistical regression to the mean

Trim outliers

Average repeated baseline measures

Compensatory Equalization of Treatments

Problem: Purposeful or inadvertent supplementation of the control or comparison group's activities that influence performance.

solutions for Equalization of Treatments

Masking the investigators

Implement Protocols for intervention administration

Different locations

Ask participants not to discuss with anyone

threat: history

events that occur during the course of a study to all or individual participants that can result in bias

Concurrent event(s) occurring unrelated to the study that are out of the investigators' control and may influence outcome

solutions for history

Control or comparison group

time the study to avoid the events

Perform statistical analysis

threat: instrumentation

Wrong measurement approach or device

Limitation of measure

Malfunction

Inaccurate application

solutions for instrumentation

Operational definitions

Selection of most rigorous instrument

Calibration of instruments

Use specific Protocols

Training and verification

threat: testing

Appearance of improvement due to familiarity with test procedures or in response to different instructions/cues

solutions to threat of testing

Provide Practice sessions

Average scores from Repeated measures

Use Protocols for test administration like scripts

Training and verification

Intention to Treat Analysis

Potential Solution(s) for attrition: Analyze all participants as randomized regardless of whether they completed the study.

Operational Definitions

Potential Solution(s) for instrumentation: Define measures clearly to ensure accurate application.

Selection Bias

Nature of the threat: Use subjects who don't represent the population of interest limits the ability to determine how well the diagnostic test differentiates between different stages of the disorder.

Threats to the research validity of studies about elements of the PCMM other than interventions

Selection: all studies

Attrition: all studies

Maturation

Sats regress to the mean

History

Instrumentation

testing

maturation studies affected

Primarily longitudinal studies of prognostic factors CPRS about prognosis and outcomes

Statistical regression to the mean studies

All study types

History study types affected

Primarily longitudinal studies of prognostic factors CPRS about prognosis and outcomes

Instrumentation study types affected

Primarily studies of diagnostic tests, clinical measures, and clinical prediction rules about diagnosis

Testing study types affected

Primarily studies of diagnostic tests, clinical measures, and clinical prediction rules about diagnosis

Investigator bias

Purposeful or inadvertent interference with the study's procedures

Selection

Assignment

Instrumentation

Selection and investigator bias related to diagnostic studies

Use subjects who don't represent the population of interest limits the ability to determine how well the diagnostic test of CPR differentiates between or among different stages of the disorder of interest

Solution to Selection and investigator bias related to diagnostic studies

adequately define inclusion and exclusion to create a sample that represents the spectrum of the disorder of interest

Selection and investigator bias related to prognostic studies

use subjects who are too sick or too healthy may result in a misrepresentation of the timetable for recovery or adverse outcome thereby undermining the usefulness of a prognostic factor(s)

solution to Selection and investigator bias related to prognostic studies

enroll subjects at a common early point in condition

Selection and investigator bias related to intervention and outcome studies

Use of subjects who don't represent the population of interest limits the ability to determine usefulness of intervention for individuals with different levels of the condition or prognostic profile

solution for Selection and investigator bias related to intervention and outcome studies

adequately define inclusion and exclusion criteria

Assignment related to investigator bias and intervention studies

Individuals responsible for enrolling subjects interfere with group assignment process and may create an unbalanced group

solution for Assignment related to investigator bias and intervention studies

create a predetermined subject assignment list and conceal info about allocation

Testing and investigator bias related to diagnostic studies

investigators apply the superior comparison test only to subjects who test positive which may overestimate the usefulness of the diagnostic test

solution to Testing and investigator bias related to diagnostic studies

apply the superior comparison test to all subjects

testing and investigator bias related to Diagnostic tests, clinical measures, prognostic factors, interventions, CPRs, outcomes

Individuals responsible for the measurement of subjects are influenced in their interpretation by knowledge of the current subject status or prior test or measurement results

solution for testing and investigator bias related to Diagnostic tests, clinical measures, prognostic factors, interventions, CPRs, outcomes

Conceal info about subject status or prior test results from individuals collecting measures

Statistical adjustment

Method for controlling extraneous variables during statistical analysis

Used when design features cannot (or do not successfully) control confounding influences

Statistical adjustment uses

-control variables (e.g., time, tester, baseline differences) in analysis.

-intention-to-treat analysis for missing data.

-Acknowledge limitations if threats cannot be controlled.

Research Validity - Construct

Relates to the specification (operational definitions) of concepts and variables as well as the manner in which they are used within the study

Is the concept/variable logically defined and appropriate for the framework (question) being studied?

Threats to construct validity

-Construct underrepresentation

-Experimenter expectancies

-Interaction between different treatments

-Interactions between testing and treatments

subject expectancies

subject-expectancy effect

Condition in which an experimental participant believes they know what the experimenter is looking for, or has expectations related to the experiment that affect his or her behaviour in the experiment.

Hawthorne effect

A change in a subject's behavior caused simply by the awareness of being studied

solutions to threats to construct validity

Clear operational definitions, direct measurement, masking, and documentation of procedures.

Construct under-representation

Occurs when a concept is not fully captured by the operational definitions used in a study.

Experimenter expectancies

When investigator cues influence participant behavior.

Believability of qualitative research designs

The issue of 'trustworthiness' of information that is subjective by definition.

Methods to assure "believability"

triangulation

reflexivity

triangulation

confirmation of information using multiple sources of data or data collection methods

Reflexivity

Continual testing of assumptions by the researcher as the study progresses.

Ongoing self-reflection by researchers on their biases and assumptions.

study relevance

"External validity" (quantitative) / "Transferability" (qualitative)

- The extent to which results of the research study can be generalized: across groups, settings, times VS to particular persons, settings, times

Threats to study relevance

Inadequate sample selection

Differences in settings

Differences in circumstances due to the passage of time

Solutions for study relevance threats

Large random samples, real-world conditions, and recent data.

Threats to Validity in Non-Intervention Quantitative Studies

Relevant threats include selection, attrition, statistical regression, history, maturation, instrumentation, and testing.

Internal validity

Asks how well the research design

Focuses on what the researcher wants to know

And prevents unwanted influences from affecting the study's results

External validity

Does the study apply to "real world"

Can the reader apply results across groups, individuals, settings or times specific to clinical situation

How well we can generalize the study results to the population

Generalizability

The degree to which study results can be applied across different groups, individuals, settings, or times.

For every study you read Ask yourself

Can competing (alternative) explanations for the results of the study be identified?

If yes

What are the specific problems with elements of research design

What unwanted influences are introduced as a result of these problems

What alternative explanations are possible due to