Chapter 6 (Evaluating the merits of quantitative research)

Logical Validity

Quality of researchers' arguments, their application of theory to support the needs for the study, and the appropriate interpretation of results based on the data

Assessing the logical validity of a study

Have the researchers provided a convincing justification for all aspects of the research project?

Construct Validity

Measures used by researchers do indeed measure what they intended to measure. Bit more concrete to evaluate than logical validity. A main focus of most research methods

Internal Validity

Researchers' ability to claim that any change in an outcome is the result of a treatment or intervention and not a result of other factors. Merits of researchers' claims that any change in an outcome is a result of their intervention and not the result of other factors

External Validity

Abilities to see similar successes of the treatment or intervention with other populations, in other contexts, and across time. Generalization of the intervention

Experimental Studies

One form of quantitative research. Four criteria (logical validity, construct validity, internal validity, and external validity) are used to evaluate the merits of experimental studies. Only two of those criteria (logical validity and construct validity) can be used to evaluate the merits of non-experimental quantitative studies

Importance to Non-Experimental Research

Questions of generalizability

Consolidated Standards of Reporting Trials (CONSORT)

Most widely used guidelines to assist researchers and readers of randomized control trials (RCTs), the "gold standard" of experimental research. Useful in training studies with random assignment to groups. Meant to serve as a guide for what researchers should report

Assessing Logical Validity

Examine the logical flow of researchers' arguments and decision making from the beginning to the end of their research. Ask: Does this logically make sense? Has it been sufficiently justified by the researchers?

Supporting Logical Validity

Entire study must flow logically from introduction through final conclusions. Ensure previous research in the field or study is cited appropriately

Support of Construct Validity

Evidence for and against construct validity ranges on a continuum and varies by context, requiring a judgment call by researchers. Test relation with other existing measures of the same construct. Expect higher correlations between measures of the same construct vs. different constructs. Inconsistency in results provides partial support for construct validity

Nomological Network

Develop a web of evidence describing how a measure should relate to other measures of different constructs and behaviors

Construct Validity Evidence

Ranges on a continuum

Inclusion of a Control Condition

Rules out a number of potential threats to internal validity

Randomization

Provides confidence that results are not due to additional threats to internal validity, such as testing/learning effects or instrument calibration drift

Having a Control Group and Random Assignment

Powerful in detecting causal relationships

Additional Strategies for Internal Validity (Beyond Control Group and Random Assignment)

Implement standardization, e.g., 10-hour overnight fast prior to conditions, standardized meals over 24-hour data collection, remove first 10 minutes of data to avoid equipment influence, ensure consistent total time (e.g., 60 minutes) for treatments across conditions

Standardization

Allows researchers to rule out many threats to internal validity

Challenges for Exercise Training Researchers (Internal Validity)

Hard to blind participants to conditions, which can introduce threats like participants working harder to please researchers. Researchers must make best decisions to rule out as many threats as possible

Low External Validity

Particular exercise training intervention works only with a very specific sample under very specific laboratory conditions

High External Validity

Researchers have increased confidence that their intervention "works" in the ways they claim it does

Key to External Validity

Sample chosen for a study is representative of the populations researchers wish to generalize results to

Enhancing External Validity

Clearly identify inclusion criteria specifying who can/cannot be part of the sample. Types of selection criteria help enhance generalizability to other groups with similar characteristics

Challenges Limiting External Validity

Reliance on non-random selection of participants. Ethical principle of autonomy (freedom of choice imposes natural limitation). Autonomous choice to participate may involve unknown characteristics (e.g., interest in physical activity) that differ from non-volunteers. Populations in studies (e.g., kinesiology students) may have heightened interest in physical activity, limiting broad claims. Relying on WEIRD samples (Western, Educated, Industrialized, Rich, Democratic) reduces generalizability, as they are outliers compared to other population