Assessing Validity: Traditional Approaches

Traditional Approaches to Assessing Validity

Statistical Tests Used in Validity Studies

  • Three main types of statistics are commonly found in research papers assessing the validity of measurement methods:
    • Correlation coefficients (Pearson's r or Spearman's rank correlation)
    • Null hypothesis tests (typically a paired t-test, or ANOVA)
    • Linear regression
Correlation Coefficients

  • Used to assess the strength of the relationship between two variables.

  • Ideally, a strong correlation indicates good validity.

  • Example:

    • McMahon et al. compared JumpPype with a force platform (criterion measure) and JumpPype calculated using a JustJump contact mat.
    • Found an extremely strong correlation with an R2R^2 value of 0.9948.
    • Correlation coefficient (r) was 0.9974, indicating an almost perfect correlation.

  • Limitation:

    • Correlation coefficients do not provide any comparison of absolute values or assess bias.
    • In the JumpPype example, despite the high correlation, the mean jump height values were significantly different:
      • Force plate mean: 0.33 meters (33 cm)
      • Contact mat mean: 0.46 meters (46 cm)
    • Correlation coefficients do not provide useful information on how different the measures are, which is crucial for determining if they can be used interchangeably.
Null Hypothesis Testing

  • Commonly used to assess systematic bias (mean difference between two measurements).

  • Examples include paired t-tests or repeated measures ANOVA.

  • Limitation:

    • Large amounts of random error can lead to non-significant results.
    • Non-significant difference does not confirm that two measures are equivalent and can be used interchangeably on an individual level.

  • Example:

    • Comparing direct 1RM measurement with 1RM estimation using a load-velocity profile.
    • No statistically significant difference may be found, leading to the incorrect assumption that the measures are the same.
    • However, a Bland-Altman plot can reveal considerable inter-individual variation and random error, indicating the measures cannot be used interchangeably at the individual level.
Linear Regression

  • Used to assess the type of bias present between measures.

  • Two main types of bias:

    • Fixed Bias: Constant or systematic difference between two variables. The difference remains the same regardless of the magnitude of the variable.
    • Proportional Bias: The difference between the two variables increases as the magnitude of the variable increases.

  • Example:

    • If the difference between jump height measured by a jump mat and a force plate increases as jump height increases, this is proportional bias.

  • Proportional bias cannot be corrected for, indicating the novel measurement is invalid.

  • Fixed bias can be corrected for and calibrated against using the regression equation from linear regression analysis.

  • Limitation:

    • Linear regression does not provide much useful information about how the two measures compare in terms of real values, including the magnitude of systematic difference and random error.

Agreement vs. Validity

  • Instead of focusing on validity (accuracy), it is more relevant to assess agreement (interchangeability) when looking at concurrent validity.
  • The key question is whether the test outcomes for two types of measurement (criterion and practical option) are interchangeable.
  • Examples:
    • Can a cheaper option like a MyJump app be used in place of an expensive force plate?
    • Can an Apple Watch be used to measure energy expenditure instead of a portable metabolic cart (Metamax)?
  • Assessing agreement determines if the practical option gives similar enough results to the criterion measure, allowing it to be used in its place.