KIN 270: Validity & Reliability in Exercise Science

Vocabulary flashcards covering measurement quality, reliability, validity, and statistical metrics (ICC, Bland-Altman, SEM, MDC, MCID, CCC, MAPE, and RMSE) from KIN 270.

Validity (of a Study)

A property of the research design as a whole that evaluates whether the study's design supports a trustworthy conclusion, encompassing internal and external validity.

Validity (of a Measurement)

A property of a single measurement tool evaluating whether the instrument actually measures what it claims to measure.

Face Validity

A form of measurement validity that evaluates on its face whether an instrument superficially appears to measure the right variable or outcome.

Content Validity

A form of measurement validity that assesses whether an instrument covers the full spectrum of the outcome rather than just a part of it.

Criterion Validity

The degree of agreement between a new or practical measurement tool and an established gold-standard criterion measure.

Concurrent Validity

A subtype of criterion validity reflecting how well a measurement tool agrees with a gold-standard criterion measured at the exact same time.

Predictive Validity

A subtype of criterion validity that reflects how well a measurement tool forecasts or predicts a future outcome.

Construct Validity

A measure of how well a tool evaluates an underlying theoretical concept, verified through convergent validity (correlating with related measures) and discriminant validity (not correlating with unrelated measures).

Test-Retest Reliability

The consistency of scores when the exact same test is administered to the same individuals across two different occasions (also referred to as intra-rater reliability when performed by the same rater).

Inter-Rater Reliability

The degree of agreement between two different raters or technicians scoring the exact same performance or test.

Internal Consistency

The extent to which different items on a multi-item survey or assessment scale hang together and measure the same underlying construct.

Precision vs. Accuracy Principle

Precision refers to reliability (how tightly repeated shots or measurements cluster together), whereas accuracy refers to validity (how close measurements land to the true target). A measure can be reliable without being valid, but it cannot be valid without being reliable.

Variance

The formal calculation of data spread expressed in squared units (such as $\text{mL}^2/\text{kg}^2/\text{min}^2$), calculated as the mean squared difference of individual scores from the mean.

Standard Deviation (SD)

A metric indicating how far individual scores sit from the sample mean, calculated as the square root of variance ($\text{SD} = \sqrt{\frac{\sum (x - \text{mean})^2}{n - 1}}$) to return the spread into original measurement units.

Empirical Rule (68-95 Rule)

In a normal distribution, approximately $68\%$ of scores fall within $\pm 1\,\text{SD}$ of the mean, about $95\%$ fall within $\pm 2\,\text{SD}$, and about $5\%$ sit beyond $2\,\text{SD}$.

Intraclass Correlation Coefficient (ICC)

A unitless metric reflecting the proportion of total variance due to true between-subject differences, calculated as $\text{ICC} = \frac{\text{Variance Between Subjects}}{\text{Variance Between Subjects} + \text{Variance Within Subjects}}$.

ICC Interpretation Thresholds

Standard guidelines for classifying reliability using the Intraclass Correlation Coefficient: poor ($< 0.50$), moderate ($0.50 - 0.75$), good ($0.75 - 0.90$), and excellent ($> 0.90$).

Bland-Altman Plot

A graphical method that plots the difference between two measurements against their mean, displaying a central bias line and $95\%$ limits of agreement ($\text{bias} \pm 2\,\text{SD}$) in original units.

Proportional Bias

An error pattern observed on a Bland-Altman plot where the disagreement between methods changes systematically across the measurement scale (e.g., showing good agreement at lower values but progressively larger error at higher values).

Standard Error of the Mean (SE)

A metric quantifying the precision of a sample mean as an estimate of the population mean, calculated as $\text{SE} = \frac{\text{SD}}{\sqrt{n}}$, which decreases as sample size ($n$) increases.

Standard Error of Measurement (SEM)

A metric reflecting the precision of an individual score by calculating how much one person's measurement could vary from error alone, calculated as $\text{SEM} = \text{SD} \times \sqrt{1 - \text{ICC}}$; it is independent of sample size.

Minimal Detectable Change (MDC)

A statistical threshold defining the smallest change that can be declared real with $95\%$ confidence rather than measurement noise, calculated as $\text{MDC} = 1.96 \times \sqrt{2} \times \text{SEM}$.

Minimal Clinically Important Difference (MCID)

A practical or clinical threshold determined by clinical reasoning rather than measurement precision (e.g., $1\,\text{MET} \approx 3.5\,\text{mL/kg/min}$ for $\text{VO}_2\text{max}$ or $10\%$ weight loss in obesity research) to identify whether a change meaningfully affects health or performance.

Pearson Correlation (Limitation in Agreement)

A statistic ($r$) that only measures how strongly two variables move together along a straight line, failing to detect systematic differences or absolute disagreement (e.g., when a device reads consistently higher than a criterion).

Lin's Concordance Correlation Coefficient (CCC)

A validity statistic calculated as $\text{CCC} = r \times C_b$, which tests whether points fall directly on the line of identity ($y = x$) by multiplying precision ($r$) by an accuracy factor ($C_b$).

Location Shift and Scale Shift

Discrepancies that penalize Lin's CCC by reducing the accuracy correction factor ($C_b$) while leaving Pearson $r$ unaffected; location shift occurs when one method consistently reads higher/lower, and scale shift occurs when one method's spread is stretched or compressed.

Mean Absolute Percentage Error (MAPE)

A validity error statistic calculated as $\text{mean}\left(\frac{|\text{device} - \text{criterion}|}{\text{criterion}}\right) \times 100$, representing typical error as a percentage of the criterion value (a common benchmark for exercise heart rate is $\le 5\%$).

Root Mean Square Error (RMSE)

A validity error statistic expressed in original units calculated as $\sqrt{\text{mean}\left((\text{device} - \text{criterion})^2\right)}$, which weights large misses more heavily than small misses due to the squaring of error values.

Coefficient of Variation (CV%)

A percentage-based reliability statistic that quantifies the consistency and repeatability of repeated measurements made by the same instrument.

Validity (of a Study)

A property of the overall research design evaluating whether the study's design supports a trustworthy and sound conclusion (encompassing internal and external validity).

Validity (of a Measurement)

A property of a single measurement tool evaluating whether the instrument accurately measures the construct it claims to measure.

Face Validity

A qualitative form of measurement validity assessing whether an instrument, on its surface or appearance, looks like it measures the intended variable.

Content Validity

The degree to which an assessment tool covers the full spectrum and all dimensions of the outcome being measured, rather than only a portion of it.

Criterion Validity

The extent to which scores from a measurement instrument agree with an established gold-standard criterion measure, evaluated concurrently or predictively.

Concurrent Validity

A subtype of criterion validity that tests whether a new or practical measurement agrees with an established gold standard measured at the exact same time.

Predictive Validity

A subtype of criterion validity assessing how accurately an instrument's current scores forecast a future performance, clinical event, or outcome.

Construct Validity

The degree to which a test measures the theoretical construct it was designed to measure, established through both convergent and discriminant evidence.

Convergent Validity

An aspect of construct validity showing that scores on the measurement tool correlate strongly with scores from other established instruments measuring related constructs.

Discriminant Validity

An aspect of construct validity showing that scores on the measurement tool do not correlate with tests measuring unrelated constructs.

Reliability

The consistency, repeatability, or stability of a measurement tool across repeated trials under identical conditions (representing measurement precision).

Test-Retest Reliability

The consistency of scores obtained when the same test is administered to the same group of individuals on two separate occasions.

Inter-Rater Reliability

The degree of agreement between two or more different raters, technicians, or examiners evaluating the exact same performance or participant.

Internal Consistency

The extent to which individual test items hang together and measure the same underlying construct on a multi-item questionnaire or scale.

Precision vs. Accuracy

Precision reflects reliability (how closely repeated measurements cluster together), whereas accuracy reflects validity (how close measurements fall to the true target value).

Reliable, Not Valid (Target Pattern)

A measurement scenario characterized by high precision (tightly clustered repeated scores) but low accuracy (a systematic bias consistently missing the true value).

Variance

A formal statistical calculation of the spread of scores around the mean, expressed in squared measurement units ($\text{units}^2$).

Standard Deviation ($SD$)

The square root of variance, rescaled back into original measurement units, describing how far individual scores sit from the sample mean: $SD = \sqrt{\frac{\sum(x - \text{mean})^2}{n - 1}}$.

Intraclass Correlation Coefficient ($ICC$)

A unitless reliability and agreement metric calculated as $\frac{\text{Variance Between Subjects}}{\text{Variance Between Subjects} + \text{Variance Within Subjects}}$, reflecting the proportion of variance attributable to true differences rather than measurement error.

Bland-Altman Plot

A graphical method that plots the difference between two measurements against their mean, displaying the mean systematic bias and the $95\%$ limits of agreement ($\pm 2\,SD$).

Proportional Bias (Bland-Altman)

A scenario in a Bland-Altman plot where the disagreement between two methods changes across the measurement range (e.g., agreeing well at low values but diverging at high values), preventing a single bias number from describing the error.

Standard Error ($SE$)

The precision of a sample mean as an estimate of the true population mean ($SE = \frac{SD}{\sqrt{n}}$), which decreases as sample size increases.

Standard Error of Measurement ($SEM$)

The precision of an individual score showing how much a person's score varies from measurement error alone ($SEM = SD \times \sqrt{1 - ICC}$); it remains independent of sample size.

Minimal Detectable Change ($MDC$)

A statistical threshold representing the smallest change in an individual score that can be declared real with $95\%$ confidence rather than measurement noise: $MDC = 1.96 \times \sqrt{2} \times SEM$.

Minimal Clinically Important Difference ($MCID$)

A clinical or practical benchmark derived from expert reasoning (e.g., $1\,\text{MET} \approx 3.5\,\text{mL/kg/min}$ in cardiorespiratory fitness) that determines whether an observed change is meaningful for health or performance.

Lin's Concordance Correlation Coefficient ($CCC$)

A unitless validity index ($CCC = r \times C_b$) evaluating both precision ($r$) and accuracy ($C_b$) to test whether data points fall directly on the line of identity ($y = x$).

Mean Absolute Percentage Error ($MAPE$)

An error-size validity metric calculating average absolute error as a percentage of the criterion value: $\text{MAPE} = \text{mean}\left(\frac{|\text{device} - \text{criterion}|}{\text{criterion}}\right) \times 100$.

Root Mean Square Error ($RMSE$)

An error-size validity metric expressed in the measurement's original units ($\text{RMSE} = \sqrt{\text{mean}((\text{device} - \text{criterion})^2)}$) that penalizes large errors more heavily due to squaring.