CR Exam 1

To quantify continuous data, what reliability coefficients can be used, and which is preferred?

Pearson correlation (r) and Intraclass correlation coefficient (ICC), ICC is preferred

To quantify discrete data, what reliability coefficients can be used, and which is preferred?

Percent agreement and Kappa, Kappa is preferred

What does quantifying reliability mean?

Expressing the consistency of a measurement or test using numbers, typically as a coefficient or score

What is a dichotomous variable

Binary

ex. yes or no, male or female, have it or you don't

What is a construct?

A construct is an unobservable thing (abstraction), a general concept that must be defined (ex: strength, balance, coordination)

What is an operational definition?

Act of turning a construct into a measurable variable

Continuous variable:

Any value along a continuum within a defined range (ROM, weight, height), it can be a portion or fraction of something

Discrete variable:

Values that are described in only whole units (ex: BPM, children, etc.), dichotomous variables fall under this category

What two levels of measurement fall under "continuous" or "metric"

Ratio and interval

Highest to lowest level of measurement:

Ratio, interval, ordinal, nominal

What is a nominal variable?

For categorization and categorical, there is no quantitative value and is primarily counting frequency only; no category is better than the other

Examples of nominal variables:

Sex, race, occupation, history of cancer

What is an ordinal variable?

Rank-ordered categories but do not know the difference in intervals between the ranks, might have unequal intervals

Examples of ordinal variables:

MMT, income categories, hypo/normal/hyper, NPRS

What is an interval variable?

Rank-ordered variables with equal intervals but no true zero

Examples of interval variables:

Years (A.D. vs B.C.), shoe size, degrees in C or F

What is a ratio interval?

Interval scale with a true zero, so no negative values

Examples of ratio variables:

ROM, height, weight, age, distance

Can you do ratios with interval variables?

No

When converting between ratio, interval, ordinal, or nominal variables; can you convert up? Can you convert down between them?

It is possible to convert scale to a lower level, but you will lost information; it is impossible to convert to a higher level

Another word for independent variable

Factor

What is the independent variable?

What you manipulate or change, have to have two or more groups

What is the dependent variable?

The measured variable, response to the changes in IV's

What is an active varible?

The IV a researchers actually manipulates or can control at the levels of "treatments"

What is an attribute variable?

Cannot be manipulated (ex: age, sex, etc.)

What are independent factors for IV?

Different groups of people for each level; AKA "between-subject" factors

What are repeated factors for IV?

Same people used to measure all levels of IV, subjects are used as their own controls; AKA "within-subject" factors

What is a multifactorial design?

Two or more IVs

What is a single-factor design?

Only has one IV

What is a 2 x 3 factorial design?

It is a design with two independent variables, one with two levels (ex: male or female) and the other with three levels (ex: low fat, low carb, paleo)

What is an univariate design?

Only one dependent variable measured (ex: only ROM)

What is a multivariate design?

More than one dependent variable measured (ex: ROM and BMI)

What is reliability?

Extent to which a measurement is consistent and free from error (consistent and reproducible)

What is validity?

Accuracy, correctness of measurement

Measurement error examples

Tied to reliability; bathroom scale +/- 1 lbs, ROM +/- 5 degrees, Oswestry scale +/- 4 points

What is the equation for measurement theory?

Observed score = true score +/- error (we never really know true score)

Types of measurement error:

Systematic error and random error

What is a systematic error?

An error that causes the measurements to differ from the true value by a consistent amount (always overestimate/underestimate)

What is random error?

Due to chance, from measurement to measurement the values are unpredictable

Sources of measurement error:

Rater, instrument, variability of characteristic being observed

What is rater measurement error?

Error in perception or reporting of measurement by observer

What is instrument measurement error?

Faulty function of mechanical instrument

What is variability of characteristic being observed measurement error?

Natural variability

Ways to improve reliability:

1) Standardize measurement methods

2) Train and test observers

3) Refine, automate, calibrate instruments

4) Blind to reduce differential bias across study groups

When does regression to the mean become an issue?

When people or groups are selected based on extreme scores, could be random error

In terms of regression to the mean, random measurement errors are random and therefore __________ distributed

Normally

What is the best estimate of "true" performance comparing average to best scores?

Average

Classical measurement theory

Every observation of a variable can be divided into two components: true score and error

Generalizability theory

A newer/more complex approach to the question of reliability that considers all types of error in reliability estimates (divides random error into components)

Definition of a variable

A property that can take on different values

What is the old approach to quantifying reliability?

Use of correlation coefficients (pearson's r), which showed the extent of a linear relationship from 0 to 1

Why is there an issue with using the old approach to quantify reliability?

Assesses relationship and not the agreement, it also only allowed for two raters or occasions to be compared

_________ and __________ give single indicators of reliability that capture strength of relationship plus agreement in a single value

ICCs; Kappa

Reliability coefficient equation

true score variability / (true score variability + error variability)

Poor agreement of ICC/Kappa values

0-0.20

Fair agreement of ICC/Kappa values

0.21-0.40

Moderate agreement of ICC/Kappa values

0.41-0.60

Strong agreement of ICC/Kappa values

0.61-0.80

Near complete agreement of ICC/Kappa values

0.81-1.00

Difference in ICC/Kappa and SEM:

ICC and Kappa are relative measure of reliability (no units), but SEM is the absolute measure of reliability

What is Standard Error of Measurement (SEM)?

Absolute measure of reliability, in units of measurement of variable; SEM = SD √1- ICC

What is the minimal detectable change (MDC)?

Smallest amount of change in a variable that must be achieved to reflect a true change/difference, will always be larger than the SEM and at a 95% confidence level

MDC equation

= X +/- (1.96) (SEM) √2

Types of reliability:

Inter-rater, intra-rater, test-retest, alternate forms, internal consistency

What is inter-rater reliability?

Variation between 2 or more raters who measure the same group of people, best assessed in a single trial at the same time

What is intra-rater reliability?

The degree that the examiner agrees with himself or herself

What is rater bias?

When one rater takes 2 measurements; influenced by memory of first score

What is test-retest reliability?

Used to establish that an instrument/tool is capable of measuring a variable consistently

What reliability is used most often for self-report questionnaires, and measures with mechanical or digital readouts?

Test-retest reliability (no rater component)

What are alternate/parallel forms reliability used for?

To see if 2 versions of the same instrument are equivalent

What is internal consistency reliability?

The extent to which different items on a test or scale measure the same underlying construct

What is the internal consistency reliability often used for?

To construct and evaluate scales/questionnaires

What is used to measure internal consistency?

Cronbach's alpha values, ideally between 0.7 to 0.9

What forms of reliability are most relevant for clinicians?

Inter-rater, intra-rater, test-retest

What types of reliability are used to look at questionnaires and surveys or for comparing different forms of testing?

Alternate forms, internal consistency

What is split half reliability?

A form of internal consistency, take all questions and divide (odd-even) then compare halves

What is measuring validity?

Extent to which an instrument measures what it is intended to measure

___________ is a prerequisite for validity

Reliability

The 3 C's for validity

Construct validity, content validity, criterion-related validity

What is content (or face) validity?

Establishes that the multiple items that make up a questionnaire, inventory, or scale adequately sample the universe of content that defines the construct being measured

What is criterion-related validity?

Establishes the correspondence between a target test and a reference or gold standard measure of the same construct

Approaches of criterion-related validity:

Concurrent validity and predictive validity

What is construct validity?

Establishes the ability of an instrument to measure the dimensions and theoretical foundation of an abstract construct

Approaches of construct validity:

Convergent, discriminant validity, known groups method

What is convergent validity?

The extent to which a test correlates with other tests of closely related constructs

What is discriminant validity?

The extent to which a test is uncorrelated with tests of distinct or contrasting constructs

What is concurrent validity?

The extend to which the target test correlates with a reference standard taken at relatively the same time

What is predictive validity?

The extent to which the target test can predict a future reference standard

What is the gold-standard, highest level of validity?

Criterion validity

What is known groups method for construct validity?

Do test results differ between 2 groups known to be different

What is responsiveness in measuring change?

The ability of an instrument to detect minimal change over time

Types of index to measure responsiveness:

MDC and MCID

What is minimal clinically important difference (MCID)?

The smallest difference that signifies an importance to the patient rather than trivial difference, usually larger than MDC

Why does MDC concern reliability rather than validity?

Because it is measurement error

Why does MCID concern validity rather than reliability?

Because it is what we are intending to measure

Why do we use research and evidence based practices?

The ultimate goal of clinical research is to maximize the effectiveness of practice, we need to be able to "upgrade" our knowledge

What is the highest form of truth for sources of knowledge?

Scientific method

What is inductive reasoning?

Starting with a specific observation and leading to general conclusion

What is deductive reasoning?

Starting with general observation and leading to specific conclusion

What falls under "clinical reasoning and experience" in the sources of knowledge chart?

Deductive/inductive reasoning (clinical reasoning) and trial and error (clinical experience)