Lecture 3 - Clinical Significance

What is Clinical Significance?

The significance associated with the clinical change and it is very independent of statistical significance and effect size measures.

Name 3 reasons why clinical significance is useful

1. It can focus on the individual

2. Been around for some time

3. Can be quite complex

Is there one definition of what clinical significance is?

It depends on the particular approach taken

Subjective, pure “clinical” definitions

• Complete absence of symptoms = medical conditions

• Reduction of symptoms to the point that the do not affect day-to-day functioning = psychological problems

Broad definition of clinical significance

An approach to considering the results obtained from intervention research that provides information over and above what is obtained from both NHST and effect size measures by considering the real-world, individual-level “impact” of the intervention. (Ogles et al., 2001)

Why do we need Clinical Significance?

1. Ancillary to both NHST and ES.

2. Provides useful information that compliments both NHST and effect sizes:

3. Helps us make pragmatic decisions about our research.

Name one type of approach to clinical significance

Jacobson and Truax’s Statistical Approach to Clinical Significance

What is Jacobson and Truax’s Statistical Approach to Clinical Significance?

• An attempt to inject statistical rigour into assessment of clinical significance

• Can be applied to any intervention study

• Reasonably straightforward to implement

• Informative and useful adjunct to NHST and effect size measures.

What are the two approaches to Jacobson and Truax’s method?

1. Establish clinical criterion a priori

• Different sources

• Established scales have pre-existing clinical cut-offs

2. A, B, C cutoffs - using normative data

What does A mean for Jacobson and Truax?

• if you have normative data about the population group

The level of functioning subsequent to intervention should fall outside the range of the clinical population, where range is defined as extending to two standard deviations beyond (i.e., in the direction of “normality”) the mean of that population. 

What does B mean for Jacobson and Truax?

• if you have normative data about the population group

The level of functioning subsequent to intervention should fall within the range of the non-clinical/normal population, where range is defined as being within two standard deviations of the mean of that population. 

What does C mean for Jacobson and Truax?

• no data from either groups

The level of functioning subsequent to intervention places that person closer to the mean of the non-clinical/“normal” population that it does to the mean of the clinical population. 

What is the easier approach to determining clinical significance?

1. Previous research

2. Clinical norms

3. Generally accepted guidelines

How do we justify whether a test is good or not?

Reliability and validity

What is reliability?

The consistency of scores obtained by the same persons when re-examined.

Tests need to be…

…reproducible, stable (reliable), and meaningful (valid).

Classical psychometric theory assumes that:

1. Individual’s possess stable traits

2. Errors are random

3. Observed Score = True Score + Error (classical test theory)

Reliability is usually expressed by

reliability coefficient or the standard error of measurement (which is derived from the reliability coefficient)

What are the 4 sources for reliability coefficients?

1. Internal consistency

2. Test-retest

3. Parallel forms

4. Inter-rater

Which reliability coefficient index should you use?

Any of them! As long as you can justify it!

• choose the one that makes the most sense

What is the standard error of measurement?

it is an estimate of the amount of error usually attached to an examinee's obtained score.

If the reliability is low the SEM is

high!

If the reliability is high the SEM is

low!

Why do you calculate SEM?

So you can determine if the change you see in people is in fact due to the treatment/actual improvement and not a reflection of error in measurement.

What are the two factors that influence SEM?

1. reliability (high) = low SEM

2. SD (small) = low SEM

According to classical test theory, reliability can be thought of as…

…the strength of the match between the true score and observed score.