5.1 reliability

5.1 reliability

Cohen’s kappa (or κ)

• is a measure of inter-rater reliability used on nominal or ordinal data.

Example of cohen´s kappa

• when different raters agree on whether some has a disorder or not, or when raters assess the severity of a disorder: mild, moderate, severe.

κ formula

refers to

• the proportion of agreements

refers to

• the proportion of agreements that could be due to chance

Example 2: Inter-rater reliability

Guidelines to interpret Cohen’s kappa

• κ = 0− 0.20 : none

• κ = 0.21 − 0.39 : minimal

• κ = 0.40− 0.59 : weak

• κ = 0.60− 0.79 : moderate

• κ = 0.80− 0.90 : strong

• κ above 0.90: almost perfect

Cohen´s kappa guidelines apply to the interpretation

• of inter-rater agreement indexes in general & applies to intra-class correlation

internal consistency

• a way of calculating reliability for a single application of the tes

Reliability as internal consistency → how it works

• we split the test in two parts or in k parts (k being the total number of items) & we calculate the reliability among these different partitions of the test

Reliability as internal consistency strategy focus on

• the internal structure of the test

Reliability as internal consistency → There are a good number of indexes that can be calculated using this approach

• Spearman-Brown

• Cronbach’s α,

• Guttman’s λ6

• McDonald’s omega

Reliability as internal consistency: split-half reliability

• We split the test in two parts → both parts should be as equivalent as possibl

Because tests can have a special structure → Reliability as internal consistency: split-half reliability

• e.g., items go from easier to harder → we typically split the test by taking odd & even items

We can calculate Spearman-Brown index to assess reliability:

refers to

• correlation between odd & even items of the test

Split-half was the first approach developed to assess

• internal consistency but nowadays we hardly use it to assess reliability

The problem about internal consistency

• we have many different ways of dividing the test in two equal parts & each of these divisions will yield a different value for Spearman-Brown.

To avoid the problem we

•  We calculate indexes that ‘split’ the test in as many parts as total number of items we have.

global internal consistency measures.

• calculate indexes that ‘split’ the test in as many parts as total number of items we have.

  • we don’t need to worry about different partitions yielding different results, we calculate indexes that use all of the possible partitions to get a better estimation of internal consistency

Cronbach’s alpha (internal consistency)

• considers the variability of all items of the test

• measure of the average covariance (or correlation) of the items of the test.

• generally ranges from 0 to 1 → the larger the value, the more reliable the test is.

• value smaller than 0.5 → typically considered not acceptable

• A value greater than 0.70 → considered good

Cronbach’s α = 0.8 → Pearson Correlations

Cronbach’s α is very popular because it is

• easy to calculate & interpret; it is also the default in most common statistical software

in practice α can be

• low for highly reliable tests (as measured by other indexes).

α is simply a measure

• measure ‘uniformity’

• in other words → it measures how homogeneous the test is as a whole

• it should not drive decisions in test development

limitations of Cronbach’s αre

• Limitations of coefficient alpha as an index of test unidimensionality.

• Coefficient alpha: Unnecessarily ambiguous; unduly ubiquitous

• On the use, the misuse, & the very limited usefulness of Cronbach’s alpha.

• Commentary on coefficient alpha → A cautionary tale

McDonald’s ω