2. Reliability and Validity

Diagnostic Reliability

Diagnostic Reliability → refers to the consistency in SZ diagnosis given to the SAME individual.

E.g the severity of the SZ symptoms. There should be consistency - every time a diagnostic system is used it should produce the same diagnosis of SZ.

How can reliability be measured?

• Inter-rater reliability evidence

• Test-retest reliability evidence

Inter-rater Reliability Evidence 

• This is whether 2 independent clinicians give similar/same diagnosis of the same person independently. Example:

The success of the DSM-III (specifically designed to provide a more reliable system because of its multi-axial system) → Carson (1991) claimed that the DSM-III had fixed the problem of inter-rater reliability once and for all. Making classification system reliable, lead to greater agreement over who did or did not have SZ.

Example:

+Soderberg et al. (2005) found 81% agreement using the DSM.

+Jakobsen et al. (2005) tested the reliability of the ICD-10, 100 Danish patients were assessed finding a concordance rate of 98% agreement, demonstrating high reliability.

+Osario et al. (2019) report high inter-rater reliability in 180 individuals using DSM5 of +0.97

Test-retest reliability Evidence

• Diagnosis consistent overtime allowing a clinician to make the same diagnosis on separate occasions. 

+Wilks et al (2003) → cognitve screening test to SZ patients over intervals varying from 1-134 days. The test reliability was high at 0.84.

+Osario et al. (2019) report excellent test-retest reliability in 180 individuals using DSM5 of +0.92.

Whalley (2001)

However, there is some research that does not present such a convincing picture.

Reliability is improving. But even if reliability is not perfect, i.e Whalley found inter-rater correlations as low as 0.11.

DSM and ICD do provide practitioners with a common language, permitting communication of findings which may ultimately lead to a better understanding of SZ.

What factors affect reliability? - Reasons for issues

• subjectivity in diagnosis:

• SZ is broad

• Reliance on self-report from SZ

• Cultural differences: Coupleland (71) - higher rates in USA (69%) compared to the UK (25%) → different healthcare priorities.

-DSM more reliabile? - reliability

The DSM is generally regarded as more reliable than the ICD because the symptoms are more specific for each category.

Indeed, Nilsson et al (2000) found only 60% inter-rater reliability using ICD. 

-Schizophrenia is too broad - reliability

It is too broad for it to provide a reliable diagnosis. One issue is that two patients could have completely different symptoms.

-Subjectivity is an issue

Mojtabi and Nicholoson (1995) asked 50 US psychiatrists to distinguish and non-bizarre delusions.

Inter-rater reliability correlation was low at around 0.40, concluding that even this central diagnostic requirement lacks sufficient reliability to distinguish between sz and non-schizophrenic patients.

However, the bvr of an individual is always open to some interpretation. A true diagnosis cannot be made until a patient is clinically interviewed.

Psychiatrists are replying on retrospective data, given by a person whose ability to recall much relevant information is unpredictable.

-Cultural Difference - issue for reliability

• The reliability of sz is challenged by the finding that there is massive variation in diagnosis between countries which implies that culture plays an influence on the diagnostic process.

• Coupeland (1971) gave 134 and 194 British psychiatrists a description of a patient.

• 69% of the US psychiatrists diagnosed schizophrenia but only 2% of British psychiatrists diagnosed sz.

• This may link to the operations of healthcare systems. There is limited time and resources available to many professionals working in the NHS.

• Diagnoses are made by professionals that are rushed, and preoccupied with only admitting the most serous cases in order to safeguard the resources of the institution they are working for.

‘On Being Dane in insane Places’ - Rosenhan’s classic study 1973 - Reliability 

• Rosenhan carried out two studies in U.S psychiatrists hospitals, to find out whether or not the medical staff could judge between mental normality and abnormality. In the first study 8 ordinary people went to hospital admission departments claiming to hear voices saying EMPTY, HOLLOW, THUD; 7 were admitted with diagnosis of sz. 

• In the 2nd study a hospital was warned that some psuedopatients would try to be admitted. There were no fake patients, but about 10% of the real patients were subsequently suspected of being fakes, despite being seen by a psychiatrist or other staff members. This shows the difficulty in distinguishing between normality and abnormality. 

• In the first study the psuedopatients, once admitted, reported experiencing depersonalisation and powerlessness, being ignored by nurses and psychiatrists. Everything they did was interpretated as a symptom of mental illness because thats what the medical staff expected to see.

• This study highlighted that diagnosus of sz is invalid, as psychiatrists could not distinguish between real and psuedopatient leading to inaccurate diagnoses and also that it is unreliable as the patients were seen by a number of staff members (inter-rate). 

• However, it was conducted over 30 years old. Since then manuals have been improved and diagnostic procedures are very different; categories and definitions are more detailed and psychiatrists now use standardised interview schedules when assessing patients.

Diagnostic Validity 

Accurate diagnosis

Refers to the extent that sz represents something that is real and distinct from other disorders and the extent that a classification system as ICD or DSM measure what it claims to measure. 

Descriptive validity

Patients with SZ should differ from other disorders

Criterion validity

Is the extent to which using different classification systems produces the same diagnosis in the same patient. If there are differences here, then it suggests there is a lack of agreement over what schizophrenia actually is.

Elie Cheniaux et al (2009) had two psychiatrists independently rate 100 clients using the DSM4 and ICD10, 68 were diagnosed using ICD criteria and 39 with DSM. This suggests criterion validity is low.

What factors affect validity? - Reasons for issues 

• Comorbidity

• Symptom overlap 

• Culture bias 

• Gender bias 

Comorbordity - issue for Validity 

→ Two conditions co-exist in the same individual at the same time, e.g 50% SZs have depression/substance abuse

• Refers to the extent that 2 or more conditions co-occur in the same individual. 

• Psychiatric comorbidities are common amongst patients with sz.

• This makes descriptive validity very difficult to achieve as it suggests that sz may not actually be a seperate disorder, as if conditions occur together frequently they might be a single condition.

• This also creates difficulties in the diagnosis of sz and in deciding what treatment to advise. 

→ → It leads to invalid diagnoses may not be able to accurately differentiate between the two disorders, e.g 50% SZ have depression, are negative symptoms due to depression or sz? SZ may not be seperate disorders e.g Schizoaffective disorder - sz + dep

Comorbidity → Buckley et al (2009) - issue for Validity

-Buckley et al (2009)’s review found that comorbid depression occurs in 50% of sz patients, 23% OCD, 29% PTSD, 47% substance abuse

• (making validity very difficult to achieve).

Comorbordity → Jeste et al (1996) - issue for validity

-Jeste et al (1996) state that schizophrenics with co-morbid conditions are often excluded from research and yet form the majority of patients.

• Therefore, research findings may not be applicable to the majority of sufferers - highlighting the consequences of this.

Symptom overlap - issue for validity 

→ Shared symptoms - i.e all SZ symptoms can be seen in other disorders, it is not pathognomic

• Another problem is symptom overlap - symptoms of sz are also found in other conditions.

• One of the biggest problems is in differentiating sz from bi-polar disorder, which can also involve delusions and negative symptoms like avolition.

• This makes sz hard to distinguish and can lead to inaccurate diagnoses (depending on the symptom being display at the time of assessment).

→ Symptom overlap leads to invalid diagnoses as it could lead to a misdiagnosis because may exhibit/report a symptom typical of schizoprenia during a clincial interview (e.g. delusions), but they could have that symptom because of bipolar.

Symptom overlap - Schneider (1959) - issue for validity

Schneider - Argued ‘first rank’ symptoms should make its descriptive validity higher. However, schizophrenia is not pathognomonic - does not have unique symptoms making it hard to differentiate from other disorders.

Symptom overlap - Ellason and Ross (1995) - issue for validity

Ellason and Ross (1995) - found people with DID (Dissociative Identity Disorder) have more first rank symptoms than schizophrenics.

-Culture bias - issue for validiy

• SZ is statistically more frequent in the UK and elsewhere in people from African - Caribbean descent than other groups.

• Africa and the West indies rates of sz arent that high, this is not due to genetic vulnerability.

• Pinto & Jones (2008) → estimate it was 9 time higher in British people of African Caribbean origin than white British group.

• This could be to do with psychosocial factors and stressors - poor housing, unemployment and social isolation.

• However the main explanation is because most British psychiatrists are white and biased when judging people from other ethnic backgrounds.

• The clinician might not speak the same language as the person they are attempting to diagnose or certain things can be ‘lost in translation’, e.g in many cultures it is normal to see and speak to recently deceased love ones, but this can be misdiagnosed as a symptom of sz, leading to inappropriate diagnosis. 

• This suggests a lack of validity in diagnosing sz cross-culturally, where ethnic differences in symptom expression are over looked. 

• Escobar (2012) → argued while psychiatrists over-interpret symptoms of African-Caribbean people due to cultural differences in language and mannerisms and difficulty relating to cultural norms that differ to their own. 

Gender bias - issue for validity

• Men have been more commonly diagnosed with sz than women and it tends to be diagnosed earlier in men (18-25 years), compared to 25-35 years for females. This is important as women may not be receiving treatment. 

• Loring and Powell (1988) → randomly selected 290 male and female psychiatrists to read 2 case articles of patients' behaviour and then asked them to offer their judgement on these individuals using standard diagnosis criteria.

  When patients were described as ‘male’ or no information was given about their gender, 56% were given a diagnosis of sz. However, When patients were described as ‘female’, only 20% were given a diagnosis of sz.

• This gender bias did not appear to be evident among the female psychiatrists. This suggests that diagnosis is influenced by gender and gender of clinician. 

• This is because female patients perhaps function better than male, being more likely to work and have good family relationships (have support systems) whilst living with the illness. 

• Therefore explains why some women have not been diagnosed, when men with similar symptoms have.

• Symptoms may be masked by better inter-personal functioning or the quality of interpersonal functioning, may make the case seem too mild to warrant diagnosis, women also recover better. 

• It could also be due to stereotypical beliefs about gender. Goldstein (1993) → suggests males are more likely to be involuntarily committed when they show mild signs of sz due to the risk of socially deviant behaviour. Females on the other hand are more likely to be voluntary patients as they are more likely to seek help. 

• Another possible cause for higher rates in males is that they have higher rates of substance abuse. 

Bentall et all (1988) - validity

Comprehensive review of symptoms (aetiology), prognosis (outcomes) and treatment concluded that sz is not useful scientific category. 

Suggested that it is invalid to categorise someone as schizophrenic, we should just treat the individuals symptoms.

The label of sz has a long lasting negative effect which seems unfair when the diagnoses are made with little evidence of validity.