Critiquing and Reliability

Module 4: Claims and Validities

Critique

as a consumer, we assess whether we can trust someones claims (deciding whether we trust is called critiquing) 

critiquing research involves understanding how constructs are measured

Operational definitions

defining the variable in terms of precisely how it is to be measures

• turn an abstract concept into something measureable

Ways to Measure- Self Report

limitations: an individual’s perception, experience, attitudes, etc.

careful what claims you make

Ways to Measure- Observational

watch/record actual behavior

Ways to Measure- Psysiological

biological data

Measuring Constructs

Challenges

• contrasts are not directly measurable

  • conceptual definition

• many ways to make the measurable

  • operational definition- measure

• Must be good measures

How do we know if our operational definition is “good”?

construct validity and construct reliability

To Critique claims

1. decide what type of claim it is (3 types)

2. Examine 3 types of reliability

3. Examine 4 types of VALIDITY

3 types of claims

• Frequency claims

• Association claims

• Casual claims

• Bogus claims

Frequency claims

descriptive only; does not explain “why”

• ex. “77% of university students report experiencing some kind of psychological distress, either moderate or severe”

Association claims

correlations and relationships only; does not explain “why”

• “there is a strong, positive correlation between high stress levels and poor sleep quality”

Causal claims

explain cause and effect; CAN explain “why”

• ex. “ good sleep hygiene leads to more efficient learning and higher scores”

**best but hard to achieve**

Bogus claims

not technically a “type” of claim but something to be aware of as a consumer

• detecting bogus claims requires critiquing reliability and validity of the research

Reliability

a measure is reliable if it is consistent

• a measure yields the same result repeatedly

  • test- retest reliability

  • internal consistency

  • interrater reliability

Validity

a measure is VALID if it is CONSISTENT and CORRECT

• it is accurate

  • face validity

  • content validity

  • criterion validity

  • discriminant validity

Types of Reliability

1. test-retest reliability

2. internal consistency

3. interrater reliability