Measuring and Validity

Types of validity

Subjective

  • Face validity: “how do experts judge the measurement instrument?”

  • Content validity: “does the measurement measure all aspects of the construct?”

Emipirical

  • Convergent validity: “do the measurements of this instrument and the measurements of another instrument that intends to measure the same concept coincide?”

  • Divergent validity: “do measurements of this instrument not coincide with measurements that intend to measure something different?”

  • Criterion validity: “suppose we know this variable is related to a different variable. Do we find this relationship when we use the measurements of the current measurement instrument?”

  • Correlation coefficient is another tool to measure validity

Errors

Systematic error: error in the procedure of the experiment, affects validity

Random error: affects reliability

Reliability

Test-retest reliability: measurements with the same instrument should be consistent over multiple tests. Used for constructs that are relatively stable over time. r>0.5 means good test-retest reliability

Inter-rater reliability: measurements with the same instrument should be consistent when different researchers use it. r>0.7 means good inter-rater reliability

Internal reliability: do participants give similar answers to different questions about the same concept? Useful for surveys. Measured with cronbach’s alpha

  • Below 0.7 - low internal validity

  • Above 0.8 - high internal validity

Surveys

Survey modes: face to face (CAPI), mail (post), telephone (CATI), internet (email, websites), mixed-mode, panel surveys

  • Mixed mode: one mode for data collection, another for reminders/follow up OR some questions in one mode, some in another (eg sensitive questions not part of a face-to-face interview)

  • Panel surveys: interview respondents over time, usually with the same content

    • Advantages: can study a long-term effects, mitigates influences of age etc

    • Disadvantages: attrition (drop-out, consecutive wave non response), learning effects if the questions stay the same → may cause a biased response

  • Differences between the modes:

    • Degree of interviewer involvement, degree of interaction with the respondent, degree of privacy

    • Channels of communication (what media can I use in the survey with the mode I am using? visual/audio/paper), technology use

Coding

Goal: create a variable (scale score) that indicates the overall score on a survey, eg severity of PTSD

  • Option 1: sum of all item scores. May cause problems when there are missing data points

  • Option 2: average all of the item scores using reversely coded items when appropriate