9/21 Reliability and Validity

Reliability - measuring the same thing multiple times and getting the same result consistently

  • Test-retest reliability - stability over time. Administer the same test twice to the same group after a certain interval of time

  • Split-half reliability - internal consistency. All of the items are randomly divided into two groups and then a correlation is calculated between two sets of scores

  • Cronbach’s alpha reliability - internal consistency. How closely related a set of items are as a group.

    • The most popular reliability test

  • Intercoder reliability - equivalency. Measurements of content analysis, or coders.

    • Degree of agreement among two or more coders when the same phenomenon is evaluated


How to improve reliability?

  • Use measures others have used in the past; consult literature reviews

  • Avoid ambiguous and vague wording when constructing items for answering

  • Use multiple items; a 5-item test is better than a 2-item test


Validity - how well an instrument measures what it’s intended to measure.

  • Are we actually measuring the concept we defined? Is it a valid way to measure that concept?

  • Face validity - subjective judgement, examining specific questions. Whether it looks right or not

  • Criteria validity - how accurately a new measure can predict a well-accepted criterion or previously validated concept


How to improve validity?

  • Use measures others have used in the past; consult literature reviews

  • Recognize relationship between reliability and validity


Reliability is a necessary condition for validity - cannot be valid if not reliable.

Reliability is not sufficient for validity. Can be reliable but completely wrong!


9/23 - Survey Design

Survey - collecting data through self-reporting

Pros to surveys:

  • Reasonable cost for a large amount of data

  • Collect + examine many variables

  • Data gathering relatively easy

  • Not constrained by geographic boundaries

Cons:

  • Hard to establish causality

  • Difficulty to recall previously-felt emotions and attitudes

  • Sensitive wordings can cause issues

  • Inaccurate responses possible - misremembering, social desirability


Types of surveys: (some can be either self-administered or monitored)

  • mail surveys

    • Pros - reach anyone, anywhere!

    • Cons - lower response rate, potential to misunderstand questions and no help

  • online surveys

    • Pros - lower cost, convenient, flexible (with audio and video)

    • Cons - sample restricted to those with internet access

  • phone surveys

    • Pros - higher accuracy, moderate cost

    • Cons - question type is limited (no visuals), questions must be SIMPLE

  • Face-to-face surveys

    • Pros - highest response rate and best response quality

    • Cons - highest cost, staff’s appearance or demeanor can influence responses (ex. Likert scale)


Types of Questions

Open-ended - respondents can freely answer in their own words.

  • Pros - respondent gets a sense of control, researcher might get unexpected results

  • Cons - answers must be coded, and are subjective and time-consuming

Close-ended - options are provided for respondents to choose from.

  • Pros - easy to codify

  • Cons - No nuance, can force meaningless responses, details unknown


How to properly construct survey questions?

    NO!

  • Keep it clear with concise questions and simple language.

  • Don’t ask two concepts in one questions (aka double barrel questions)

  • No double negatives

  • Avoid biased language and leading words

  • Avoid leading questions (assumed or implies a specific answer)

  • Avoid asking for extremely detailed information

  • Avoid sensitive questions


    YES!

  • Provide instructions for answering questions

  • Keep research purpose in mind

  • Keep ordering of questions in mind

  • Think about logical flow


Structure of survey:

1.) Instructions

2.) Easy items at the beginning

3.) Main survey

4.) Final section