9/21 Reliability and Validity
Reliability - measuring the same thing multiple times and getting the same result consistently
Test-retest reliability - stability over time. Administer the same test twice to the same group after a certain interval of time
Split-half reliability - internal consistency. All of the items are randomly divided into two groups and then a correlation is calculated between two sets of scores
Cronbach’s alpha reliability - internal consistency. How closely related a set of items are as a group.
The most popular reliability test
Intercoder reliability - equivalency. Measurements of content analysis, or coders.
Degree of agreement among two or more coders when the same phenomenon is evaluated
How to improve reliability?
Use measures others have used in the past; consult literature reviews
Avoid ambiguous and vague wording when constructing items for answering
Use multiple items; a 5-item test is better than a 2-item test
Validity - how well an instrument measures what it’s intended to measure.
Are we actually measuring the concept we defined? Is it a valid way to measure that concept?
Face validity - subjective judgement, examining specific questions. Whether it looks right or not
Criteria validity - how accurately a new measure can predict a well-accepted criterion or previously validated concept
How to improve validity?
Use measures others have used in the past; consult literature reviews
Recognize relationship between reliability and validity
Reliability is a necessary condition for validity - cannot be valid if not reliable.
Reliability is not sufficient for validity. Can be reliable but completely wrong!
9/23 - Survey Design
Survey - collecting data through self-reporting
Pros to surveys:
Reasonable cost for a large amount of data
Collect + examine many variables
Data gathering relatively easy
Not constrained by geographic boundaries
Cons:
Hard to establish causality
Difficulty to recall previously-felt emotions and attitudes
Sensitive wordings can cause issues
Inaccurate responses possible - misremembering, social desirability
Types of surveys: (some can be either self-administered or monitored)
mail surveys
Pros - reach anyone, anywhere!
Cons - lower response rate, potential to misunderstand questions and no help
online surveys
Pros - lower cost, convenient, flexible (with audio and video)
Cons - sample restricted to those with internet access
phone surveys
Pros - higher accuracy, moderate cost
Cons - question type is limited (no visuals), questions must be SIMPLE
Face-to-face surveys
Pros - highest response rate and best response quality
Cons - highest cost, staff’s appearance or demeanor can influence responses (ex. Likert scale)
Types of Questions
Open-ended - respondents can freely answer in their own words.
Pros - respondent gets a sense of control, researcher might get unexpected results
Cons - answers must be coded, and are subjective and time-consuming
Close-ended - options are provided for respondents to choose from.
Pros - easy to codify
Cons - No nuance, can force meaningless responses, details unknown
How to properly construct survey questions?
NO!
Keep it clear with concise questions and simple language.
Don’t ask two concepts in one questions (aka double barrel questions)
No double negatives
Avoid biased language and leading words
Avoid leading questions (assumed or implies a specific answer)
Avoid asking for extremely detailed information
Avoid sensitive questions
YES!
Provide instructions for answering questions
Keep research purpose in mind
Keep ordering of questions in mind
Think about logical flow
Structure of survey:
1.) Instructions
2.) Easy items at the beginning
3.) Main survey
4.) Final section