Surveys

Mail Surveys

Advantages

▪ Ability to reach large number of people in a short amount of time

▪ Low cost with minimum staff needed (postage)

▪ Possibility of response bias lower than other methods

▪ Same questions delivered in same format to everyone

Disadvantages

▪ Lowest response rate

▪ Inability to clarify any confusion

▪ Incentives to get return

▪ Appearance of questionnaire is important

Electronic Surveys

Advantages

▪ Reduced response time

▪ Low cost of materials

▪ Ease of data collection

▪ Flexibility in the design and format of the questionnaire

▪ Control over the administration

▪ Technology – data can be entered directly into a spread sheet

Disadvantages

▪ Access to the internet and comfort with computer

▪ Obtaining email addresses

▪ Response rate is mixed; no control over answers

▪ Security

▪ Sample bias

Telephone interviews

Advantages

▪Easy

▪Moderate cost

▪Standardized questionnaire

Disadvantages

▪Wireless houses

▪Requires training

▪Bias from clarification

▪No visual data collection

Face to face interviews 

Advantages

▪ Allows interviewer to develop rapport with population

▪ Flexibility and visual cues all to gain more complete data

▪ Must train interviewer

▪ Important to establish and FOLLOW procedures

▪ Response rate is usually high

Disadvantages

▪ Time consuming- requires travel

▪ Small number of respondents

▪ More expensive

▪ Require more staff and training

▪ Variation in interviews

▪ differences between interviewers all can influence the results

Types of questions and best practices

Test your questions 

▪Validity

▪Face-questions actually measure what you want

▪Content-covering entire concepts of meaning

▪Criterion-scores relate to measure 

▪Construct-measuring all aspects

▪Reliability

▪Test-retest

▪Inter-item

consistency

▪Comparison

▪Interrater

reliability

Get help 

Making your own

Wording:

▪ Do you believe you should brush 4x a day? vs. How often should you brush your teeth?

▪ Leading questions: Most people seek counseling when they have addiction. Do you believe you should seek counseling?

▪ Double-barreled: Do you brush and floss your teeth?

▪ Jargon: What nutritional benefits are gained by eating healthy?

▪ Specificity: How helpful was this nutrition class? vs. As a result of this class, how prepared are you to assess your food habits?

Types of questions to avoid

▪ Leading questions -Questions worded in a way that sways the reading to one side of the argument

▪ Loaded questions -Written in a way that forces the respondent into an answer that does not accurately reflect his/her opinion or situation

▪ Double-barreled questions-Written to cause the respondents to answer 2 questions at once

▪ Question has two subjects

▪ Using absolutes

▪ Typically yes/no questions with wording “always”, “never”, “every” “all”

▪ Use the respondents language-Use clear, concise, and uncomplicated language

Sampling

▪Do you need to sample ALL of the population or just a sample?

▪Random

▪Systematic

▪Stratified

▪Convenience

▪Snowball

▪Quota

▪Purposive

▪Cluster

Developing a draft

▪Keep your respondents in mind

▪Are the questions applicable

▪Easy to answer

▪Is the first question interesting

▪Put sensitive questions in the middle

▪Demographic questions at the end

Pilot test and final survey

▪Have your respondents take your survey

▪If this is not possible, have others take it and provide feedback

▪Revise

▪Creating the final survey consider:

▪Length

▪Readability

▪Few open-ended questions

▪Good directions

Pilot testing

Types continued:

Cognitive testing = testing with the priority population in small groups with feedback

Pre-pilots = five or six members of the population to assess quality of the materials

Pilot test = actual implementation of the instrument

• Can check validity and reliability if enough subjects are used

• 10% of study population FYI

Types of pilot testing:

▪ Preliminary review

▪ Experts or colleagues review it

▪ Title

▪ Statement

▪ Directions

▪ Order of questions

▪ Questions

▪ Length

▪ Method of returning

What are we looking for in the results

patterns

terms

themes

ideas 

Responses 

▪ What is it you are looking for?

▪ Assessing the population may be different than evaluating the program.

▪ Identifying ages of participants to plan a program vs.

▪ How well each age group performed on the post-test

▪ Ratio vs. ordinal

▪ How old are you = ratio

▪ Which age range best describes you (15-24, 25-34, 35-44, etc.) = ordinal

▪ Close-ended or fixed responses are better for analysis

▪ Tell us about your experiences vs.

▪ Which of the following best describes your experience (fixed response –Likert scale)