Midterm Political Surveyss

Survey quality

Defining “quality” is actually difficult 

• Often assume that accuracy and reliability are only dimensions of quality 

• Accurate: bias, validity, difference between estimate and true value- have a way to indicate what the true value is sometimes by comparing, sometimes not at all and have to guess- we do this to think about where there could be a source of error

• Reliable: variance, constitency/stability in estimate (in repeated trials)

reliability

how consistent are our results, how stable 

Might be instability based on the ways you are writing the question

an unreliable question- being vague, you then get a huge variety of answers, can be on different ends, therefore answers arent conscise 

dimensions of survey quality

Survey quality should be defined as “fitness for use”- need to take an all inclusive view of survey

• Accurate and reliable estimates

• Relevance: meets user needs

• Timeliness: available when needed 

• Acessiblity: results interpretable and in appropriate format for use 

• Comparability: can be compared across space and time if needed 

• Cost- if survey is expensive may not be worth doing 

sampling error

(mistakes in who is taking your survey)- error you get because you didnt take a census 

Must consider sampling and non-sampling error 

Accuracy

the difference between your actual answer and the estimate answer  

Probability sample

Random selection from sample frame 

 every unit in the sample frame has a known (nonzero) probability of selection •

Unit nonresponse

selected units from your sample frame that are not in your set of respondants

you want them in particular (particular set of people) but they dont respond

Reasons for unit nonresponse 

Unable 

• Individuals lack tech or language skills to complete a survey 

Unavailable 

• New technologies make contact more difficult 

• Mobile lives make likelihood of contact rarer

Unwilling 

• Concerns about privacy 

• Survey overboard 

• Dislike pollsters or sponsors 

Response rate

The number of complete interviews divided by the number of eligible units in the sample

Components of unit nonresponse 

 Refusals (reached but declined or breakoff) • 

Non-contact (never reached) • 

Household never at home or telephone never answered. • 

Mail surveys returned not delivered. •

 Failure to locate the sampled person/household. •

 Inability to participate • Language, literacy, etc.

Calculating response rates 

Calculated differently for different purposes •

 Not always meaningful from one survey to another 

• Outcome Rates: Response Rates, Cooperation Rates, Refusal Rates, Contact Rates •

 Sample Frame, Coverage, Respondent Selection, etc., can all impact response rates • 

Often, the best way to increase a response rate is to “cut corners” on sample frame, stratification, or respondent selection 

calculating nonprobability samples

A “Response Rate” should not be calculated for nonprobability samples 

• No frame from which to be ‘‘selected’’ means there is no definable denominator

 Online probability samples 

• Need a cumulative response rate to account for nonresponse during recruitment into the panel

Decline in response Rates

Explanations for decline in response • 

Declining contact rates • Caller ID, Answering machines, Cell Phones, Multiple telephone numbers, unlisted numbers.

 • Gated communities, limited access apartment buildings.

 • Privacy expectations Explanations for decline in response 

• Increased refusals •

 Time constraints (“too busy”) •

 Lessened sense of civic responsibility or sense of reciprocity

 • Too many survey requests 

• Concerns about safety, fraud, and misrepresentation

 • Human Subjects requirements 

 A low response rate indicates potential for nonresponse bias; it is not itself a metric of bias

Not a strong empirical relationship between response rate and nonresponse bias

Item nonresponse 

Someone starts survey doenst finish, or skips question they dont wanna answer 

For sensitive questions, people not only lie but skip question

Include a dk (dont know) option will reduce ite nonresponse, but might cause other problems

The greater the burden of question, more likely people skip it (ex: open ended questions) 

More likely to skip questions in surveys without interviewer (web/mail) 

Respondent fatigue

more nonresponse later in survey

What to do about non response

Data collection design: incentives, nonresponse follow up, etc…

Adaptive design- reduce bias during implementation (keep track and modify things if necessary)- will change questions based on your answers

Use post survey adjustments 

Data collection 

Sponsorship

University sponsorship often helps over commercial organizations 

Government sponsorhsup usually the best

Mutli mode survey

Combine web and mail survey

Follow up mail survey with telephone contact

best way to deal with missing data:

dont have any

Data collection pt 2

Reduce respondent burden 

Shorter survey instruments 

May be more important in mail surveys 

Advance notification

Incentives 

Confidentiality pledge 

Interviewer training

Missing Data 

When we have people who didnt take survey or skipped questions 

Missingness mechanism 

Mssiing completel at random (MCAR)

missingness is random

Completely at random, we dont know anything 

Rare unless part of design

People who r skipping your survey qeuestions doing so at random 

Missing at random (MAR)

missingness depends on observed but not unobserved variable 

Conditional on covariates, missingness is random 

• Women are more likely to answer vote Q

• Based on a pattern that you see ^, you know why they skip it 

• People skipping questions based on an observable pattern- things you can see (data you have, their age, sex, etc..) 

Missing Not at Random (MNAR)

missingness depends on unobserved variables 

E.g., if those with hig income dont answer question 

This is typically the world we are in 

Common missing data treatments 

Whatever you do, you are doing something 

• Complete-case analysis •

 Only analyze results from people who answered the questions •

 Imputation of missing data • 

• Mean Imputation • 

• Decision rules (e.g., code as nonvoter someone not registered) 

• Conditional Distribution Imputation (matching)

Post-survey data adjustments 

Weighting • 

• Based on external information; typically demographics from government survey • 

Imputation • 

• Typically used for item response

Sampling error (probability sample) 

• Random variability in sample estimates that arises because you took a sample and not a census 

Non-sampling error 

Estimation error that arises from sources other than random variation 

non-response 

undercoverage of survey 

• poorly-trained interviewers 

• non-truthful answers • 

non-probability sampling • 

This type of error is a bias

nonresponse bias

when people dont respond because of who theyre like

Types of nonprobability samples • 

Convenience sampling- 

Opportunistic sampling. Take whatever is available and convenient. •

 Snow ball sampling- 

Selection of additional respondents based on referrals from initial respondents.

Convenient samples r convenient

Purposive sampling- 

Based on judgement of researchers. The researcher determines and includes those items in the sample considered more relevant for the research project 

 Quota sampling- 

 A specific number (quota) of observations is required in final sample to match population estimates 

E.g., 50% women, 50% men 

Population is divided into subpopulations (stratas) but quotas are not filled using random selection from sample frame 

Typically use known population estimates (e.g. census distributions) for quotas

Nonprobability sampling- no samplin frame- dont try to calculate response rate

Public opinion

the views of a group of people 

Definition of public opinion deepnds on how you measure it 

Measuring it with polls has some problems 

• Are we talking about peoples private or public opinions 

• Are we asking structured vs unstructured questions?

Characteristics of public opinion

Dimensions of Opinion 

Direction 

Extremity

Intensity 

Stability 

Informational content 

Variation among groups 

Ambivalence

Direction

What side does opinion favor? 

Example: self identified ideaology 

When it comes to politics, do you usually think of yourself as a liberal, a conservative, a moderate, or haven’t you thought much about this?

Conservative 

Moderate 

Liberal 

Dont know 

Extemity 

Does opinion fall toward the end of the range of possible opinions, or towards the middle? 

Example: ideology 

Intensity

Personal importance 

Definition: strength of feeling about object or opinion (how much you care) 

Sample question: how important is that issue to you personally 

Salience 

Societal importance 

Defintion: how important is the issue, or how does it rank on the agenda 

Example: most important problem

Stability

Is overall opinion changing? If so what direction? 

If we ask an individual the same question at two points in time, do we get the same answer? 

Examples: 

Opinions about gay marriage 

Identification with a political party 

Ambivalence

Defintion: opinion about an issue is characterized by controlling underlying values 

Examples: attitudes toward death penalty and abortion

Characteristics of public opinion: ignorance

Most people dont know much,

Dont overestimate the publics knowledge, but dont underestimate their intelliengece 

Political knowledge measurement 

People will offer an opinion even when they do not know the topic asked in a question 

Open-ended vs closed questions 

• People might skip open-ended question for other reason 

• Multiple choice format means some people get ir right by guessing