pubpol quiz 2 - quantitative & qualitative research methods

controls

didn’t receive treatment/intervention

treated

received treatment/intervention

what are the 3 types of research design?

non-experimental/observational, quasi-experimental, and experimental

non-experimental/observational

measure impact of policy for participants ONLY

no comparison group

counterfactual is unknown - we don’t know what would happen in the absence of the policy OR control group doesn’t control for bias

haven’t considered confounders

regression adjustment

mechanically adjust by making treated and control groups statistically equivalent with respect to an observed confounder

change in outcome =

policy treatment + observed confounders

quasi-experimental

measure impact for participants and non-participants

no randomized assignment, attempt to make assignment into T + C as good as random

regression discontinuity design

assign a T + C based on proximity to eligibility cutoff

treated participants right above and below the cutoff do NOT differ on any other observable characteristics

pros of regression discontinuity design

stronger internal validity compared to non-experimental approaches

more confidence in accounting for unobserved confounders

cons of regression discontinuity design

weaker external validity bc the results may not reflect the broader population

difference-in-difference analysis

compare the changes in outcome over time btw a population enrolled in a program vs a population not enrolled in the program

parallel trends

the difference between what actually happened and the counterfactual is interpreted as the effect of the intervention

what are the pros of parallel trends?

“differences out” time-constant difference between T + C

accounts for general trends

assumes that the only difference over time btw T + C groups is the intervention

interrupted time series

tracking the outcome for T over a long-term time period before AND after the point of intervention to assess its effects

what are the pros of the interrupted time series?

doesn’t need non-participant group

strong adjustment for individual confounders

examine both levels

no other observed changes occurring around the same time as the intervention

experimental

control and treatment groups

randomized assignment

field experiments

research done in a real world environment
e.g. classic JTPA field experiment done in the 1980s

3 approaches in qualitative analysis

1. focus on subjective experiences, understandings, and meanings of policies

2. emphasizes local knowledge - experiences and understandings of a policy based on a lived experience

3. helps us answer how and why policies have the effects they do

qualitative/in-depth interviews

intensive, in-depth interviews with open-ended questions

small # of respondents to explore their opinions and experiences of a policy/program

more contextual evidence and holistic approach

allows for surprises/discoveries the analyst may not have anticipated

what is an example of a question that could be aksed in a qualitative/in-depth interview?

tell me what it’s been like trying to get the healthcare you need for you and your family

focus groups

open-ended interviews with small groups of people

encourages interaction and open dialogue

bring tgt diff stakeholders and present their experiences

what is an example of a question that could be asked in a focus group?

what were the key barriers to fair housing in suburban jurisdiction?

document content analysis

analyze existing information in textual documents e.g. hearing testimonies, meeting minutes

focus on the meanings conveyed via words, topics, and concepts in texts or images

ethnography and observation

study daily life in a particular setting over an extended period of time

understand interactions and experiences in their natural setting

gather data thru direct observation

intensive process

what are examples of the ethnography and observation process?

observing doctors’ offices, classrooms, policing practices

anecdote

information collected in an informal or ad-hoc manner

presented without analysis

cherry-picked to support a predetermined position

good for generating ideas and humanizing an issue

research

information collected in a systematic and transparent manner

relies on the testimonies and experiences of a well-defined group

data collected in an objective manner

what are the 4 criteria to assess the rigour of qualitative research?

1. self-awareness

2. positionality

3. transparency

4. level of engagement

self-awareness

the policy analyst acknowledges they are engaged in interpretive, subjective acts

they recognize that a policy will affect each person differently bc of their unique lived experiences

positionality

doesn’t assume they know what a policy action or decision means for everyone

identifies local knowledge as expertise

seeks out possible varying interpretations of policies across different groups or stakeholders

transparency

level of detail in the sample, data collection and reproducibility

level of engagement

analysts who have a more direct and sustained engagement with the affected group will get better qualitative insight

deeper understanding of perceptions, meanings and motivations

identify contradictions and complexities

data and evidence are more concrete