Week 2: Defining Policy Problems and Research Questions - Notes

Week 2: Defining Policy Problems and Research Questions

Learning Objectives for Week 2

  • Understand the process for basic social science research.

  • Understand how the deductive research process is used in social science research.

  • Understand the steps involved in defining research questions.

  • Understand how research questions help define policy problems (and the difference between research questions and policy problems).

  • Understand the process for defining policy problems.

Overview of the Class Tonight

Lecture

  • Overview of The Research Process.

  • Deductive Approach to Social Science Research.

  • Defining Research Questions and using Research to Help Define Policy Problems.

  • BREAK.

Lecture Continued

  • Defining Policy Problems.

  • Small Group Discussions.

  • Instructions for Assignment #1: Policy Problem Definition - Due Week 3 (Sept. 25).

  • Readings for Wk. #3: Qualitative Research Methods, Setting Criteria and Identifying Alternatives.

Readings for Week #2

Course Text

  • Section 3.1 – Identifying and Collecting Data (Pages 69-89).

  • Section 4.1 – Identifying and Defining Problems, and Section 4.2 – Developing the Problem Statement (Pages 141-146).

British Columbia/Yukon Open Authoring Platform. Research Methods for the Social Sciences

  • Section 1.1 - What are Research Methods?

  • Section 1.2 - The Process of Undertaking Research.

  • Section 1.3 - Where Do Research Ideas Come From?

  • Section 1.6 - Inductive Approaches to Research.

  • Section 1.7 - Deductive Approaches to Research.

  • Section 3.1 – Normative Versus Empirical Statements.

  • Section 3.2 – Exploration, Description, Explanation.

  • Section 3.3 – Developing a Researchable Research Question.

  • Section 3.4 - Hypotheses.

Overview of The Research Process

  • Choose a topic.

  • Review the literature (past research).

  • Formulate the problem (find the gap in past research).

  • Develop a research question.

  • Choose and organize the research design.

  • Gather the data.

  • Analyze the data.

  • Interpret the data.

  • Communicate the findings.

Four Basic Rules for Social Science Research

  1. The goal is inference:

    • Describing phenomena, behavior, and events is an important first step.

    • Inference is a conclusion formed because of evidence, allowing analysis to uncover aspects not directly observed.

    • Example:

      • Descriptive: Some arid countries have water shortages, so expect similar conditions in analogous countries.

      • Causal: Arid countries have less water because they are hot.

  2. Procedures are public:

    • Research methods must be clear and replicable.

    • Other researchers should be able to recreate the study and reach the same conclusions.

  3. Conclusions are uncertain:

    • The interpretation of patterns in data is inherently uncertain.

    • In social science research, there are no final answers.

  4. The method itself is an important part of the content:

    • Methods are valuable due to their transferability.

    • Documentation of methods is a primary component of social science research, in addition to the conclusions.

Inductive Approach in Social Science Research

  • Starts with data.

  • Looks for patterns within the data.

  • Develops a theory to explain these patterns.

  • Moves "from data to theory" (from specific observations to general explanations).

Deductive Approach in Social Science Research

  • Opposite order of the inductive approach.

  • Starts with a theory (hypothesis).

  • Analyzes data to "test" the theory.

  • Confirms support or non-support for the hypothesis.

  • The theory must be falsifiable (meaning it can be proven wrong).

  • It is not a concern to be wrong; confirming non-support for a hypothesis is also valuable.

  • Most social science research follows a deductive approach.

Hypothesis

  • Conjectures put forth to explain the relationship between variables.

Variables

  • Measurable characteristics that can take on more than one value.

  • Unidimensional: e.g., height, weight.

  • Multidimensional (complex): e.g., well-being, happiness.

Key Considerations

  • Validity of the data:

    • Do the instruments accurately measure what they intend to measure?

    • Do the results truly represent what they are supposed to measure?

  • Reliability of the data:

    • Can the results be reproduced under the same conditions?

Examples of Deductive Social Science Research

Example – Why Do So Few People Vote?

  • Theory: People of low socioeconomic status are less prone to vote.

  • Hypothesis: income/education
    ightarrow voting (meaning income/education influences voting).

  • Dependent variable: "Did you vote in the last election (yes/no)".

  • Independent variables:

    • Income.

    • Education.

    • Occupation.

    • Parent Income.

    • Parent education.

    • Parent occupation.

    • Average education in the community.

Example – Why Do Some People Drive to Work?

  • Theory: Transportation options are better in high-density areas.

  • Hypothesis: As population density
    ightarrow share of workers who drive
    ightarrow (meaning as pop. density increases, the share of workers who drive decreases).

  • Dependent Variable: % of population who drive to work.

  • Independent Variables:

    • Population Density.

    • Other (e.g., Income?).

  • Data (Vancouver, 2016):

    • Arbutus Ridge: 41.3 people/ha, 61.9% drive.

    • City Overall: 55.0 people/ha, 45.4% drive.

    • Downtown: 167.6 people/ha, 30.4% drive.

  • Observation: Data suggests an inverse relationship: as population density increases, the share of workers who drive decreases.

Example – Why Are There No Kids Downtown?

  • Theory: The housing stock downtown is not attractive to families with kids.

  • Hypothesis: As % of homes in high-rise apartments
    ightarrow % of population < age 19
    ightarrow (meaning as % of apartments increases, % of population < 19 decreases).

  • Dependent Variable: % of population in local area < age 19.

  • Independent Variables:

    • % of homes in apartments.

    • Other?

  • Data (Vancouver, 2016):

    • Area: Downtown; % of homes in high-rise apartments: 93.9%; % of population < 19: 8.4%.

    • Area: City Overall; % of homes in high-rise apartments: 29.3%; % of population < 19: 15.6%.

    • Area: Arbutus Ridge; % of homes in high-rise apartments: 23.7%; % of population < 19: 21.2%.

  • Initial Observation: Data suggests an inverse relationship: as the share of high-rises goes down, the share of the population < age 19 goes up.

  • Further Data (Downtown 1996-2016):

    • The prevalence of kids downtown has actually increased despite an increasing share of high-rise housing.

    • Total Downtown Population: increased from 17,405 in 1996 to 62,030 in 2016.

    • Share < age 19: increased from 5.5% in 1996 to 8.4% in 2016.

    • Number of Kids: increased from 957 in 1996 to 5,211 in 2016.

    • Comparison: In Arbutus Ridge, 21.2% of the population is < age 19, but this only represents 3,242 kids, fewer than downtown despite the lower percentage.

  • Conclusion: This challenges the initial hypothesis and indicates other factors might be at play.

Data Sources and Open Data Examples (Vancouver)

  • CoV Open Data – Parks Inventory Example:

    • Contains Park Name, Park Size, Park Location (Neighbourhood).

    • Possible theory to test: Areas with more park space per person are more attractive to families with children.

    • Method: Calculate total park space/person per neighborhood; compare with % of population < age 19.

  • CoV Storefront Report Example:

    • Provides data on every Business Improvement Area (BIA): Storefront Vacancy Rate, Population within 1 KM, Retail Mix, Sales and visitation data.

    • Possible theory to test: Areas with low population density have higher storefront vacancy rates.

Defining Research Questions

Normative vs. Empirical Statements

  • Normative Statements: Express value judgments or opinions (e.g., "Vancouver is one of the most livable cities in the world").

  • Empirical Statements: Are factual, objective, and can be verified (e.g., "In the 2023 Global Liveability Index, Vancouver ranked 4th").

  • Social science research is most concerned with empirical statements.

Types of Research

  • Exploratory Research: Used to test the feasibility of conducting a more extensive study.

  • Descriptive Research: Used to describe or define a particular phenomenon.

  • Explanatory Research: Used to answer "why" questions.

Key Characteristics of a Good Research Question

  • Written in the form of a question.

  • Well-focused.

  • Cannot be answered with a simple "yes" or "no".

  • Has more than one plausible answer.

  • Considers relationships among multiple concepts.

Contribution to Literature

  • Choose an untested hypothesis.

  • Choose an accepted hypothesis that you suspect may be false.

  • Test unquestioned assumptions.

  • Re-test an accepted hypothesis with updated/improved data.

  • A proposed topic that cannot be refined into a specific research project permitting valid inference should be modified or abandoned.

Problematic vs. Improved Research Questions

  • Problematic: "How many residents use the community centre?" (Too specific, simple statistic).

  • Problematic: "How does community centre use impact community well-being?" (Too broad, difficult to test empirically).

  • Improved: "Does resident age impact likelihood of community centre use?"

Identifying and Gathering Data

  • Analysts often lack time to become subject matter experts; rely on existing expertise.

  • Start with existing data; creating original data is time-consuming.

  • Consult colleagues, advocacy groups, and subject matter experts.

Sources of Data

  • Federal Government: Rental Market Survey (CMHC), Census (Statistics Canada), Canadian Survey on Business Conditions (Statistics Canada), Labour Force Survey (Statistics Canada).

  • Provincial Government: BC Labour Market Outlook (Work BC).

  • Local Government: City of Vancouver Open Data Catalogue.

  • Other Public Bodies: Trip Diary Survey (Translink), Generalized Land Use by Parcel (Metro Vancouver), BC Assessment property databases.

  • Real Estate Interests: Monthly Market Report (Real Estate Board of Metro Vancouver), Office Market Report (Colliers International), Co-Star property databases.

  • Libraries: Librarians are experts on databases and data sources.

  • Media and Public Opinion Research Organizations.

  • Literature Review: Almost all topics have been investigated. Academic journals provide data and sources. Ensure data is current and transferable.

  • Surveys, Interviews, Focus Groups.

  • Observation: (e.g., use of public space, CoV Storefront Inventory).

Using Research to Help Define Policy Problems

Research Questions Can:

  • Define the "gap" or "target" that needs to be met for a desired outcome.

  • Improve understanding of the situation and the need for government intervention.

  • Define the likely impacts of alternative actions.

Defining Policy Problems

Stakeholder Analysis

  • Understand the positions and influence of various individuals and groups concerned about the issue, not just the client.

  • Identify: Who is concerned? Why? What are their stakes and power to affect policy?

  • Recommendations should consider actions for the client and actions the client could advocate from others.

Using Numbers to Define the Problem

  • Quantify the problem/gap.

  • Justify why government action is warranted.

  • Project what will happen if no action is taken.

  • Identify available data sources to quantify the problem and potential impacts of alternatives.

Example – Teenage Automobile Accidents

  • Possible Dependent Variables:

    • Too many accidents.

    • Too many injuries/deaths.

  • Possible Independent Variables:

    • Minimum age to drive is too low?

    • Inadequate driver education courses?

    • Inadequate driver testing procedures?

    • Unsafe older vehicles operated by teenagers?

    • Teenagers driving while intoxicated?

Key Steps in Policy Problem Definition

  1. Think about the problem:

    • Create a precise and complete statement of the empirical situation, using numbers to show extent.

    • Clarify how the values of the client, analyst, affected publics, and other groups shaped the problem.

  2. Delineate the boundaries of the problem:

    • Specify location, duration, and historical events.

    • Be aware of connections to other problems, as their resolution or worsening can affect the problem under analysis.

  3. Develop a fact base:

    • Consult multiple data sources and use several estimating techniques.

    • Verify data and compare with established benchmarks.

  4. List goals and objectives:

    • Acceptability of solutions depends on actors' goals/objectives.

    • Specify measures for each objective to know if it's being achieved.

  5. Identify the policy envelope:

    • The range of variables considered affects the alternatives examined.

    • Define dependent and independent variables.

    • Clearly state what is "out of scope" and why.

  6. Display potential costs and benefits:

    • Report potential costs and benefits to interested parties.

    • Consider what each actor would gain or lose if the problem is solved.

    • Focus on views about the problem and theoretical solution, not alternative solutions' impacts (quantification is preferable, but narrative is acceptable for problem definition).

  7. Review the problem statement:

    • Formulate single-sentence definitions of the problem.

    • Ensure the problem is stated in a way that allows for client action.

    • Confirm enough insights have been developed to suggest possible alternatives.

    • Challenge assumptions.

A Policy Problem is Not the Same as a Research Question

  • A policy problem describes an undesirable state or gap. A research question investigates the causes or relationships contributing to that state.

  • Example: West Point Grey Storefront Vacancies

    • Policy Problem: "The vacancy rate is too high."

    • Research Question: "Low population density is contributing to a high storefront vacancy rate." (Recognizes other independent variables might impact vacancy rates).

  • Example: Housing Affordability

    • Policy Problem: "The # of households spending > 50% of income on rent is too high."

    • Research Question: "Slow permit processing times are contributing to high housing costs." (Recognizes other independent variables might impact housing costs).

  • Example: Transit Ridership

    • Policy Problem: "The # of trips made by private vehicle is too high."

    • Research Question: "Low density housing contributes to low transit ridership." (Recognizes other independent variables might impact transit ridership).

Small Group Discussion - Case Studies

Discussion points for case studies:

  1. What is the policy problem?

  2. What hypothesis is the research testing?

  3. What is the dependent variable?

  4. What are the independent variables?

  5. What is the data source for each variable?

  6. Is the research successful?

  7. Are there other theories/hypotheses that should be tested? Are there other independent variables that might have an impact?

Instructions for Assignment #1: Policy Problem Definition (Due Week 3 - Sept. 25)

  1. Choose an urban planning policy problem for your major project.

  2. Prepare a report addressing:

    • Context, problem setup, and need for government intervention.

    • Stakeholder Analysis:

      • Who is the decision-maker client? What is their jurisdiction and interest?

      • Who else is concerned and why?

    • How numbers can be used to define this problem.

    • "Policy envelope":

      • Dependent and independent variables.

      • What's "out of scope"?

    • Provide 3 options for defining the policy problem in a single sentence.

  • Reports should use 1.5 line spacing and not exceed 3 pages total.

  • Students should come to class next week prepared to discuss their selected topic.

Tips and Tricks: How to Find a Policy Problem for the Major Project

  1. Review media articles on a topic of interest.

  2. Review research studies on the topic of interest.

  3. Find an overarching strategy document for local or regional government (e.g., Sustainability Strategy, Housing Strategy, Transportation Plan, Economic Action Plan).

    • Identify a specific goal/action in the plan for further investigation.

    • Consider how to investigate the goal/action for this course.

    • There should be underlying research questions that can be tested.

    • There should be examples of actions other cities have attempted to deal with the problem (or similar problems).

Readings for Week #3

  • British Columbia/Yukon Open Authoring Platform. Research Methods for the Social Sciences (Read Section 3.5 - Quantitative, Qualitative, & Mixed Methods Research Approaches).

  • Course Text: (Read Section: 5.2.1 - The Free-Market Model, Section 5.2.2 - Costs, and Section 5.2.3 - Benefits, Pages 180-184).

  • Course Text: (Read Section: 5.3 – Commonly Employed Evaluation Criteria, Pages 194-204).

  • Course Text: (Read Chapter 6 – Identifying Alternatives, Pages 215-237).