Slides 5.2 - Evaluation

EVALUATION

Date: February 6, 2025

HOUSEKEEPING

• Policy Memo: Due today by 11:59 PM!

  • Important to follow the rubric for optimal assessment!

  • Submission does not have to be perfect; it is essential to submit something to receive feedback.

A LITTLE REVIEW

Three Types of Policies

1. Informational: Designed to provide knowledge and information without strong coercive measures. Examples include public health campaigns and informational pamphlets.

2. Inducement: Policies that motivate or encourage specific behaviors through incentives. Examples include tax credits for certain actions or subsidies for businesses.

3. Regulatory / Prohibitive: Policies intended to limit or control behavior through mandates and regulations. These typically involve varying levels of coercion. Examples include laws and regulations governing business practices or environmental protocols.

Key Variables Defining Types of Policies:

• Level of Coercion Involved: Degree to which compliance is enforced or mandated, ranging from voluntary participation to mandatory adherence.

• Level of Targeting Involved: Specificity of the group or individuals affected by the policy, which can affect the effectiveness and perceived fairness of the policy.

WHY DO WE NEED TO EVALUATE POLICY?

• Unintended Outcomes: Policies can produce unexpected results that may undermine their goals or create new issues. Evaluations help to identify these outcomes early.

• Changing Policy Outcomes: Over time, external factors can alter the effectiveness or relevance of a policy, necessitating reassessment.

• Bias in Evaluations: Relying solely on outside evaluations can introduce biases, known as publication bias, where favorable results are more likely to be published than unfavorable ones.

Guest Speaker: Brenna Farris

• Insights on current policies from a seasoned expert in public policy evaluation.

IMPORTANCE OF POLICY EVALUATION

• Resource Scarcity: The limited availability of resources necessitates thorough evaluation to ensure they are allocated efficiently and effectively.

• Reallocation of Ineffective Policies: If policies are found to be ineffective, resources can be redirected to more impactful initiatives.

• Notable Examples of Changing Policy Outcomes:

  • CompStat Policing: Examining outcomes through data-driven policing strategies and their effects on crime rates.

  • Unexpected Externalities: Policy interventions can lead to unforeseen positive or negative outcomes, such as the impact of certain regulations on business operations.

  • Universal Preschool: Demonstrating mixed results in academic achievement over time, with variations across demographics.

  • Ban the Box and Eviction Moratoriums: Understanding the complexities and contradictions in intended and actual policy impacts.

  • Publication Bias: Academic research may not accurately reflect what works in practice due to the tendency to publish only positive findings, exemplified in minimum wage studies showing varying outcomes.

COMPSTAT POLICING

• Definition and Purpose: CompStat (Comparative Statistics) is a policing accountability system that tracks and analyzes crime data to enhance police response strategies.

• Goals: Aimed at reducing crime rates through data-driven decision-making. Needs ongoing evaluation to adapt strategies to changing crime patterns.

UNIVERSAL PRESCHOOL

• Limited Long-term Impacts: Research indicates universal preschool may have only modest benefits on long-term academic success.

Key Reasons for Limited Impact:

• Fadeout Effect: The concept that initial educational advantages can diminish as children enter mixed schooling environments post-pre-k.

  • Supported by research from List and Uchida (2024).

• Positive Economic Impact: Parents experience increased earnings during pre-k years, fostering a 22% increase with persistence noted over approximately six years according to Humphries et al. (2024).

  • Potential for increased tax revenue to offset government expenses related to pre-k programs.

BAN THE BOX

• Definition: Ban the Box policies aim to eliminate the requirement for job applicants to disclose prior criminal convictions initially, promoting fair chance employment.

• Goals: To reduce discrimination against individuals with criminal records in the hiring process. However, policy outcomes have yielded unforeseen negative consequences, including decreased job opportunities for some demographics.

• Scholar Disagreement: Ongoing debate among scholars surrounding the analysis of these policies highlights the complexities involved in their implementation.

POTENTIAL ALTERNATIVES

• Identifying Areas of Failure: Policy evaluation plays a crucial role in pinpointing ineffective strategies and informing revisions.

• Inspiring Better Decision-Making Models: Through learning from previous failures, policymakers can innovate more effective frameworks.

EVICTION MORATORIA

• Definition: A policy designed to halt evictions, offering tenants temporary protection during crises.

• Evaluation Criteria: Policymakers often assess effectiveness based on metrics like housing stability and displacement rates, balancing tenant needs with landlord rights.

PUBLICATION SELECTION BIAS

• Definition: This term refers to the tendency of journals to favor publication of studies that yield statistically significant results over those that do not.

• Significance: This bias can skew the perception of research integrity and efficacy, impacting both academic discourse and media representations of policy research.

KEY TAKEAWAYS

• Evaluation Criteria & Implementation Assessment: The same standards applied in policy implementation assessments should guide evaluations to maintain consistency and reliability.

• Critical Importance of Evaluations: Neglecting the evaluation process can lead to false assumptions about policy effectiveness, efficiency, or enduring success.

• Alternative Analysis Methods: Scholarly evaluations should not be viewed as the only form of analysis; comprehensive reviews often reveal implementation flaws hidden within superficial assessments (e.g., the case of food stamp uptake in California).

GUEST SPEAKER: BRENNA FARRIS

• Detailed insights and experiences to be shared by the guest speaker, centering on current best practices in policy evaluation and adaptation processes.