Applying Evaluation Criteria Thoughtfully - OECD Study Notes

Applying Evaluation Criteria Thoughtfully

OECD Better Policies for Better Lives

Introduction
  • This work reflects the views and responsibilities of the Secretary-General of the OECD, not necessarily representing official views of member countries.
    • The document emphasizes the impartiality of conclusions and expresses no prejudice regarding international boundaries or territorial disputes, particularly concerning Israel and Turkey's positions on Cyprus.
Foreword
  • Evaluation is fundamental to designing and delivering policies aimed at fostering sustainable development.
    • Better evaluations support decision-makers and communities to create lasting benefits for both people and the environment.
  • In 1991, the OECD Development Assistance Committee (DAC) introduced five evaluation criteria that have guided development evaluations for three decades.
    • In 2018-2019, these criteria were updated, incorporating feedback from evaluations and international frameworks such as the 2030 Agenda for Sustainable Development and the Paris Agreement.
Acknowledgements
  • Acknowledges global stakeholders for their input during the 2017-19 revision process, crediting various organizations and individuals for substantial contributions to the document.
Table of Contents
  1. Purpose and Use of the Guidance
    • Introduction to the guidance, its importance, and applications.
  2. The Six Criteria: Their Purpose and Role within Evaluation
    • Detailed definitions and applications of each critical criterion.
  3. Using the Evaluation Criteria in Practice
    • Practical advice on incorporating each of the six criteria.
  4. Understanding the Six Criteria
    • In-depth examination of each criterion and associated challenges.
  5. Conclusion
  • References & Notes.

1. Purpose and Use of the Guidance: Better Criteria, Better Evaluation

Why Guidance is Needed
  • The original DAC criteria (1991) have become a common benchmark across development cooperation.
  • The recent updates align with evolving needs expressed by users regarding practical application rather than mere definitions.
  • This guidance facilitates understanding and correct application of evaluation criteria across diverse contexts.
Who Can Use this Guidance?
  • This document serves evaluators, managers, and students in various sectors, enhancing the quality of evaluations and interventions.

2. The Six Criteria: Their Purpose and Role within Evaluation

What is a Criterion?
  • A criterion is a standard used in evaluation to make judgements about interventions:
    • Relevance: Is the intervention doing the right things?
    • Coherence: How well does the intervention fit?
    • Effectiveness: Is the intervention achieving its objectives?
    • Efficiency: How well are resources being used?
    • Impact: What difference does the intervention make?
    • Sustainability: Will the benefits last?
Types of Interventions Evaluated
  • Includes projects, policies, programmes, strategies, and actions across various sectors.

3. Using the Evaluation Criteria in Practice

Principles for Application
  1. Thoughtful Application
    • Encourage critical thinking to generate high-quality evaluations.
  2. Adapt Criteria to Purpose
    • Tailor application of criteria based on the evaluation’s objectives.
Key Aspects for Thoughtful Application
  • Evaluate based on context, purpose, stakeholders, and intervention specifics.
    • Context: Consider intervention specifics and stakeholder requirements.
    • Purpose: Define what the evaluation seeks to achieve.
    • Stakeholder Engagement: Identify and engage relevant parties in the evaluation process.
Challenges and Consideration in Applying Criteria
  • Evaluate adaptability over various stages of the intervention, such as design, mid-course adjustments, and end evaluations.

4. Understanding the Six Criteria: Definitions, Elements for Analysis and Key Challenges

4.1 Relevance
  • Definition: Examines the alignment of the intervention's objectives with the needs and priorities of stakeholders.
  • Assessment Areas:
    • Needs of beneficiaries
    • Context sensitivity
    • Quality and adaptability of design
  • Challenges: Often involve competing stakeholder priorities, poorly articulated objectives, and changing contexts.
4.2 Coherence
  • Definition: Looks at how the intervention aligns with other interventions.
  • Elements of Analysis:
    • Internal** and external coherence
  • Challenges: Access to comprehensive data can be difficult, leading to suboptimal analysis.
4.3 Effectiveness
  • Definition: Measures the extent to which the intervention meets its objectives.
  • Key Concepts: Actual results, relative importance, and differential results among groups.
  • Challenges: Requires clarity in objectives and strong baseline data.
4.4 Efficiency
  • Definition: Evaluates the optimal use of resources to achieve results.
  • Elements:
    • Economic efficiency
    • Operational efficiency
    • Timeliness
  • Challenges: Vary by intervention complexity and availability of relevant data.
4.5 Impact
  • Definition: Focuses on long-term changes and significant effects caused by the intervention.
  • Components:
    • The significance of outcomes
    • Differential impacts across groups
  • Challenges: Assessing impact requires comprehensive data and clear methodologies.
4.6 Sustainability
  • Definition: Explores whether the benefits of the intervention will continue.
  • Analysis Areas: Financial, economic, social, and environmental sustainability.
  • Challenges: Evaluating prospective sustainability can be difficult, especially regarding external factors influencing results.

Examples and Practical Applications

Examples of Evaluating Each Criterion
  • Relevance Example: Evaluating general budget support by assessing compatibility with partners' strategies and contextual needs.
  • Coherence Example: Assessing Norway's commitments for policy coherence across development assistance domains.
  • Effectiveness Example: Evaluating the effectiveness of Australia’s electoral assistance across different developing nations.
  • Efficiency Example: World Bank’s efficiency evaluation concerning rural electrification projects.
  • Impact Example: Assessing the impacts of health interventions in Colombia.
  • Sustainability Example: Japan's project in Afghanistan evaluated for long-term sustainability post-reconstruction.
Conclusion
  • The six criteria provide a nuanced framework that facilitates comprehensive analysis and supports high-quality evaluations in various contexts. Their thoughtful application helps ensure interventions stay relevant and effective across changing environments, particularly within the frameworks established by the Sustainable Development Goals and the 2030 Agenda.