PH 4800

Program Evaluation Definition

The systematic collection of information about program activities, characteristics, and outcomes to make judgments and improve effectiveness.

Purpose of Program Evaluation

To monitor progress, determine outcome production, justify funding, find quality improvement opportunities, and ensure effective programs.

CDC Framework Step 1

Engage stakeholders.

CDC Framework Step 2

Describe the program (develop mission, goals, objectives, logic model, and read literature).

CDC Framework Step 3

Focus the evaluation design (evaluation questions, evaluation type, design, sampling, and validity threats).

CDC Framework Step 4

Gather credible evidence (indicators, survey/interview items, data collection training, and data organization).

CDC Framework Step 5

Justify conclusions (variable measurement levels, quantitative analysis, and qualitative thematic analysis).

CDC Framework Step 6

Ensure use and share lessons (components of evaluation reports and methods to ensure utilization).

Utility Standards

Ensuring the evaluation provides relevant, timely information for those who need the results.

Feasibility Standards

Ensuring planned evaluation activities are realistic given available time, resources, and expertise.

Accuracy Standards

Ensuring the evaluation produces valid and reliable findings.

Propriety Standards

Ensuring the evaluation protects individual rights/welfare and engages those directly affected.

Four Standards Groups

Utility, feasibility, accuracy, and propriety standards.

Pre-Formative Evaluation

Planning evaluation during the program planning stage rather than after implementation.

Pre-Formative Evaluation Developer

Dr. Muhsin Michael Orsini.

Benefits of Pre-Formative Evaluation

Clearly defined program purpose, written SMART objectives, and pre-prepared design and survey/interview items.

Challenges to Pre-Formative Evaluation

Planners must think about evaluation early and need training or collaboration with trained evaluators.

Pre-Formative Evaluation Citation

Orsini, Wyrick, and Milroy (2012) published in Evaluation and Program Planning.

Traditional Evaluation Timing Challenge

Waiting until implementation to think about evaluation causes planners to struggle with purpose and SMART objectives.

Survey and Interview Item Challenge

Creating survey and interview items to measure program success is difficult when planned after implementation.

Definition of a Public Health Program

Any set of related activities undertaken to achieve an intended outcome or organized public health action.

Definition of Program Planning

A multi-step process beginning with problem definition and evaluation plan development, including a feedback loop.

Importance of Program Planning

Helps avoid errors, keeps everyone informed, helps implementation, and shows where the program fits overall.

Step 1 of the Generalized Model

Identify primary health issues in your community using primary and secondary data.

SMART Objectives Acronym

Specific, Measureable, Attainable, Realistic, Time-phased.

Types of SMART Objectives

Process, impact learning, impact behavioral, and outcome objectives.

Logic Models

1-page visual representations of 'if/then' logic connecting inputs, activities, outputs, and outcomes.

Task Development Timelines

1-page tables with rows for tasks and columns for deadlines based on process objectives.