Program evaluation

program evaluation

a systematic process of collecting and analyzing data to detrmine how effective or efficient a progrma, plocy, or organization is

program

any set of activities that are related to achieve an certain outcome

how do we use program evaluations

• make changes in practice

• learn to change behaviors in a specific population more effectively

• help in determining to continuation or change of something

types of program evaluation

• formative

• summative

formative evaluation

assess the feasibility, appropriateness, and acceptability of a program before full implementation

• focuses on learning and improvement

summative evaluation

purpose is to assess the effectiveness of the program (at the end), and the extent to which awareness, knowledge, attitudes, behaviors or health status changed, as a result of your program.

– Requires the measurement and establishment of a baseline value, prior to implementation.

Qualitative data

– Verbal/written; most of the time these are from semi-structured interviews or focus groups

– Review of documents (e.g., policies) and abstracting major themes from them

collecting quantitative data

• Collection of data from patients

  • Blood pressure, BMI, Hemoglobin a1c, etc.

• Surveys

  • Paper, online, mail, phone, etc

  • Now, mostly online via a mobile phone or tablet

  • These are collecting demographic variables, knowledge question, attitude/belief Qs, and/or behaviors.

• Direct observations, where you’re scoring patients’ behavior or subjects’ interaction

• Using numbered rubrics to grade/score participants’ writing or journaling

• Anything that provides you with a number— quantitative!

collecting qualitiative data

• semi-structured, one-on-one interviews

• focus groups

• pictures, songs, poetry—anything that is NOT numerical.

• For most health promotion, tend to see focus groups, one-on-one interviews and sometimes photographs used in evaluating programs

how is evaluation different from research

Research seeks to contribute to generalizable knowledge & test a hypothesis, whereas evaluation seeks to continuously improve programs and produce findings and recommendations for decision makers.

PRECEDE-PROCEED

A community-oriented, participatory model for creating successful community health promotion interventions

4 phases of PRECEDE

1. social assessment

2. epidemiological assessment

3. educational and ecological assessment

4. health program and policy development

social assessment

• Seeks to subjectively define the quality of life (pblms & priorities) of those in the priority population

• Involve priority population in the assessment of their own needs and aspirations

• Examples of social indicators of quality of life are achievement, alienation, comfort, crime, discrimination, happiness, self-esteem, unemployment, and welfare.

epidemiological assessment

• You identify the health or other issues that most clearly influence the outcome of the community.

• Use data to ID and rank the health goals & pblms that may contribute to or interact with health problems from Step 1.

• Data include vital stats, as well as genetic, behavioral and/or environmental factors

  • Important to rank health problems; risk factors must be prioritized

• Phase 2 is crucial for planning health programs.

  • Must weigh out what is most important to the community (not to you or us!)

• Both Phase 1 & 2 facilitate investment, enhance community trust, cultural insight and ensure environmental factors are sought.

• In these two phases, you create the objectives for your intervention.

educational & ecological assessment

• You identify the behaviors and lifestyles, and/or environmental factors that must be changed to affect the health issues identified in Phase 2

• predisposing factors

• enabling factors

• reinforcing factors

predisposing factors

include knowledge and many affective traits such as one’s attitudes, values, beliefs & perceptions

– These factors can facilitate or hinder a person’s motivation to change

– Can be altered through an intervention with a direct communication component

enabling factors

barriers or facilitators created mainly by societal forces or systems, which include access to healthcare facilities (as an example), or services, availability of resources, transportation, and problem-solving skills

– These are the resources & skills required to make desired behavioral & environmental changes

reinforcing factors

barriers or facilitators created mainly by societal forces or systems, which include access to healthcare facilities (as an example), or services, availability of resources, transportation, and problem-solving skills

– These are the resources & skills required to make desired behavioral & environmental changes

health program & policy

• Phase 4 is the final step of PRECEDE and the 1st phase of PROCEED

• Includes intervention alignment and administrative & policy assessment

• The intent of intervention alignment is to match appropriate strategies & interventions with the 3 factors from phase 3

• Also, you are identifying the internal administrative issues & internal and external policy issues that can affect the success of the proposed intervention.

• This is a critical phase.

  • You must determine here if the capabilities & resources of existing personnel and participating stakeholders are available to develop and implement the proposed program.

  • Sometimes have to generate additional funding; obtain additional resources (including people) in order to carry out the intervention.

• It is between Phases 4 and 5 that PRECEDE ends and PROCEED begins.

Implementation

• With appropriate resources in hand, planners select the intervention(s) & strategies. The actual implementation of the intervention/program begins.

• Process evaluation: you monitor the implementation; measurements of implementation to control, assure, or improve the quality of the program.

– Assessing the extent to which the intervention is implemented with fidelity to the plan (aka “fidelity of implementation.”

– Is the intervention going according to plan? If not, adjust accordingly

Phases 6-8

• These phases address short-term evaluation (Phase 6), intermediate evaluation (Phase 7), and long-term evaluation (Phase 8).

impact evaluation

– You’re evaluating whether the intervention is having the intended impact on the behavioral & environmental factors it’s aimed at, and adjust accordingly.

– Efficacy and effectiveness

outcome evaluation

– You evaluate whether the intervention’s effects are in turn producing the outcome(s) the community identified in Phase 1, and adjust accordingly.

– To assess the changes that occur as a result of an intervention; some changes may have been anticipated and others unanticipated

– This is generally several years or more after the intervention.

– Long-term effects on QofL indicator, overall health