Week 13 + 14: Evaluating health promotion & disease prevention programs

0.0(0)
Studied by 11 people
call kaiCall Kai
learnLearn
examPractice Test
spaced repetitionSpaced Repetition
heart puzzleMatch
flashcardsFlashcards
GameKnowt Play
Card Sorting

1/44

flashcard set

Earn XP

Description and Tags

45 Terms

1
New cards
evaluation
- is CRITICAL
- is the process of collecting and using information to guide decision making
- SHOULD be done throughout program processes
- frightening how many programs have not and do not get evaluated = wasted money, resources, opportunities
2
New cards
evaluation is:
A systematic study that is designed, conducted & reported in order to assist a client group to judge and/or to improve the worth and/or merit of some object of interest
3
New cards
Evaluation judges 2 things:
worth = whether or not object/program is needed

merit = whether or not object/ program is good b
4
New cards
Pine River institute
- UTAH
- aims to provide the highest quality of care for families and victims of substance abuse
- wilderness therapy, aftercare, treatment
- no government funding = community for support
lives
families
futures
5
New cards
PRI = measuring impact
- measuring impact = hope
- resulted in government investing in more beds
6
New cards
2 critical purposes of program evaluation
1. Assessing & improving quality

2. Determining program effectiveness
7
New cards
stakeholders - why do they want programs evaluated?
- to determine the the achievement of the objective
- to improve program implementation
- to provide accountability to funders, community and other stakeholders
- improve community support
- inform policy decisions
- to contribute to the scientific base for community public health intervention
8
New cards
When should evaluation be done?
EARLY
- all the way through the intervention
- begins when goals and objectives are developed
- must be timely
9
New cards
Who should be involved?
- collaboration effort among all stakeholders
- anyone relevant
like:
- program planners
- administrators
- facilitators
- funding agency
10
New cards
Threats to good evaluations
- fear of cuts
- political skewing - report only successes, not weaknesses
- getting hard data = management can get rid of certain programs
- cost effective eval vs impact eval
- other political and ethical ramifications
11
New cards
Stage = initial planning
T.O.E = needs assessment with market analysis
TOE also = cost analysis

- addresses program efficiency
- the systematic appraisal of program costs in relation to program results

*runs during these stages:
- initial planning
- development
- stable operation
12
New cards
Stage = development
Type of Evaluation = Formative evaluation + market testing
formative evaluation:
systematic appraisal of prototype promotional & delivery strategies in the dev't or draft stage before full scale production
13
New cards
formative evaluation
- immediate feedback
- comprehensive
- get info before & during program implementation
14
New cards
stage = early implementation
TOE = implementation evaluation
systematic appraisal of the extent to which a program or service conforms to the original plan
- eg: active living program
15
New cards
stage = routine operation
TOE = Process evaluation
systematic appraisal of program delivery and usage under routine or normal operation
16
New cards
Process evaluation
- Provides documentation during program implementation
- enables planners to make adjustments for improvement of program
- eg. fun & fitness
17
New cards
stage = Stable Operation
TOE = Outcome Evaluation
systematic appraisal of program's impact on clients, in relation to level of program participation and baseline characteristics
18
New cards
Outcome evaluation
determines whether program met stated long-term goals & objectives
such as:
- an increase in functional ability
- a decrease in mortality
- any negative effects

- DO NOT CONDUCT AN OUTCOME EVAL PREMATURELY
- programs should be operating at least a year prior to the OE
- settling period occurs in every program ( 1 yr is safe standard)
19
New cards

3 reasons why you should be wary of simply asking clients if they feel they have benefited from the program
1) retrospective reporting bias
- you forget what you were like before the program

2) Assumption that client would not have changed without the program but... they might have without intervention

3) social desirability response
- people will just say yes to please everyone
20
New cards
Stage = further planning
TOE = Periodic needs assessment
checking in
- taking notes and data the whole time the program is running
21
New cards
purpose of evaluation
- determine achievement of objectives
- improve program implementation
- provide accountability to stakeholders
- increase support for initiatives
- to contribute to the scientific base
- to inform policy decisions
- determine achievement of objectives 
- improve program implementation
- provide accountability to stakeholders 
- increase support for initiatives 
- to contribute to the scientific base
- to inform policy decisions
22
New cards
PHO 10 step program
* non linear (arrows)

Step 1: Clarify what is being evaluated

Step 2: Engage key people

Step 3: Assess resources and evaluability

Step 4: Determine evaluation questions

Step 5: Determine appropriate methods

Step 6: Develop an evaluation plan

Step 7: Collect data

Step 8: Process data & analyze results

Step 9: Interpret and disseminate results

Step 10: Apply evaluation findings
* non linear (arrows)

Step 1: Clarify what is being evaluated 

Step 2: Engage key people 

Step 3: Assess resources and evaluability 

Step 4: Determine evaluation questions 

Step 5: Determine appropriate methods 

Step 6: Develop an evaluation plan 

Step 7: Collect data 

Step 8: Process data & analyze results 

Step 9: Interpret and disseminate results

Step 10: Apply evaluation findings
23
New cards
Step 1: Clarify program
creates a strong foundation

steps:
1. define program goals
2. define population of interest
3. define outcome objectives
4. define strategies, activities and resources

at the end of stage 1, u should have:
- clear program goals + population of interest
- logic model + program description
- agreed-upon terminology
24
New cards
Step 2: Engage stakeholders
stakeholders = anyone the program/ evaluation affects

steps:
1. Identify potential stakeholders
- core, involved, supportive, peripheral
2. Identify stakeholders interests and roles
3. engage stakeholders to review
4. stakeholder brainstorming

by step 2:
- identified stakeholders
- categorized their level of involvement
- formed evaluation commitee
- brainstormed purpose and use
25
New cards
Step 3: Assess resources and evaluability
clarify resources available

steps:
1. Identify available resources
- data, time, costs, equipment
2. determine whether evaluation is appropriate for your program

By step 3:
- sense of available resources
- determine feasibility and scope
26
New cards
Step 4: Determine your evaluation questions
you + stakeholders should have a good understanding of the program, now focus on questions

steps:
1. select and refine evaluation questions
- consider logic model, previous evaluations
2. determine the appropriate evaluation approaches
- needs assessment, process evaluations, outcome evaluations

at the end of step 4:
- gathered evaluation questions
- selected highest priority questions
- identified most appropriate evaluation approaches
27
New cards
Step 5: Determine appropriate methods of measurement and procedures
decide what to measure, how, when, and whom?

steps:
1. decide how to collect data
2. decide what to measure
- process + outcome indicators
3. When will data be collected?
4. from whom will data be collected?

at the end of step 5:
- chosen methods of measurement
- selected indicators
- determined frequency and sample
- developed data collection tools
- considered ethical issues
28
New cards
Step 6: Develop evaluation plan
write a written document - detailing how the program will be monitored and evaluated

1. create an evaluation plan
2. create an evaluation data collection matrix
3. finalize budget

at the end of this step u should have:
- evaluation plan
- matrix
- budget
29
New cards
step 7: Collect data
specify procedures, pilot test, collection tools

steps:
1. develop data collection procedures
2. Train data collectors
3. pilot test procedures and tools
4. collect data

you should have:
- standardized procedures and tools
- ways to obtain consent
- piloted and revised data collection tools
- trained and supported data collectors
- collected data
30
New cards
Step 8: Process data and analyze results
technical step = engaging with data

steps:
1. Data entry (direct or indirect)
2. organize data to enable analysis
3. analyze data
4. set the stage for interpretation

after step 8:
- entered data
- reviewed data during and after collection
- cleaned and organized data
- conducted data analysis
- prepared evaluation results
31
New cards
Step 9: Interpret and disseminate results
what does the data mean?
disseminate = making info available and usable

steps:
1. interpret data
2. develop recommendations
3. develop a communication strategy
4. create and distribute communication products

by the end:
- interpreted results and disseminated key messages to intended users
32
New cards
Step 10: Apply evaluation findings
using the results to direct change and new thinking

steps:
- review recommendations and brainstorm actions
- prioritize changes
- create an action plan
evaluate the evaluation

at the end:
- action is taken and the evaluation can be used to inform subsequent evaluations
33
New cards
formative vs summative vs process evaluation
formative: relates to assessing and improving quality

summative: determines effectiveness

process: occurs at the end of a program and looks backward to measure achievement of process objectives
(closely related to formative)
34
New cards
6 reasons why stakeholders may want programs evaluated

1. To determine achievement of objectives related to improved health status
2. To improve program implementation


3. To provide accountability to funders
4. To increase community support for initiatives
5. To contribute to the scientific base for community public health interventions
6. To inform policy decisions
35
New cards
4 standards of evaluation
Utility - users satisfied
Feasibility - viable + pragmatic
Propriety - ethical
Accuracy - program produces correct findings
Utility - users satisfied 
Feasibility - viable + pragmatic 
Propriety - ethical 
Accuracy - program produces correct findings
36
New cards
Institutional review boards (IRBs)
Because evaluations may have ethical considerations for the individuals involved, most colleges, universities, school systems, and large health organizations have boards to review the evaluation design and potential risk to participants.

- These boards serve to safeguard the rights, privacy, health, and well-being of those involved in the evaluation/research.
37
New cards
Summative evaluations
combo of both:

impact evaluation:
- awareness,
- knowledge,
- attitudes,
- skills
- behaviours

outcome evaluation:
- long-term program measures
- mortality
- morbidity
- disability
combo of both:

impact evaluation: 
- awareness, 
- knowledge, 
- attitudes, 
- skills
- behaviours 

outcome evaluation:
- long-term program measures 
- mortality
- morbidity
- disability
38
New cards
experimental design
- most powerful design
- participants are randomly assigned to the experimental and control groups. The purpose of random assignment is to equally distribute any of the participant characteristics, such as age, gender, and race, between the different groups.
39
New cards
quasi-experimental design
provides results that suggest program effectiveness, but due to nonrandom assignment, the internal validity of the results is not certain.

That is, there is no way to know that the intervention was the cause of the changes because, without random assignment, groups were likely no equivalent or similar on key independent variables before the intervention.
40
New cards
non-experimental design
This is where none of the criteria for true experimentation exists in the structure of sample selection and assignment, exposure to an experimental condition, and data collection
41
New cards
The letter O
refers to measurement or observations involving data derived from questionnaires, tests, interviews, observations, or other information-seeking methods.

*Used for pretests and post-tests
42
New cards
Internal validity
is the degree to which the change in the dependent variable can be attributed to the program (independent variable) and allows evaluators to speak with more confidence that the program itself made a difference.
43
New cards
main threats to vaildity
* social desirability
* expectancy effect
* Hawthorne effect
* Placebo effect
* testing
* instrumentation
* statistical regression
* compensatory rivalry

best ways to prevent threats to validity
^^__**= randomization!!!**__^^

4 cannot be controlled by randomization:

* interaction
* compensatory equalization
* compensatory rivalry
* demoralization
44
New cards
PADS
P: personnel planning

A: Acquisition

D: development

S: sanction
45
New cards
5 stages of team development
forming

storming

norming

performing

adjourning