Community Psychology exam 2

Process evaluation

Information about what the program or program staff have to do to create change

Outcome evaluation

Information about changes that the program wants to make for clients

Snapshot process evaluation

Limited time frame done for evaluation purposes

Process monitoring

Ongoing documentation of program performance that determines if a program is operating according to a set standard

Why process monitoring

Confirm that the program is reaching its targets

Understand the outcome/impact data

Improve the program without outcome data

Facilitate program management by making sure administrative standards are being met

Fidelity (process evaluation)

If a program is being delivered as designed, and is consistent with the underlying theory

Dose delivered (process evaluation)

The amount of program being delivered

Dose received (process evaluation)

The extent to which participants engage with materials of the program

Satisfaction (process evaluation)

Participant satisfaction with the program and staff

Reach (process evaluation)

proportion of the target who participates in the program

Recruitment (process evaluation)

Procedures to attract participants to the program

Service utilization

The extent to which the target population receives the services (who are we serving?)

Coverage (service utilization)

Is the program participation reaching its intended levels, and how many people are we serving?

Bias (service utilization)

Is the program serving who it was designed to serve?

Skimming

Systematically eliminating clients who are difficult to serve (different from referring clients who cannot be successfully served by the program)

Organizational functioning

Determining if the program is delivering the intended services according to the plan (what are we doing?)

Causes of implementation failure

Intervention not delivered

Wrong intervention delivered

Intervention is unstandardized

Ways of collecting process data

Existing program records

Develop simple databases

MIS systems

Management information system (MIS)

Computer systems developed by the program to inform decisions within the organization

Ethical process evaluation/monitoring

Designed to avoid being overly punitive to staff and avoid encouraging unwanted changes in staff behavior

Outcome

Characteristic of target population or social condition that a program is expected to change

Outcome level

The status of an outcome at a certain point in time

Outcome change

The difference in outcome levels at different points in time (before and after program)

Reliability

Consistency

Test-retest reliability

Consistency of responses overtime

Inter-item reliability

The extent to which items on a scale are consistent with each other

Interrater reliability

Consistency of ratings across observers

Validity

Is the measure assessing what it’s supposed to

Face validity

Does the measure look like its assessing what it should

Construct validity

Does the measure relate to others of the same construct

Types of construct validity

Convergent - a measure correlating with similar measures

Discriminant - a measure not correlating with dissimilar measures

Criterion validity

Assessing the relationship between the measure and a behavioral criterion

Types of criterion validity

Concurrent validity - scores on a measure are related to a criterion assessed at the same time

Predictive validity - scores on a measure are related to a criterion assessed in the future

Sensitivity

The extent to which a measure can identify true positives

Monitoring outcomes

Continued measurement and reporting of indicators that a program is responsible for changing

Moderator variables

Variables that characterize subgroups in a client population (e.g gender, race, risk factors)

Program effect/impact

What outcomes occurred, and how many were caused by the program?

Variable program effects

Using a moderation analysis to determine if the program impacts subpopulations in different ways

Dose-response analysis

What is a sufficient dose for the population

Relationship between fidelity and impact

Implementation failure - poor fidelity, no program impacts found

Theory failure - good fidelity, no program impacts found

Efficacy trial

Demonstrates the program can achieve desired outcomes, carried out in ideal circumstances

Effectiveness trial

Can the program achieve its outcome in the real world?

When to use impact evaluation

New program or policy - efficacy trial, followed by effectiveness trial

Ongoing programs - Enhance effectiveness, match revised goals, and reduce costs

Counterfactuals

What would have happened without the program? (Estimating the change that could have occurred without the intervention, very difficult to do due to lack of control over environment and naturally-occurring changes)

Internal validity (program effect estimates)

Was the change in the sample caused by the program valid? (increases with full participation of all participants and using reliable and valid measures)

External validity (program effect estimates)

The extent to which the program effect estimates will generalize to the entire target population

Comparison group

Control group that is drawn from a pool of program nonparticipants

Unbiased program effects

Evaluation design works as intended and provides accurate estimates of program effects

Biased program effects

Evaluation designs don’t provide accurate estimates of the program effects (making an incorrect judgement about if a program works)

Sources of bias

Outcome measurements

Research design

Secular trends - naturally occurring trends that affect one group but not both

Interfering events - discrete, short-term event that affects one group but not both

Selection bias

Systematic pre-intervention difference between the two groups influences the outcomes

Selection bias causes

Unknown differences between the groups

Attrition (difference in the number of participants who drop out between the groups)

Missing data (difference in the amount or type of missing data in each group)

Naive estimates of program effects

The average outcome for participants compared to non-participants (does not consider or control for bias)

Covariate-adjusted regression-based estimates of program effects

Similar to naive effects, but measures covariates (variables expected to influence outcomes) at baseline and statistically removes them prior to outcome analysis

Matched comparison groups

Intervention group chosen first, then comparison group is chosen by matching characteristics to the intervention group that could cause differences between the two

Exact/clone matching

A clone is selected from comparison group for each member of the intervention group (very difficult to find a large enough population to draw from and find an exact match for every participant)

Propensity score matching

Intervention group and potential matches placed in dataset and all covariates are used to predict the likelihood of each participant being in the intervention group. Each intervention participant is matched with a comparison group member of the same score, and any intervention group member without a match is dropped.

Interrupted time-series design

Compares outcomes for a period before program implementation with outcomes afterwards.

Threats to internal validity in interrupted time-series design

Any event that occurs at the same time of the interruption (interfering events, secular trends, maturation, regression to the mean)

Cohort design

Comparing outcomes for cohorts who receive services before a program changes with cohorts who receive the service after it changes (bias: interfering events, secular trends, and changes that could affect who is receiving the program)

Fixed effect designs

Measuring an outcome several times as a control in a group of individuals, and then measuring the same outcome after (each data point serves as its own control)

Difference-in-difference design

Comparing pre and post change in outcomes for the intervention group to the pre and post change for the comparison group (accounts for naturally occurring change without the intervention) (protects internal validity if groups are similar)

Comparative interrupted time series designs

Same as difference-in-difference, but includes sufficient pre-intervention data to model the trend overtime (minimum four periods of data) (protects internal validity if groups are similar)