insh6300 - week 10 (Program Evaluation and Economic Evaluation)

0.0(0)
Studied by 0 people
call kaiCall Kai
learnLearn
examPractice Test
spaced repetitionSpaced Repetition
heart puzzleMatch
flashcardsFlashcards
GameKnowt Play
Card Sorting

1/119

flashcard set

Earn XP

Description and Tags

A comprehensive vocabulary review set covering all core concepts, evaluability assessment procedures, five domains of evaluation, implementation metrics, outcome evaluation designs, and economic evaluation methodologies from the lecture notes.

120 Terms

1
New cards

Program evaluation

A systematic assessment used to assess and improve the design and performance of organizational programs.

2
New cards

Program

A set of planned activities, supported by dedicated resources, that are intended to produce specified changes for an identified target population.

3
New cards

Target population

The identified group, such as clients, communities, or an organization’s own staff, for whom a program is intended to produce specified changes.

4
New cards

Client and staff surveys

A program evaluation method involving structured questionnaires administered to clients or program staff to understand program components and outcomes.

5
New cards

Direct observation of program activities

An evaluation method where evaluators observe program operations and activities firsthand in the field to assess implementation.

6
New cards

Qualitative analysis of program documents

The review and content analysis of administrative records, grant proposals, annual reports, or meeting minutes to understand program design and context.

7
New cards

Experimental design in program evaluation

A methodology utilizing random assignment to assess program impact and isolate causal effects on intended outcomes.

8
New cards

Quasi-experimental analysis

An evaluation method using administrative or survey data with non-randomized comparison groups to estimate program effects.

9
New cards

Economic evaluation (general definition)

An evaluative method used to assess program costs, return on investment, and the comparative value of intervention effects against resources expended.

10
New cards

Formative evaluation

An evaluation intended to provide information for guiding program improvement by helping form or shape the program so it performs better.

11
New cards

Summative evaluation

An evaluation conducted to determine whether a program has produced the intended outcomes or benefits.

12
New cards

Process/implementation evaluation

An evaluation designed to understand a program’s activities, components, operations, clients, staff perceptions, differences across sites, and quality of implementation.

13
New cards

Outcome/impact evaluation

An evaluation designed to test the extent of a program’s impact, specifically its causal effect on intended outcomes.

14
New cards

Evaluability assessment (EA)

An assessment designed to determine whether a program can be evaluated, which kind of evaluation would be useful right now, or what needs to change to make it evaluable.

15
New cards

Staff resistance in evaluability

A barrier to evaluability where program personnel are opposed or hesitant toward being evaluated, rendering an evaluation unfeasible.

16
New cards

Program model clarity

An evaluability prerequisite requiring an agreed-upon, clearly defined operational structure outlining how a program operates and achieves its goals.

17
New cards

Program stability in evaluability

The operational and resource consistency of a program, without which volatility makes conducting an evaluation unfeasible.

18
New cards

Identifiable impact condition

The requirement in an outcome evaluation that sufficient data and a viable control group or condition exist to identify program effects.

19
New cards

Program definition consistency

An EA focus assessing whether program components are clearly defined and sufficiently distinguished to be implemented and measured.

20
New cards

Theory of change

A conceptual explanation and visualization showing why you believe a program will work and how it will get from point A to point B.

21
New cards

Program performance indicators

Agreed-upon, realistic, and measurable metrics used by managers and policymakers to track program objectives.

22
New cards

Agreement on evaluation use

A criterion examined during an EA ensuring that stakeholders actively plan to use findings to improve, expand, or fund the program.

23
New cards

Counterfactual

A reasonable comparison condition representing what would have occurred in the absence of the program, necessary for an inconclusive-free impact evaluation.

24
New cards

EA work group

A collaborative team consisting of the evaluation team and key program stakeholders established to learn about the program and agree on evaluation uses.

25
New cards

Program document review (in EA)

The systematic examination of grant proposals, annual reports, meeting minutes, and websites to understand program history and operations.

26
New cards

Causal model clarification (in EA)

The step in an evaluability assessment where evaluators develop or clarify the explicit link between program activities and outcomes.

27
New cards

Empirical support review (in EA)

A review of existing research literature to determine the level of empirical evidence backing a program’s proposed causal model.

28
New cards

Key stakeholder interviews (in EA)

Structured discussions with program personnel and relevant parties to gauge their understanding of program goals, reality, and context.

29
New cards

Field data collection (in EA)

Gathering real-world operational data through site visits, administrative data reviews, surveys, and observation to assess program reality.

30
New cards

Synthesis of findings (in EA)

Integrating field data and document reviews to outline options for program modifications and present evaluation recommendations to leaders.

31
New cards

5 domains of evaluation questions

A sequential framework of five question types evaluators ask once a program is evaluable, each building upon the previous domain.

32
New cards

Domain 1 evaluation question

The evaluation inquiry asking: 'Is the program needed?'

33
New cards

Domain 2 evaluation question

The evaluation inquiry asking: 'Is the program’s design sound?'

34
New cards

Domain 3 evaluation question

The evaluation inquiry asking: 'Has the program been implemented as intended?'

35
New cards

Domain 4 evaluation question

The evaluation inquiry asking: 'Has the program achieved its intended outcomes?'

36
New cards

Domain 5 evaluation question

The evaluation inquiry asking: 'Do the program’s benefits justify its costs?'

37
New cards

Need

The gap between a population’s current condition and its desired condition.

38
New cards

Needs assessment

An assessment addressing whether a problem is serious enough to warrant intervention and whether a specific program is an appropriate response.

39
New cards

Administrative data analysis (in needs assessment)

Analyzing existing institutional records to establish the prevalence, size, and nature of the social or operational problem.

40
New cards

Key informants

Knowledgeable individuals within a community or agency who are surveyed or interviewed during a needs assessment to understand target needs.

41
New cards

Focus group interviews (in needs assessment)

Facilitated small-group discussions with members of the target population to examine specific needs, barriers, and perceptions.

42
New cards

Community forums

Public meetings observed or participated in by evaluators to assess community sentiment and perceived local problems.

43
New cards

Field setting observations (in needs assessment)

Direct in-person observation of neighborhood or organizational conditions to evaluate the actual extent of community need.

44
New cards

Program theory and design

The structural framework and operational logic that explicate how a program is expected to achieve its intended outcomes.

45
New cards

Fuzzy broad goals

Vague or overly broad program aspirations (e.g., 'to reduce hunger') that lack a well-defined operational process for achievement.

46
New cards

Theory of change model

A visual and conceptual representation showing why you believe a program will work.

47
New cards

Logic model

A visual diagram showing what the program consists of (inputs, activities) and what it produces (immediate and ultimate goals).

48
New cards

Causal model

A graphical representation tracing program inputs and mediator activities to immediate mechanisms and ultimate outcomes.

49
New cards

Crisis counseling (victim service model)

An immediate support activity within victim services designed to mitigate trauma and foster engagement.

50
New cards

Criminal justice support/advocacy

Advocacy activities assisting victims navigating legal systems to build trust and encourage victim engagement.

51
New cards

Investigative capacity (victim service model)

An intermediate outcome where on-scene advocacy frees up investigators, generating new investigative leads.

52
New cards

Arrest and adjudication (victim service model)

The ultimate criminal justice outcome produced through strengthened investigative capacity and victim engagement.

53
New cards

Inputs / Funding (in logic models)

The dedicated resources, staffing, and financial support invested into a program to enable its operations.

54
New cards

Activities (in logic models)

The specific actions, services, and treatments delivered by program staff using dedicated resources.

55
New cards

Immediate goals (in logic models)

The short-term direct accomplishments or changes resulting immediately from program activities.

56
New cards

Ultimate goals (in logic models)

The long-term desired societal or client impacts that a program is fundamentally designed to achieve.

57
New cards

Organizational capacity

The internal capabilities—including resources, staffing, training, and partnerships—required to deliver a program as designed.

58
New cards

Target population reach

A process evaluation dimension assessing whether a program actually engages and serves its intended target recipients.

59
New cards

Subgroup service disparities

Variations in service delivery where certain demographic or operational subgroups are over-served or under-served.

60
New cards

Case management data analysis

A process evaluation method examining administrative client files to verify service receipt and client progression.

61
New cards

Implementation fidelity

The extent to which an executed program adheres to its documented program theory and original design.

62
New cards

Implementation gap

A discrepancy between a program as it was designed or intended and the program that was actually delivered.

63
New cards

Null impact finding

A finding of no program effect, which without process data cannot distinguish between a flawed theory and failure of delivery.

64
New cards

Multi-site implementation variation

Differences in how a program is delivered across distinct locations, staff teams, or client populations.

65
New cards

Lattimore et al. (2005)

A multi-site evaluation study examining the implementation of prisoner reentry programs under the Serious and Violent Offender Reentry Initiative (SVORI).

66
New cards

Serious and Violent Offender Reentry Initiative (SVORI)

A large-scale multi-site reentry initiative evaluated by Lattimore et al. (2005) to assess program delivery and offender outcomes.

67
New cards

Causal effect

An impact evaluation finding proving that the program itself directly generated changes in outcomes, rather than a mere correlation.

68
New cards

Baseline comparison data limitation

An obstacle to identifying causal effects caused by the absence of data for a sufficient period before or after program rollout.

69
New cards

Ambiguous program start date

A methodological issue where an unclear implementation start date makes isolating post-intervention changes difficult.

70
New cards

Gradual program effects

Anticipated program impacts that accrue slowly over extended periods, making short-term causal identification difficult.

71
New cards

Ariel et al. (2016)

A randomized controlled trial study evaluating the impact of 'soft policing' by police community support officers on crime and calls for service.

72
New cards

Soft policing

A visible, community-focused policing approach tested by Ariel et al. (2016) at hot spots using non-sworn community support officers.

73
New cards

Police Community Support Officers (PCSOs)

Uniformed personnel in Ariel et al. (2016) deployed to hot spots to deliver scheduled patrols and community visits.

74
New cards

Adherence tracking via GPS

The use of location tracking technology in Ariel et al. (2016) to verify whether PCSOs adhered to assigned visit frequencies and durations.

75
New cards

Treatment intensity as a moderator

An analysis examining how varying dosages of an intervention associate with outcomes, treated as an observational association when dosage is not randomly assigned.

76
New cards

Single-site outcome evaluation

An evaluation assessing a program in a single location, which often possesses good internal validity but limited external validity.

77
New cards

Multi-site evaluation

An evaluation where the identical program model is evaluated across multiple sites to establish generalizability.

78
New cards

External validity

The degree to which an evaluation's findings can be generalized across diverse site conditions, client populations, and times.

79
New cards

Bundled treatment

A phenomenon where an evaluator's feedback changes program implementation mid-study, causing the evaluation itself to become part of the intervention.

80
New cards

Researcher-involved evaluation effect

The tendency for programs with active external researcher oversight to perform differently than programs operating unobserved or self-evaluating.

81
New cards

Economic evaluation

A research design that compares the costs and effects of one or more interventions, weighing effects against sacrificed opportunity costs.

82
New cards

Opportunity cost

The financial and practical value of alternatives and opportunities that were sacrificed by implementing a program.

83
New cards

Business as usual comparator

The baseline, standard operating alternative against which a new intervention is compared in an economic evaluation.

84
New cards

Cost analysis

An assessment involving the calculation of total costs and incremental costs of an intervention, serving as the foundation for economic evaluations.

85
New cards

Incremental costs

Costs incurred by an intervention that are above and beyond the cost of an alternative intervention or standard practice.

86
New cards

Societal perspective (costs)

A cost accounting framework where all costs are counted regardless of who pays for them, including in-kind resources and volunteer hours.

87
New cards

Funder's perspective (costs)

A cost accounting perspective that counts only the direct financial expenditures incurred by the funding agency.

88
New cards

Ingredients Method

A systematic cost calculation method that involves identifying all resource inputs, valuing/pricing them, and summing total costs.

89
New cards

In-kind resources

Non-monetary resources utilized by a program, such as volunteer time or donated space, that must be valued under a societal cost perspective.

90
New cards

Personnel ingredients

Human resource inputs whose values are determined by detailing staff qualifications, education, experience, and training.

91
New cards

Facilities ingredients

Physical plant inputs whose costs are estimated based on characteristics like square footage, technology, and building usage.

92
New cards

Market prices

Existing commercial monetary rates used to price identified program ingredients.

93
New cards

Shadow prices

Estimated monetary prices assigned to program resources when actual market prices do not exist.

94
New cards

Close substitutes pricing

A technique for deriving shadow prices by applying the known market price of a comparable alternative resource.

95
New cards

Annualizing capital items

Spreading the upfront purchase cost of durable equipment or facilities across their expected useful lifespan.

96
New cards

Discounting future costs

A financial calculation that adjusts costs incurred in future years to their equivalent present-day value.

97
New cards

Cost-feasibility analysis

An economic evaluation that estimates intervention costs to determine whether an agency can afford to implement it within its budget.

98
New cards

Cost drivers

The primary expenditure components—such as personnel, equipment, or facilities—that account for the majority of program costs.

99
New cards

Cost-outcome analysis

An economic analysis comparing the cost and effect of a single program to determine cost per outcome, without comparing it to an alternative intervention.

100
New cards

Partial evaluation

An economic study that examines only one intervention or course of action, which can lead to misleading conclusions about cost-effectiveness.