1/119
A comprehensive vocabulary review set covering all core concepts, evaluability assessment procedures, five domains of evaluation, implementation metrics, outcome evaluation designs, and economic evaluation methodologies from the lecture notes.
Program evaluation
A systematic assessment used to assess and improve the design and performance of organizational programs.
Program
A set of planned activities, supported by dedicated resources, that are intended to produce specified changes for an identified target population.
Target population
The identified group, such as clients, communities, or an organization’s own staff, for whom a program is intended to produce specified changes.
Client and staff surveys
A program evaluation method involving structured questionnaires administered to clients or program staff to understand program components and outcomes.
Direct observation of program activities
An evaluation method where evaluators observe program operations and activities firsthand in the field to assess implementation.
Qualitative analysis of program documents
The review and content analysis of administrative records, grant proposals, annual reports, or meeting minutes to understand program design and context.
Experimental design in program evaluation
A methodology utilizing random assignment to assess program impact and isolate causal effects on intended outcomes.
Quasi-experimental analysis
An evaluation method using administrative or survey data with non-randomized comparison groups to estimate program effects.
Economic evaluation (general definition)
An evaluative method used to assess program costs, return on investment, and the comparative value of intervention effects against resources expended.
Formative evaluation
An evaluation intended to provide information for guiding program improvement by helping form or shape the program so it performs better.
Summative evaluation
An evaluation conducted to determine whether a program has produced the intended outcomes or benefits.
Process/implementation evaluation
An evaluation designed to understand a program’s activities, components, operations, clients, staff perceptions, differences across sites, and quality of implementation.
Outcome/impact evaluation
An evaluation designed to test the extent of a program’s impact, specifically its causal effect on intended outcomes.
Evaluability assessment (EA)
An assessment designed to determine whether a program can be evaluated, which kind of evaluation would be useful right now, or what needs to change to make it evaluable.
Staff resistance in evaluability
A barrier to evaluability where program personnel are opposed or hesitant toward being evaluated, rendering an evaluation unfeasible.
Program model clarity
An evaluability prerequisite requiring an agreed-upon, clearly defined operational structure outlining how a program operates and achieves its goals.
Program stability in evaluability
The operational and resource consistency of a program, without which volatility makes conducting an evaluation unfeasible.
Identifiable impact condition
The requirement in an outcome evaluation that sufficient data and a viable control group or condition exist to identify program effects.
Program definition consistency
An EA focus assessing whether program components are clearly defined and sufficiently distinguished to be implemented and measured.
Theory of change
A conceptual explanation and visualization showing why you believe a program will work and how it will get from point A to point B.
Program performance indicators
Agreed-upon, realistic, and measurable metrics used by managers and policymakers to track program objectives.
Agreement on evaluation use
A criterion examined during an EA ensuring that stakeholders actively plan to use findings to improve, expand, or fund the program.
Counterfactual
A reasonable comparison condition representing what would have occurred in the absence of the program, necessary for an inconclusive-free impact evaluation.
EA work group
A collaborative team consisting of the evaluation team and key program stakeholders established to learn about the program and agree on evaluation uses.
Program document review (in EA)
The systematic examination of grant proposals, annual reports, meeting minutes, and websites to understand program history and operations.
Causal model clarification (in EA)
The step in an evaluability assessment where evaluators develop or clarify the explicit link between program activities and outcomes.
Empirical support review (in EA)
A review of existing research literature to determine the level of empirical evidence backing a program’s proposed causal model.
Key stakeholder interviews (in EA)
Structured discussions with program personnel and relevant parties to gauge their understanding of program goals, reality, and context.
Field data collection (in EA)
Gathering real-world operational data through site visits, administrative data reviews, surveys, and observation to assess program reality.
Synthesis of findings (in EA)
Integrating field data and document reviews to outline options for program modifications and present evaluation recommendations to leaders.
5 domains of evaluation questions
A sequential framework of five question types evaluators ask once a program is evaluable, each building upon the previous domain.
Domain 1 evaluation question
The evaluation inquiry asking: 'Is the program needed?'
Domain 2 evaluation question
The evaluation inquiry asking: 'Is the program’s design sound?'
Domain 3 evaluation question
The evaluation inquiry asking: 'Has the program been implemented as intended?'
Domain 4 evaluation question
The evaluation inquiry asking: 'Has the program achieved its intended outcomes?'
Domain 5 evaluation question
The evaluation inquiry asking: 'Do the program’s benefits justify its costs?'
Need
The gap between a population’s current condition and its desired condition.
Needs assessment
An assessment addressing whether a problem is serious enough to warrant intervention and whether a specific program is an appropriate response.
Administrative data analysis (in needs assessment)
Analyzing existing institutional records to establish the prevalence, size, and nature of the social or operational problem.
Key informants
Knowledgeable individuals within a community or agency who are surveyed or interviewed during a needs assessment to understand target needs.
Focus group interviews (in needs assessment)
Facilitated small-group discussions with members of the target population to examine specific needs, barriers, and perceptions.
Community forums
Public meetings observed or participated in by evaluators to assess community sentiment and perceived local problems.
Field setting observations (in needs assessment)
Direct in-person observation of neighborhood or organizational conditions to evaluate the actual extent of community need.
Program theory and design
The structural framework and operational logic that explicate how a program is expected to achieve its intended outcomes.
Fuzzy broad goals
Vague or overly broad program aspirations (e.g., 'to reduce hunger') that lack a well-defined operational process for achievement.
Theory of change model
A visual and conceptual representation showing why you believe a program will work.
Logic model
A visual diagram showing what the program consists of (inputs, activities) and what it produces (immediate and ultimate goals).
Causal model
A graphical representation tracing program inputs and mediator activities to immediate mechanisms and ultimate outcomes.
Crisis counseling (victim service model)
An immediate support activity within victim services designed to mitigate trauma and foster engagement.
Criminal justice support/advocacy
Advocacy activities assisting victims navigating legal systems to build trust and encourage victim engagement.
Investigative capacity (victim service model)
An intermediate outcome where on-scene advocacy frees up investigators, generating new investigative leads.
Arrest and adjudication (victim service model)
The ultimate criminal justice outcome produced through strengthened investigative capacity and victim engagement.
Inputs / Funding (in logic models)
The dedicated resources, staffing, and financial support invested into a program to enable its operations.
Activities (in logic models)
The specific actions, services, and treatments delivered by program staff using dedicated resources.
Immediate goals (in logic models)
The short-term direct accomplishments or changes resulting immediately from program activities.
Ultimate goals (in logic models)
The long-term desired societal or client impacts that a program is fundamentally designed to achieve.
Organizational capacity
The internal capabilities—including resources, staffing, training, and partnerships—required to deliver a program as designed.
Target population reach
A process evaluation dimension assessing whether a program actually engages and serves its intended target recipients.
Subgroup service disparities
Variations in service delivery where certain demographic or operational subgroups are over-served or under-served.
Case management data analysis
A process evaluation method examining administrative client files to verify service receipt and client progression.
Implementation fidelity
The extent to which an executed program adheres to its documented program theory and original design.
Implementation gap
A discrepancy between a program as it was designed or intended and the program that was actually delivered.
Null impact finding
A finding of no program effect, which without process data cannot distinguish between a flawed theory and failure of delivery.
Multi-site implementation variation
Differences in how a program is delivered across distinct locations, staff teams, or client populations.
Lattimore et al. (2005)
A multi-site evaluation study examining the implementation of prisoner reentry programs under the Serious and Violent Offender Reentry Initiative (SVORI).
Serious and Violent Offender Reentry Initiative (SVORI)
A large-scale multi-site reentry initiative evaluated by Lattimore et al. (2005) to assess program delivery and offender outcomes.
Causal effect
An impact evaluation finding proving that the program itself directly generated changes in outcomes, rather than a mere correlation.
Baseline comparison data limitation
An obstacle to identifying causal effects caused by the absence of data for a sufficient period before or after program rollout.
Ambiguous program start date
A methodological issue where an unclear implementation start date makes isolating post-intervention changes difficult.
Gradual program effects
Anticipated program impacts that accrue slowly over extended periods, making short-term causal identification difficult.
Ariel et al. (2016)
A randomized controlled trial study evaluating the impact of 'soft policing' by police community support officers on crime and calls for service.
Soft policing
A visible, community-focused policing approach tested by Ariel et al. (2016) at hot spots using non-sworn community support officers.
Police Community Support Officers (PCSOs)
Uniformed personnel in Ariel et al. (2016) deployed to hot spots to deliver scheduled patrols and community visits.
Adherence tracking via GPS
The use of location tracking technology in Ariel et al. (2016) to verify whether PCSOs adhered to assigned visit frequencies and durations.
Treatment intensity as a moderator
An analysis examining how varying dosages of an intervention associate with outcomes, treated as an observational association when dosage is not randomly assigned.
Single-site outcome evaluation
An evaluation assessing a program in a single location, which often possesses good internal validity but limited external validity.
Multi-site evaluation
An evaluation where the identical program model is evaluated across multiple sites to establish generalizability.
External validity
The degree to which an evaluation's findings can be generalized across diverse site conditions, client populations, and times.
Bundled treatment
A phenomenon where an evaluator's feedback changes program implementation mid-study, causing the evaluation itself to become part of the intervention.
Researcher-involved evaluation effect
The tendency for programs with active external researcher oversight to perform differently than programs operating unobserved or self-evaluating.
Economic evaluation
A research design that compares the costs and effects of one or more interventions, weighing effects against sacrificed opportunity costs.
Opportunity cost
The financial and practical value of alternatives and opportunities that were sacrificed by implementing a program.
Business as usual comparator
The baseline, standard operating alternative against which a new intervention is compared in an economic evaluation.
Cost analysis
An assessment involving the calculation of total costs and incremental costs of an intervention, serving as the foundation for economic evaluations.
Incremental costs
Costs incurred by an intervention that are above and beyond the cost of an alternative intervention or standard practice.
Societal perspective (costs)
A cost accounting framework where all costs are counted regardless of who pays for them, including in-kind resources and volunteer hours.
Funder's perspective (costs)
A cost accounting perspective that counts only the direct financial expenditures incurred by the funding agency.
Ingredients Method
A systematic cost calculation method that involves identifying all resource inputs, valuing/pricing them, and summing total costs.
In-kind resources
Non-monetary resources utilized by a program, such as volunteer time or donated space, that must be valued under a societal cost perspective.
Personnel ingredients
Human resource inputs whose values are determined by detailing staff qualifications, education, experience, and training.
Facilities ingredients
Physical plant inputs whose costs are estimated based on characteristics like square footage, technology, and building usage.
Market prices
Existing commercial monetary rates used to price identified program ingredients.
Shadow prices
Estimated monetary prices assigned to program resources when actual market prices do not exist.
Close substitutes pricing
A technique for deriving shadow prices by applying the known market price of a comparable alternative resource.
Annualizing capital items
Spreading the upfront purchase cost of durable equipment or facilities across their expected useful lifespan.
Discounting future costs
A financial calculation that adjusts costs incurred in future years to their equivalent present-day value.
Cost-feasibility analysis
An economic evaluation that estimates intervention costs to determine whether an agency can afford to implement it within its budget.
Cost drivers
The primary expenditure components—such as personnel, equipment, or facilities—that account for the majority of program costs.
Cost-outcome analysis
An economic analysis comparing the cost and effect of a single program to determine cost per outcome, without comparing it to an alternative intervention.
Partial evaluation
An economic study that examines only one intervention or course of action, which can lead to misleading conclusions about cost-effectiveness.