CH.5, Part 1: Understanding the Program
5.1 Embedded Evaluation
Chapter 5 begins the step-by-step explanation of the Embedded Evaluation (EMB-E) framework. The overall idea is that programs should continually learn from evaluation and use that information to improve. Program planning is therefore not something that happens only at the beginning. Programs operate in changing environments, so their design and implementation may need to change as new information becomes available.
Evaluation helps determine:
Whether a program is actually helping the people it intends to serve
How effective the program is
For whom the program is effective
Under what conditions it works
How and why the program produces its outcomes
Chapters 5 through 11 introduce a framework for integrating evaluation into program planning, implementation, and decision making so that evaluation supports continuous improvement.
5.1.1 Continuous Improvement Centered, Theory Based, Utilization Focused
Embedded Evaluation, or EMB-E
Embedded evaluation (EMB-E) is a comprehensive approach that incorporates evaluation directly into a program's design and operations.
EMB-E combines several evaluation perspectives:
Continuous improvement principles
Evaluation should help programs continually learn and improve.
Theory-based evaluation
Evaluation examines the reasoning behind how and why a program is expected to work.
Participatory evaluation
Stakeholders contribute to evaluation planning and decision making.
Utilization-focused evaluation
Evaluation should produce information that stakeholders can actually use.
Stakeholder partnerships are especially important because stakeholders help identify meaningful metrics and determine what information will be useful for program improvement.
EMB-E as both an approach and a framework
The EMB-E framework involves several connected activities:
Define program goals and objectives as measurable outcomes.
Explain the theory behind the program.
Develop a logic model representing the program theory.
Create indicators based on elements of the logic model.
Develop an evaluation matrix.
Use the evaluation matrix to guide data collection and analysis.
Interpret findings with stakeholders.
Develop recommendations intended to support program improvement.
The ultimate purpose is to create a continuous cycle in which evaluation findings inform future program decisions.
Five Steps of Embedded Evaluation
The EMB-E framework contains five major steps:
Step 1: DEFINE
What is the program?
The evaluator works to understand:
What the program does
What it intends to accomplish
Its goals and objectives
Its strategies and activities
Step 2: PLAN
How do I plan the evaluation?
Questions include:
How are program strategies connected to program goals?
What is the program's underlying logic?
What short-term and intermediate objectives lead toward long-term goals?
Is the program theory supported by research?
Step 3: IMPLEMENT
How do I evaluate the program?
This involves determining:
What questions the evaluation should answer
What indicators should be used
What evaluation methods are appropriate
What evaluation design is both rigorous and feasible
Step 4: INTERPRET
How do I interpret the results?
This includes decisions about:
Data collection
Data organization
Data management
Data analysis
Step 5: INFORM and REFINE
How do I use the results?
The evaluator considers:
What the results mean
Whether program goals were achieved
How the program could improve
How findings should be communicated
How to encourage stakeholders to actually use the findings
The framework also emphasizes refinement, meaning information from evaluation should continually feed back into the program and lead to improvement.
5.1.2 Dynamic and Cyclical
Although EMB-E contains numbered steps, the evaluation process is not linear.
The steps should not be treated like a checklist where one step is completed permanently before moving to the next.
Instead:
Evaluation is dynamic.
Program environments change.
Populations may change.
Program conditions may change.
Findings that apply to one population or context may not apply to another.
New information may require evaluators to revisit earlier decisions.
Evaluation should continue for as long as a program is operating.
Continuous improvement cycle
The EMB-E framework is based on repeated cycles of:
Data collection → data use → program refinement → new data collection
For example:
Problems discovered during implementation may require changes to the evaluation plan.
Interpretation of results might reveal that additional data need to be collected.
Findings could reveal that the program itself needs to be redefined.
Embedded evaluation can therefore help organizations both:
Improve programs that are working, and
Recognize when a program is not working and should potentially be discontinued.
5.1.3 EMB-E Steps and Nodes
The embedded evaluation model contains both steps and nodes.
A node represents the product or result produced by completing a step.
The general relationship is:
Step 1: Define → Program Logic
Step 2: Plan → Evaluation Design
Step 3: Implement → Evaluation Data
Step 4: Interpret → Evaluation Results
Step 5: Inform/Refine → Program Decisions and Improvements
However, refinement does not have to wait until Step 5.
For example:
Implementation problems could lead the evaluator back to Step 2.
Unexpected findings might require additional data collection.
Results might raise questions about how the program itself was originally defined.
The model is therefore intended to guide evaluators rather than force them into an inflexible process.
Evaluators must remain flexible and responsive to:
What they learn about the program
Stakeholder needs
Program conditions
Information needed by leadership for decision making
5.2 Understanding the Program
The first major task in evaluation is to understand the program being evaluated.
Evaluators should not assume that what they already know about a program is sufficient.
Instead, they should gather information from multiple sources to understand:
Different stakeholder perspectives
Program expectations
Program operations
Program history
How the program is actually supposed to work
Understanding the program is the foundation for evaluating whether it is effective.
Two primary components
Understanding a program requires identifying:
1. What the program wants to accomplish
These are the program's:
Long-term goals
Objectives
2. What the program does to accomplish those goals
These are the program's:
Strategies
Activities
The evaluator must also understand the assumed connection between these two components.
Key terminology
Goals
Represent the long-term "why" of the program.
Describe what the program ultimately intends to accomplish.
Objectives
Represent outcomes that occur before the long-term goal is achieved.
Strategies
Groups of activities intended to produce program outcomes.
Activities
Specific things the program does.
The chapter sometimes uses strategies and activities interchangeably, although technically a strategy may consist of several related activities.
5.2.1 Previous Evaluations and Program Documentation
One way to understand a program is to examine information that already exists.
Previous evaluations
Evaluators should search for previous evaluations of:
The same program
Similar programs
Similar interventions
Previous evaluations may provide information about:
How the program works
Important program components
Lessons learned
Potential implementation problems
Evaluation methods
Measurement instruments
Both published and unpublished evaluation reports can be useful.
Existing program documentation
For established programs, evaluators should review documents such as:
Strategic plans
Training materials
Meeting minutes
Federal requirements
State requirements
Local requirements
Program materials
Grant proposals
Grant proposals
Grant proposals can be especially useful because funders often require applicants to describe:
Program goals
Program strategies
Planned activities
However, evaluators must read them carefully because programs do not always distinguish clearly between goals and strategies.
For example:
“Provide families with healthy food alternatives”
may appear to be a goal, but it is actually a strategy.
The actual goal might be:
Increase healthy food intake
Improve eating habits
Improve health
The distinction is important because:
Strategy = what the program does
Goal = what the program intends to accomplish
Other information sources
Program and organizational websites or professional pages can also provide information about:
Program services
Activities
Mission
Intended outcomes
However, evaluators still need to separate descriptions of what the program does from descriptions of what the program is trying to accomplish.
Important distinction
For EMB-E:
Goals = what the program intends to accomplish
Strategies = what the program does to accomplish those goals
A program may describe its goals differently in its official materials. The evaluator is not necessarily trying to rewrite those official goals. Instead, the evaluator needs to define goals clearly enough to support evaluation planning.
5.2.2 Conversations, Interviews, and Group Discussions
Written materials alone may not provide a full picture of a program.
Evaluators should also speak with people who understand the program, including:
Program staff
Community members
Partner organizations
Staff from organizations that have implemented similar programs
Researchers or professionals familiar with the topic
The chapter encourages evaluators to actively reach out to knowledgeable individuals rather than assuming certain people are inaccessible.
Interviews and discussions
Even if the evaluator already knows the program well, they should still collect information from others.
Interviews and group discussions can help reveal:
How the program actually operates
What staff believe the program is trying to change
What activities occur in practice
What outcomes stakeholders expect
Instead of simply asking staff:
"What are the program's goals?"
it may be more useful to ask:
What is the program trying to change?
What happens if the program works?
What does the program do to create that change?
These conversations can reveal assumptions or expectations that may not appear in formal program documents.
5.3 Delineating Goals and Strategies
After developing a strong understanding of the program, the evaluator begins documenting it systematically.
There are three main tasks:
Describe program goals
Identify program strategies
Create a program statement
5.3.1 Documenting Program Goals
A program goal is a broad statement describing the program's desired result or what the program ultimately intends to accomplish.
Goals should reflect a shared understanding among stakeholders.
Questions that can help identify goals include:
What does the program intend to accomplish?
How would we know if the program worked?
If the program were successful, what would happen?
What would change?
Goals versus objectives
Goals
Broad
Long term
Represent overall program intent
Objectives
More specific
Usually short term or intermediate
Help explain the pathway toward the goal
At this stage, evaluators do not need every objective fully developed. Identifying the major long-term goals is sufficient.
Types of program goals
Programs may have one goal or several goals.
Goals can address changes in:
Behavior
Quality of life
Knowledge
Attitudes
Living conditions
Environmental conditions
Community involvement
Community safety
For example, a program might seek to:
Decrease risky behavior
Reduce arrests
Increase graduation rates
Improve community relations
Goals should be written as outcomes
Within EMB-E, goals should describe outcomes, not program activities.
Goal statements should usually begin with words showing a direction of change, such as:
Increase
Decrease
Improve
Reduce
Avoid vague action words such as:
Help
Show
Provide
Make
For example:
Weak goal:
"Provide employees with leadership training."
This describes what the program does.
Outcome-focused goal:
"Improve leadership effectiveness among new managers."
This describes what the program intends to accomplish.
The SMART framework is discussed later in the book. At this point, the focus is simply on writing long-term goals as clear outcomes.
The key question for identifying goals
Ask:
If the program were successful, what observable change would likely occur?
Program activities are only a means to an end.
The goal is the end.
The chapter compares this to a destination:
Goal = destination
Program design and strategies = route used to reach the destination
Program design, implementation, and evaluation should therefore remain centered on the intended goals rather than treating completion of activities as proof that the program succeeded.
5.3.2 Identifying Program Strategies
Once goals have been identified, evaluators determine the program's strategies.
Questions include:
How will the program accomplish its goals?
What strategies will be used?
What activities need to occur?
Where will those activities occur?
Possible settings include:
Community
Home
Workplace
Schools
Multiple settings
Examples of strategies
Strategies may involve:
Increasing access to financial resources
Increasing access to other resources
Support services
Professional guidance
Training
Skill development
Mentoring
Counseling
Strategies may:
Occur continuously, or
Be introduced at specific stages of the program.
Evaluators should identify all major strategies because later evaluation will examine whether those strategies contributed to program outcomes.
Existing versus new programs
For an existing program, document what the program currently does.
For a new program, document what the program proposes to do.
The same basic process applies in either situation:
Understand the program → identify goals → document strategies
Spending enough time understanding these strategies is critical for designing an effective evaluation.
Strategies are interconnected
Strategies should not simply be viewed as unrelated activities on a checklist.
Instead, they are generally:
Purposeful
Connected
Designed to work together
Directed toward one or more common goals
At this stage, evaluators primarily need to identify and document them. Their relationships and implementation will be examined more deeply later.
5.3.3 Creating a Program Statement
A program statement is a concise, one-sentence description of:
What the program intends to accomplish, and
What strategies it uses to accomplish it.
The format is:
The [program name] program intends to [goals] by [strategies].
Why program statements are useful
A strong program statement quickly communicates:
Why the program exists + what the program does
The goals should come first because outcomes are the main focus.
Strategies follow because they explain how the program intends to reach those outcomes.
Stakeholder involvement in program statements
The final program statement should be developed collaboratively with stakeholders.
An evaluator might create an initial draft using:
Program documents
Research
Previous evaluations
Stakeholder interviews
However, stakeholders should ultimately determine whether the statement accurately represents the program.
The program statement should therefore be:
Realistic
Accurate
Collaborative
Agreed upon
Once the program statement is established, the next task is to examine why the strategies are expected to produce the stated goals.
5.4 Explaining the Program Theory
After identifying goals and strategies, evaluators examine the assumptions connecting them.
These assumptions form the program theory.
Program theory
Program theory = the reasoning behind why the program should work.
More specifically, program theory explains why program strategies are expected to produce desired outcomes and eventually reach the program's goals.
Building program theory
Evaluators should take each strategy and ask:
Why should this strategy produce the intended outcome?
What changes should happen after this strategy is implemented?
What outcome should occur next?
How will that outcome eventually contribute to the larger goal?
A useful repeated question is:
If I implement this strategy or achieve this objective, what outcome should I see next?
This produces a chain of expected outcomes.
For example:
Strategy → Early outcome → Intermediate outcome → Later outcome → Long-term goal
The chapter's community example demonstrates this sequence:
Town meetings
→ access to meetings
→ residents attend
→ communication increases
→ sense of community belonging increases
→ community relations improve
Similarly:
Conflict-resolution training
→ access to training
→ residents participate
→ conflict-resolution skills improve
→ community relationships improve
Program theory therefore explains the assumed pathway connecting activities to eventual outcomes.
Evidence behind program theory
Evaluators should also investigate why they believe these relationships exist.
Ask whether assumptions are based on:
Strong scientific research
Previous evaluations
Emerging research
Professional experience
Personal experience
Untested ideas
This matters because the strength of the existing evidence should influence the evaluation design.
Strong existing evidence
A program supported by substantial previous research may not require the same level of exploratory evaluation.
Limited or emerging evidence
Innovative programs or programs based on emerging knowledge generally require more extensive and rigorous evaluation.
The less established the theory, the more important it becomes to carefully test whether the assumed pathways actually occur.
5.5 Determining Contextual Conditions
Programs do not operate in isolation. Most depend on outside resources or infrastructure.
These necessary outside resources are called contextual conditions.
Contextual conditions
A contextual condition is something necessary for the program to operate successfully but which is not part of the program itself.
Examples include:
Availability of trained counselors
Availability of skilled professionals
Facilities
Meeting spaces
Partner organizations
Transportation
Necessary equipment or materials
Existing infrastructure
Evaluators should clearly document these conditions when planning an evaluation.
Why contextual conditions matter
Suppose a program offers training but participants cannot reach the training facility because transportation is unavailable.
Transportation may originally have been considered an outside contextual condition.
However, if the lack of transportation threatens the program's effectiveness, program leaders might decide to make transportation part of the program strategy itself.
Understanding contextual conditions can therefore identify barriers that may need to be addressed.
Contextual Conditions vs. Contextual Influences
These concepts are related but different.
Contextual conditions
These are assumptions about the resources and infrastructure required for the program to work.
Examples:
Staffing
Facilities
Transportation
Equipment
Contextual influences
These are broader environmental factors that may influence program operations or outcomes.
The chapter identifies four categories through the PEST framework:
P = Political
E = Economic
S = Social
T = Technological
Although Chapter 5 primarily focuses on contextual conditions, evaluators and program staff should also consider broader contextual influences.
A PEST analysis can be conducted before implementation and periodically while the program is operating.
The chapter also connects this idea to systems thinking, which deliberately examines external forces that can affect a program.
5.6 Program Theory and Other Theories
The word theory is used differently across fields.
At its most basic level, a theory provides an explanation for why something happens or works the way it does.
A theory can be understood as a system of beliefs or propositions explaining how something works.
Theories usually develop through previous:
Research
Hypothesis testing
Evidence
Experience
Therefore, theories are generally more than simple guesses.
Innovation can also emerge from promising theories that have not yet been extensively tested.
Theories are dynamic
Theories should change as new knowledge becomes available.
Different theories have different levels of evidence supporting them.
The stronger and more extensively tested the evidence:
→ the greater confidence practitioners can have in using the theory.
However, limited evidence does not automatically mean a theory should be rejected.
An untested theory may lead to:
New ideas
New approaches
Innovative programs
The important issue is that programs based on less-tested theories need appropriately rigorous evaluation.
What Works Clearinghouse Evidence Tiers
The What Works Clearinghouse (WWC) classifies programs and interventions based on the strength of evidence supporting their effectiveness.
Tier 1: Strong Evidence
Requires evidence from at least one experimental evaluation.
Tier 2: Moderate Evidence
Requires evidence from at least one quasi-experimental evaluation.
Tier 3: Promising Evidence
Requires evidence from at least one rigorous correlational study.
Tier 4: Demonstrates a Rationale
The program is based on a rationale or theory supported by high-quality research.
The Every Student Succeeds Act (ESSA) uses similar levels of evidence. At its rationale level, the program should also have:
A well-developed logic model, and
An evaluation planned or already underway.
Programs based on relatively untested theory should ideally use rigorous evaluation designs capable of producing meaningful evidence about effectiveness.
Increasing evidence rigor
From lower to higher evidence:
Demonstrates a rationale
→ Logic model + research/evaluation support + study planned or underway
Promising evidence
→ Rigorous correlational study
Moderate evidence
→ Rigorous quasi-experimental study
Strong evidence
→ Rigorous experimental study
5.6.1 Program Theory Revisited
Most programs and policies are based on some type of theory, even when that theory is never clearly stated.
Evaluators should avoid assuming that a program works simply because:
It exists
People believe in it
It has been used for a long time
Programs intended to help people could potentially create unintended harm if their assumptions are incorrect.
Evaluation therefore systematically investigates:
What works
How it works
For whom it works
Under what conditions it works
Embedded evaluation can be thought of as a systematic way of testing the program theory.
Definition of program theory
Program theory:
Focuses on program goals.
Explains why the program should work.
Explains why strategies should produce goals.
Describes the outcomes expected to occur.
Identifies the assumptions connecting strategies with outcomes.
A program theory can be represented in different formats, including:
Tables
Narratives
Diagrams
Logic models
A logic model is one of the most common graphical representations of program theory.
Regardless of the format:
Program theory links what the program does to what the program ultimately intends to accomplish.
5.6.2 Theories of Change
A theory of change (ToC) describes how change is expected to occur.
It may be represented:
Narratively
Visually
Through a model
A theory of change generally connects:
Strategies → Outcomes → Goals
Theories of change may also be called causal models because they visually suggest causal connections between program activities and outcomes.
Important caution about causality
A theory of change does not prove causation.
It only represents the program's theorized relationships.
For example:
Training → increased skills → improved performance
The model suggests that training causes improved skills and performance.
However, the evaluation design determines whether an evaluator can actually make a causal claim.
Therefore:
Theory of Change = hypothesized relationship
Evaluation design = evidence used to test that relationship
Certain evaluation designs are better able to approximate cause-and-effect relationships than others.
Program theory and theory of change
Program theory is a type of theory of change.
A program logic model can provide a graphical representation of that theory.
Theories of change are especially useful because they help explain:
Why the program uses particular strategies
What outcomes are expected
How short-term and intermediate outcomes relate to long-term goals
What the evaluation should examine
However, theories of change focus on expected outcomes, not on the detailed sequence of how program services are delivered.
In other words:
Theory of change = Why should change happen?
It does not primarily explain:
Exactly how and when should services be delivered?
5.6.3 Theories of Action
A theory of action (ToA) explains the process through which change occurs.
It is essentially the operational side of a theory of change.
A ToA focuses on:
Program processes
Services
Activities
How strategies are implemented
When activities occur
How the program is delivered
Theory of Change vs. Theory of Action
Theory of Change | Theory of Action |
|---|---|
Explains why change should occur | Explains how change will occur |
Focuses on outcomes | Focuses on implementation |
Connects strategies to outcomes and goals | Connects program activities to delivery processes |
Outcome oriented | Operationally oriented |
Implementation theory
Implementation theory is a type of theory of action.
It focuses specifically on:
Program service delivery
Activities needed to meet objectives
What services should be delivered
At what stage services should be delivered
Program theory and implementation theory therefore complement one another:
Program theory = Why should this program work?
Implementation theory = How should the program be carried out?
Together, they provide a more complete understanding of program operation and effectiveness.
5.7 Considering Alternative Theories
After documenting the program's:
Goals
Strategies
Program statement
Program theory
the evaluator should reconsider the assumptions behind the theory.
Questions to ask include:
Why do we believe these strategies will produce these outcomes?
What evidence supports these assumptions?
Are they based on rigorous previous evaluations?
Are they based on broader research?
Are they based primarily on emerging evidence?
Are they based on professional experience?
Could another explanation be equally plausible?
Considering alternative theories can strengthen the evaluation because it prevents evaluators from automatically assuming the program's preferred explanation is correct.
Strength of evidence and evaluation rigor
The amount of existing evidence supporting a program's theory should influence how evaluation is conducted.
When theory has limited evidence
Evaluators should pay especially close attention to:
Program implementation
Early outcomes
Intermediate outcomes
Whether strategies are functioning as intended
The less evidence available to support the theory, the more carefully program implementation should be monitored.
Developmental evaluation
Developmental evaluation is particularly useful when:
A program is still being developed.
Evaluation findings will help shape its continued development.
The evaluator wants to understand how the program evolves over time.
When previous research contradicts the program theory
If strong previous evaluations suggest that the proposed theory may not work, evaluators should ask:
What is different about this program?
Why do stakeholders believe this version will succeed?
Are there alternative explanations for potential outcomes?
Documenting alternative theories can help evaluators later understand unexpected results.
Importantly, a theory does not need extensive existing evidence to be worthwhile.
A well-developed and clearly documented theory with little prior empirical evidence could eventually become the basis for innovation if a rigorous evaluation generates supportive evidence.