CH.5, Part 1: Understanding the Program

5.1 Embedded Evaluation

Chapter 5 begins the step-by-step explanation of the Embedded Evaluation (EMB-E) framework. The overall idea is that programs should continually learn from evaluation and use that information to improve. Program planning is therefore not something that happens only at the beginning. Programs operate in changing environments, so their design and implementation may need to change as new information becomes available.

Evaluation helps determine:

  • Whether a program is actually helping the people it intends to serve

  • How effective the program is

  • For whom the program is effective

  • Under what conditions it works

  • How and why the program produces its outcomes

Chapters 5 through 11 introduce a framework for integrating evaluation into program planning, implementation, and decision making so that evaluation supports continuous improvement.


5.1.1 Continuous Improvement Centered, Theory Based, Utilization Focused

Embedded Evaluation, or EMB-E

Embedded evaluation (EMB-E) is a comprehensive approach that incorporates evaluation directly into a program's design and operations.

EMB-E combines several evaluation perspectives:

  • Continuous improvement principles

    • Evaluation should help programs continually learn and improve.

  • Theory-based evaluation

    • Evaluation examines the reasoning behind how and why a program is expected to work.

  • Participatory evaluation

    • Stakeholders contribute to evaluation planning and decision making.

  • Utilization-focused evaluation

    • Evaluation should produce information that stakeholders can actually use.

Stakeholder partnerships are especially important because stakeholders help identify meaningful metrics and determine what information will be useful for program improvement.

EMB-E as both an approach and a framework

The EMB-E framework involves several connected activities:

  1. Define program goals and objectives as measurable outcomes.

  2. Explain the theory behind the program.

  3. Develop a logic model representing the program theory.

  4. Create indicators based on elements of the logic model.

  5. Develop an evaluation matrix.

  6. Use the evaluation matrix to guide data collection and analysis.

  7. Interpret findings with stakeholders.

  8. Develop recommendations intended to support program improvement.

The ultimate purpose is to create a continuous cycle in which evaluation findings inform future program decisions.


Five Steps of Embedded Evaluation

The EMB-E framework contains five major steps:

Step 1: DEFINE

What is the program?

The evaluator works to understand:

  • What the program does

  • What it intends to accomplish

  • Its goals and objectives

  • Its strategies and activities

Step 2: PLAN

How do I plan the evaluation?

Questions include:

  • How are program strategies connected to program goals?

  • What is the program's underlying logic?

  • What short-term and intermediate objectives lead toward long-term goals?

  • Is the program theory supported by research?

Step 3: IMPLEMENT

How do I evaluate the program?

This involves determining:

  • What questions the evaluation should answer

  • What indicators should be used

  • What evaluation methods are appropriate

  • What evaluation design is both rigorous and feasible

Step 4: INTERPRET

How do I interpret the results?

This includes decisions about:

  • Data collection

  • Data organization

  • Data management

  • Data analysis

Step 5: INFORM and REFINE

How do I use the results?

The evaluator considers:

  • What the results mean

  • Whether program goals were achieved

  • How the program could improve

  • How findings should be communicated

  • How to encourage stakeholders to actually use the findings

The framework also emphasizes refinement, meaning information from evaluation should continually feed back into the program and lead to improvement.


5.1.2 Dynamic and Cyclical

Although EMB-E contains numbered steps, the evaluation process is not linear.

The steps should not be treated like a checklist where one step is completed permanently before moving to the next.

Instead:

  • Evaluation is dynamic.

  • Program environments change.

  • Populations may change.

  • Program conditions may change.

  • Findings that apply to one population or context may not apply to another.

  • New information may require evaluators to revisit earlier decisions.

Evaluation should continue for as long as a program is operating.

Continuous improvement cycle

The EMB-E framework is based on repeated cycles of:

Data collection → data use → program refinement → new data collection

For example:

  • Problems discovered during implementation may require changes to the evaluation plan.

  • Interpretation of results might reveal that additional data need to be collected.

  • Findings could reveal that the program itself needs to be redefined.

Embedded evaluation can therefore help organizations both:

  • Improve programs that are working, and

  • Recognize when a program is not working and should potentially be discontinued.


5.1.3 EMB-E Steps and Nodes

The embedded evaluation model contains both steps and nodes.

A node represents the product or result produced by completing a step.

The general relationship is:

Step 1: Define → Program Logic

Step 2: Plan → Evaluation Design

Step 3: Implement → Evaluation Data

Step 4: Interpret → Evaluation Results

Step 5: Inform/Refine → Program Decisions and Improvements

However, refinement does not have to wait until Step 5.

For example:

  • Implementation problems could lead the evaluator back to Step 2.

  • Unexpected findings might require additional data collection.

  • Results might raise questions about how the program itself was originally defined.

The model is therefore intended to guide evaluators rather than force them into an inflexible process.

Evaluators must remain flexible and responsive to:

  • What they learn about the program

  • Stakeholder needs

  • Program conditions

  • Information needed by leadership for decision making


5.2 Understanding the Program

The first major task in evaluation is to understand the program being evaluated.

Evaluators should not assume that what they already know about a program is sufficient.

Instead, they should gather information from multiple sources to understand:

  • Different stakeholder perspectives

  • Program expectations

  • Program operations

  • Program history

  • How the program is actually supposed to work

Understanding the program is the foundation for evaluating whether it is effective.

Two primary components

Understanding a program requires identifying:

1. What the program wants to accomplish

These are the program's:

  • Long-term goals

  • Objectives

2. What the program does to accomplish those goals

These are the program's:

  • Strategies

  • Activities

The evaluator must also understand the assumed connection between these two components.

Key terminology

Goals

  • Represent the long-term "why" of the program.

  • Describe what the program ultimately intends to accomplish.

Objectives

  • Represent outcomes that occur before the long-term goal is achieved.

Strategies

  • Groups of activities intended to produce program outcomes.

Activities

  • Specific things the program does.

The chapter sometimes uses strategies and activities interchangeably, although technically a strategy may consist of several related activities.


5.2.1 Previous Evaluations and Program Documentation

One way to understand a program is to examine information that already exists.

Previous evaluations

Evaluators should search for previous evaluations of:

  • The same program

  • Similar programs

  • Similar interventions

Previous evaluations may provide information about:

  • How the program works

  • Important program components

  • Lessons learned

  • Potential implementation problems

  • Evaluation methods

  • Measurement instruments

Both published and unpublished evaluation reports can be useful.


Existing program documentation

For established programs, evaluators should review documents such as:

  • Strategic plans

  • Training materials

  • Meeting minutes

  • Federal requirements

  • State requirements

  • Local requirements

  • Program materials

  • Grant proposals

Grant proposals

Grant proposals can be especially useful because funders often require applicants to describe:

  • Program goals

  • Program strategies

  • Planned activities

However, evaluators must read them carefully because programs do not always distinguish clearly between goals and strategies.

For example:

“Provide families with healthy food alternatives”

may appear to be a goal, but it is actually a strategy.

The actual goal might be:

  • Increase healthy food intake

  • Improve eating habits

  • Improve health

The distinction is important because:

Strategy = what the program does

Goal = what the program intends to accomplish


Other information sources

Program and organizational websites or professional pages can also provide information about:

  • Program services

  • Activities

  • Mission

  • Intended outcomes

However, evaluators still need to separate descriptions of what the program does from descriptions of what the program is trying to accomplish.

Important distinction

For EMB-E:

Goals = what the program intends to accomplish

Strategies = what the program does to accomplish those goals

A program may describe its goals differently in its official materials. The evaluator is not necessarily trying to rewrite those official goals. Instead, the evaluator needs to define goals clearly enough to support evaluation planning.


5.2.2 Conversations, Interviews, and Group Discussions

Written materials alone may not provide a full picture of a program.

Evaluators should also speak with people who understand the program, including:

  • Program staff

  • Community members

  • Partner organizations

  • Staff from organizations that have implemented similar programs

  • Researchers or professionals familiar with the topic

The chapter encourages evaluators to actively reach out to knowledgeable individuals rather than assuming certain people are inaccessible.

Interviews and discussions

Even if the evaluator already knows the program well, they should still collect information from others.

Interviews and group discussions can help reveal:

  • How the program actually operates

  • What staff believe the program is trying to change

  • What activities occur in practice

  • What outcomes stakeholders expect

Instead of simply asking staff:

"What are the program's goals?"

it may be more useful to ask:

  • What is the program trying to change?

  • What happens if the program works?

  • What does the program do to create that change?

These conversations can reveal assumptions or expectations that may not appear in formal program documents.


5.3 Delineating Goals and Strategies

After developing a strong understanding of the program, the evaluator begins documenting it systematically.

There are three main tasks:

  1. Describe program goals

  2. Identify program strategies

  3. Create a program statement


5.3.1 Documenting Program Goals

A program goal is a broad statement describing the program's desired result or what the program ultimately intends to accomplish.

Goals should reflect a shared understanding among stakeholders.

Questions that can help identify goals include:

  • What does the program intend to accomplish?

  • How would we know if the program worked?

  • If the program were successful, what would happen?

  • What would change?


Goals versus objectives

Goals

  • Broad

  • Long term

  • Represent overall program intent

Objectives

  • More specific

  • Usually short term or intermediate

  • Help explain the pathway toward the goal

At this stage, evaluators do not need every objective fully developed. Identifying the major long-term goals is sufficient.


Types of program goals

Programs may have one goal or several goals.

Goals can address changes in:

  • Behavior

  • Quality of life

  • Knowledge

  • Attitudes

  • Living conditions

  • Environmental conditions

  • Community involvement

  • Community safety

For example, a program might seek to:

  • Decrease risky behavior

  • Reduce arrests

  • Increase graduation rates

  • Improve community relations


Goals should be written as outcomes

Within EMB-E, goals should describe outcomes, not program activities.

Goal statements should usually begin with words showing a direction of change, such as:

  • Increase

  • Decrease

  • Improve

  • Reduce

Avoid vague action words such as:

  • Help

  • Show

  • Provide

  • Make

For example:

Weak goal:
"Provide employees with leadership training."

This describes what the program does.

Outcome-focused goal:
"Improve leadership effectiveness among new managers."

This describes what the program intends to accomplish.

The SMART framework is discussed later in the book. At this point, the focus is simply on writing long-term goals as clear outcomes.


The key question for identifying goals

Ask:

If the program were successful, what observable change would likely occur?

Program activities are only a means to an end.

The goal is the end.

The chapter compares this to a destination:

  • Goal = destination

  • Program design and strategies = route used to reach the destination

Program design, implementation, and evaluation should therefore remain centered on the intended goals rather than treating completion of activities as proof that the program succeeded.


5.3.2 Identifying Program Strategies

Once goals have been identified, evaluators determine the program's strategies.

Questions include:

  • How will the program accomplish its goals?

  • What strategies will be used?

  • What activities need to occur?

  • Where will those activities occur?

Possible settings include:

  • Community

  • Home

  • Workplace

  • Schools

  • Multiple settings


Examples of strategies

Strategies may involve:

  • Increasing access to financial resources

  • Increasing access to other resources

  • Support services

  • Professional guidance

  • Training

  • Skill development

  • Mentoring

  • Counseling

Strategies may:

  • Occur continuously, or

  • Be introduced at specific stages of the program.

Evaluators should identify all major strategies because later evaluation will examine whether those strategies contributed to program outcomes.

Existing versus new programs

For an existing program, document what the program currently does.

For a new program, document what the program proposes to do.

The same basic process applies in either situation:

Understand the program → identify goals → document strategies

Spending enough time understanding these strategies is critical for designing an effective evaluation.

Strategies are interconnected

Strategies should not simply be viewed as unrelated activities on a checklist.

Instead, they are generally:

  • Purposeful

  • Connected

  • Designed to work together

  • Directed toward one or more common goals

At this stage, evaluators primarily need to identify and document them. Their relationships and implementation will be examined more deeply later.


5.3.3 Creating a Program Statement

A program statement is a concise, one-sentence description of:

  1. What the program intends to accomplish, and

  2. What strategies it uses to accomplish it.

The format is:

The [program name] program intends to [goals] by [strategies].


Why program statements are useful

A strong program statement quickly communicates:

Why the program exists + what the program does

The goals should come first because outcomes are the main focus.

Strategies follow because they explain how the program intends to reach those outcomes.


Stakeholder involvement in program statements

The final program statement should be developed collaboratively with stakeholders.

An evaluator might create an initial draft using:

  • Program documents

  • Research

  • Previous evaluations

  • Stakeholder interviews

However, stakeholders should ultimately determine whether the statement accurately represents the program.

The program statement should therefore be:

  • Realistic

  • Accurate

  • Collaborative

  • Agreed upon

Once the program statement is established, the next task is to examine why the strategies are expected to produce the stated goals.


5.4 Explaining the Program Theory

After identifying goals and strategies, evaluators examine the assumptions connecting them.

These assumptions form the program theory.

Program theory

Program theory = the reasoning behind why the program should work.

More specifically, program theory explains why program strategies are expected to produce desired outcomes and eventually reach the program's goals.


Building program theory

Evaluators should take each strategy and ask:

  • Why should this strategy produce the intended outcome?

  • What changes should happen after this strategy is implemented?

  • What outcome should occur next?

  • How will that outcome eventually contribute to the larger goal?

A useful repeated question is:

If I implement this strategy or achieve this objective, what outcome should I see next?

This produces a chain of expected outcomes.

For example:

Strategy → Early outcome → Intermediate outcome → Later outcome → Long-term goal

The chapter's community example demonstrates this sequence:

Town meetings
→ access to meetings
→ residents attend
→ communication increases
→ sense of community belonging increases
→ community relations improve

Similarly:

Conflict-resolution training
→ access to training
→ residents participate
→ conflict-resolution skills improve
→ community relationships improve

Program theory therefore explains the assumed pathway connecting activities to eventual outcomes.


Evidence behind program theory

Evaluators should also investigate why they believe these relationships exist.

Ask whether assumptions are based on:

  • Strong scientific research

  • Previous evaluations

  • Emerging research

  • Professional experience

  • Personal experience

  • Untested ideas

This matters because the strength of the existing evidence should influence the evaluation design.

Strong existing evidence

A program supported by substantial previous research may not require the same level of exploratory evaluation.

Limited or emerging evidence

Innovative programs or programs based on emerging knowledge generally require more extensive and rigorous evaluation.

The less established the theory, the more important it becomes to carefully test whether the assumed pathways actually occur.


5.5 Determining Contextual Conditions

Programs do not operate in isolation. Most depend on outside resources or infrastructure.

These necessary outside resources are called contextual conditions.

Contextual conditions

A contextual condition is something necessary for the program to operate successfully but which is not part of the program itself.

Examples include:

  • Availability of trained counselors

  • Availability of skilled professionals

  • Facilities

  • Meeting spaces

  • Partner organizations

  • Transportation

  • Necessary equipment or materials

  • Existing infrastructure

Evaluators should clearly document these conditions when planning an evaluation.

Why contextual conditions matter

Suppose a program offers training but participants cannot reach the training facility because transportation is unavailable.

Transportation may originally have been considered an outside contextual condition.

However, if the lack of transportation threatens the program's effectiveness, program leaders might decide to make transportation part of the program strategy itself.

Understanding contextual conditions can therefore identify barriers that may need to be addressed.


Contextual Conditions vs. Contextual Influences

These concepts are related but different.

Contextual conditions

These are assumptions about the resources and infrastructure required for the program to work.

Examples:

  • Staffing

  • Facilities

  • Transportation

  • Equipment

Contextual influences

These are broader environmental factors that may influence program operations or outcomes.

The chapter identifies four categories through the PEST framework:

  • P = Political

  • E = Economic

  • S = Social

  • T = Technological

Although Chapter 5 primarily focuses on contextual conditions, evaluators and program staff should also consider broader contextual influences.

A PEST analysis can be conducted before implementation and periodically while the program is operating.

The chapter also connects this idea to systems thinking, which deliberately examines external forces that can affect a program.


5.6 Program Theory and Other Theories

The word theory is used differently across fields.

At its most basic level, a theory provides an explanation for why something happens or works the way it does.

A theory can be understood as a system of beliefs or propositions explaining how something works.

Theories usually develop through previous:

  • Research

  • Hypothesis testing

  • Evidence

  • Experience

Therefore, theories are generally more than simple guesses.

Innovation can also emerge from promising theories that have not yet been extensively tested.


Theories are dynamic

Theories should change as new knowledge becomes available.

Different theories have different levels of evidence supporting them.

The stronger and more extensively tested the evidence:

→ the greater confidence practitioners can have in using the theory.

However, limited evidence does not automatically mean a theory should be rejected.

An untested theory may lead to:

  • New ideas

  • New approaches

  • Innovative programs

The important issue is that programs based on less-tested theories need appropriately rigorous evaluation.


What Works Clearinghouse Evidence Tiers

The What Works Clearinghouse (WWC) classifies programs and interventions based on the strength of evidence supporting their effectiveness.

Tier 1: Strong Evidence

Requires evidence from at least one experimental evaluation.

Tier 2: Moderate Evidence

Requires evidence from at least one quasi-experimental evaluation.

Tier 3: Promising Evidence

Requires evidence from at least one rigorous correlational study.

Tier 4: Demonstrates a Rationale

The program is based on a rationale or theory supported by high-quality research.

The Every Student Succeeds Act (ESSA) uses similar levels of evidence. At its rationale level, the program should also have:

  • A well-developed logic model, and

  • An evaluation planned or already underway.

Programs based on relatively untested theory should ideally use rigorous evaluation designs capable of producing meaningful evidence about effectiveness.

Increasing evidence rigor

From lower to higher evidence:

Demonstrates a rationale
→ Logic model + research/evaluation support + study planned or underway

Promising evidence
→ Rigorous correlational study

Moderate evidence
→ Rigorous quasi-experimental study

Strong evidence
→ Rigorous experimental study


5.6.1 Program Theory Revisited

Most programs and policies are based on some type of theory, even when that theory is never clearly stated.

Evaluators should avoid assuming that a program works simply because:

  • It exists

  • People believe in it

  • It has been used for a long time

Programs intended to help people could potentially create unintended harm if their assumptions are incorrect.

Evaluation therefore systematically investigates:

  • What works

  • How it works

  • For whom it works

  • Under what conditions it works

Embedded evaluation can be thought of as a systematic way of testing the program theory.

Definition of program theory

Program theory:

  • Focuses on program goals.

  • Explains why the program should work.

  • Explains why strategies should produce goals.

  • Describes the outcomes expected to occur.

  • Identifies the assumptions connecting strategies with outcomes.

A program theory can be represented in different formats, including:

  • Tables

  • Narratives

  • Diagrams

  • Logic models

A logic model is one of the most common graphical representations of program theory.

Regardless of the format:

Program theory links what the program does to what the program ultimately intends to accomplish.


5.6.2 Theories of Change

A theory of change (ToC) describes how change is expected to occur.

It may be represented:

  • Narratively

  • Visually

  • Through a model

A theory of change generally connects:

Strategies → Outcomes → Goals

Theories of change may also be called causal models because they visually suggest causal connections between program activities and outcomes.

Important caution about causality

A theory of change does not prove causation.

It only represents the program's theorized relationships.

For example:

Training → increased skills → improved performance

The model suggests that training causes improved skills and performance.

However, the evaluation design determines whether an evaluator can actually make a causal claim.

Therefore:

Theory of Change = hypothesized relationship

Evaluation design = evidence used to test that relationship

Certain evaluation designs are better able to approximate cause-and-effect relationships than others.


Program theory and theory of change

Program theory is a type of theory of change.

A program logic model can provide a graphical representation of that theory.

Theories of change are especially useful because they help explain:

  • Why the program uses particular strategies

  • What outcomes are expected

  • How short-term and intermediate outcomes relate to long-term goals

  • What the evaluation should examine

However, theories of change focus on expected outcomes, not on the detailed sequence of how program services are delivered.

In other words:

Theory of change = Why should change happen?

It does not primarily explain:

Exactly how and when should services be delivered?


5.6.3 Theories of Action

A theory of action (ToA) explains the process through which change occurs.

It is essentially the operational side of a theory of change.

A ToA focuses on:

  • Program processes

  • Services

  • Activities

  • How strategies are implemented

  • When activities occur

  • How the program is delivered

Theory of Change vs. Theory of Action

Theory of Change

Theory of Action

Explains why change should occur

Explains how change will occur

Focuses on outcomes

Focuses on implementation

Connects strategies to outcomes and goals

Connects program activities to delivery processes

Outcome oriented

Operationally oriented


Implementation theory

Implementation theory is a type of theory of action.

It focuses specifically on:

  • Program service delivery

  • Activities needed to meet objectives

  • What services should be delivered

  • At what stage services should be delivered

Program theory and implementation theory therefore complement one another:

Program theory = Why should this program work?

Implementation theory = How should the program be carried out?

Together, they provide a more complete understanding of program operation and effectiveness.


5.7 Considering Alternative Theories

After documenting the program's:

  • Goals

  • Strategies

  • Program statement

  • Program theory

the evaluator should reconsider the assumptions behind the theory.

Questions to ask include:

  • Why do we believe these strategies will produce these outcomes?

  • What evidence supports these assumptions?

  • Are they based on rigorous previous evaluations?

  • Are they based on broader research?

  • Are they based primarily on emerging evidence?

  • Are they based on professional experience?

  • Could another explanation be equally plausible?

Considering alternative theories can strengthen the evaluation because it prevents evaluators from automatically assuming the program's preferred explanation is correct.


Strength of evidence and evaluation rigor

The amount of existing evidence supporting a program's theory should influence how evaluation is conducted.

When theory has limited evidence

Evaluators should pay especially close attention to:

  • Program implementation

  • Early outcomes

  • Intermediate outcomes

  • Whether strategies are functioning as intended

The less evidence available to support the theory, the more carefully program implementation should be monitored.

Developmental evaluation

Developmental evaluation is particularly useful when:

  • A program is still being developed.

  • Evaluation findings will help shape its continued development.

  • The evaluator wants to understand how the program evolves over time.



When previous research contradicts the program theory

If strong previous evaluations suggest that the proposed theory may not work, evaluators should ask:

  • What is different about this program?

  • Why do stakeholders believe this version will succeed?

  • Are there alternative explanations for potential outcomes?

Documenting alternative theories can help evaluators later understand unexpected results.

Importantly, a theory does not need extensive existing evidence to be worthwhile.

A well-developed and clearly documented theory with little prior empirical evidence could eventually become the basis for innovation if a rigorous evaluation generates supportive evidence.