Chapter 4 - Intervention and Evaluation
Design of Interventions
Interventions are strategies intended to influence behavior and improve functioning regarding social or practical problems.
Some interventions aim to increase knowledge, awareness (e.g., recycling benefits), or change attitudes (e.g., support for recycling).
The ultimate goal of most interventions is behavior change (e.g., increased recycling).
Interventions can be conducted at different levels of analysis:
Individual
Group
Organization
Community
Ultimately, changes occur when individual members change in some way.
Two broad types of intervention:
Personal: Carried out in daily life to improve one's own or others' circumstances, e.g., waitpersons improving their tips.
Programmatic: Organized activities designed to reach certain objectives and address social or practical problems to prevent, reduce, or eliminate their negative consequences. Terms "intervention" and "program" are used interchangeably.
*Trial interventions: Implemented to determine if interventions have intended positive consequences (program efficacy studies).Researchers design a study to test an intervention strategy (e.g., Sherif, 1966b; Hodges et al., 2000).
Organizations conduct a pilot program before permanent implementation or wider scale.
Key Tasks in Intervention Design and Delivery
The intervention design and implementation process involves four steps:
Identifying a Problem
Arriving at a Solution
Setting Goals and Designing the Intervention
Implementing the Intervention (Oskamp & Schultz, 1997)
Step 1: Identifying a Problem
Programs are initiated to address social or practical problems.
The first step is to identify the existence and severity of a problem.
Stakeholders define a problem.
Stakeholders: Individuals or groups with a vested interest in the program's development who may be affected by it.
Include potential recipients, program funders, administrators, managers, and frontline staff.
Difficulties arise when stakeholders disagree about the existence, seriousness, or priority of a problem.
Needs assessment: Process of establishing whether a need or problem warrants program development.
May be informal based on experience, or formal relying on systematic research procedures.
Problems can be investigated using qualitative and/or quantitative procedures (e.g., interviews, questionnaires).
A formal needs assessment gauges the availability of existing programs/services and possible barriers to or gaps in service.
Step 2: Arriving at a Solution
Ascertaining a problem is different than determining how best to address it.
Identify the factors responsible for causing the problem.
Distinguish between precipitating factors (trigger the problem) and perpetuating factors (sustain the problem).
The factors that lead to a problem aren't always involved in its continuation.
Find out if interventions addressing the same needs already exist through a literature review.
Solutions should be based on relevant social psychological theory and research evidence, as well as theory and evidence from any other relevant field.
Ross, Lepper, and Ward (2010) identified the "nonobviousness" of experimental outcomes as an important theme of social psychology research, extending it to intervention strategies.
Ross and Nisbett (1991) cautioned against interventions based on conventional lay understanding and intuition because "lay predictions are often both wrong and too confidently made", instead supplement common sense with strategies from social psychology.
Smoking prevention intervention example based on McGuire's (1964) research on the "inoculation effect".
McGuire discovered that being immunized against attacks on beliefs is possible.
McAlister, Perry, and Maccoby (1980) developed a program using role-playing to help students resist peer pressure to smoke.
For clarity, solutions should be expressed as intervention hypotheses (Lodzinski, 2003).
Intervention hypotheses are "if–then" statements summarizing the intervention and expected outcomes.
Example: "If seventh-grade students are provided the opportunity to rehearse rebuttals to inducements to smoke in realistic role-playing, peer pressure scenarios, then in their daily lives they will be more likely to resist pressures to smoke and less likely to begin smoking."
Step 3: Setting Goals and Designing the Intervention
Develop program activities, which are the specific components and procedures of the program.
Set the goals and objectives of the intervention, which guides the selection of program activities.
Goals refer to the ultimate or long-term outcomes that one hopes to accomplish through an intervention.
Objectives refer to short-term outcomes and intermediate-term changes that occur as a result of the intervention and are required for the attainment of the program goals.
Goals refer to the ends, whereas objectives refer to the means or steps by which the ends are achieved.
Stakeholders may use the acronym SMART to develop outcomes that have five criteria: Specific, Measureable, Achievable, Relevant, and Timebound.
Determine the program activities. Ask the question: What objective (and ultimate goal) will the proposed activity help to meet?
The process of specifying program components requires a sound rationale, often referred to as a program logic model.
A program logic model is an explanation or a blueprint of how the program activities lead to the attainment of the program objectives, and in turn, how the objectives logically and operationally contribute to the eventual achievement of the program goal(s) (Wholey, 1983).
Logic models vary in complexity and detail, but all of them stress a "cause and effect" flow as expressed in the intervention hypothesis.
The use of a program logic model ensures a careful theoretically and empirically based articulation of the program and increases the likelihood of its success.
Some evaluators have begun to employ the notion of a theory of change model instead of program logic to underscore the need to make explicit an intervention’s underlying theory (Mackinnon, Amott, & McGarvey, 2006; Organizational Research Services, 2004).
Step 4: Implementing the Intervention
Implementation refers to the actual process of enacting the intervention activities.
Practical details might include securing an appropriate facility, hiring staff members, ensuring adequate training, and developing things, such as operating budgets, management structures, job descriptions, performance appraisal methods, promotional strategies, and cross-agency referral protocols.
An intervention always should be designed and implemented in such a way that its degree of effectiveness can be evaluated.
Evaluation of Interventions
Evaluation: Applied inquiry process for collecting and synthesizing evidence that culminates in conclusions about the state of affairs, value, merit, worth, significance, or quality of a program, person, policy, proposal, or plan.
Evaluation encompasses both an empirical aspect and a normative aspect.
Evaluation is essential to ensuring financial responsibility.
A good intervention design includes a plan for program evaluation (Lodzinski, 1995).
Testimony to the importance of evaluating interventions is the existence of a separate field in the social sciences called evaluation research (or program evaluation).
Reasons for Evaluating Interventions
The effectiveness of all programmatic interventions should be subjected to evaluation.
A major reason is scientific: To test the theoretical assumptions underlying the intervention.
An ethical one: Program implementers have an ethical responsibility to determine whether their recipients are receiving the intended benefits, and whether they are experiencing any unintended undesirable consequences.
The need for financial accountability: Demonstrations of program effectiveness are needed for the sake of the funders of programs.
Moreover, even if a program is found to be effective in accomplishing its goals, the program might be too costly.
Program development may be viewed as the overriding reason for evaluation research.
Ineffective Interventions
Programs must be subjected to evaluation research because their success cannot be assumed.
The intervention world is replete with ineffective programs.
Systematic review of "Scared Straight" programs found that not only do the programs not deter criminal behavior, but they actually lead to an increase in offending behavior.
The Cambridge–Somerville project was a dismal failure despite being a multifaceted intervention; highlights the notion that an apparently well-designed and intuitively compelling intervention can fail.
Possible explanations for the failure of the Cambridge–Somerville project include:
The situational factors that were manipulated as the intervention were “trivial” compared with the environmental forces that the boys faced on an ongoing basis.
Being identified with the program might have had a stigmatizing effect on the boys.
The usual community sources of help might have been less likely to provide their assistance.
One is that the theoretical rationale—the program logic model—may be inadequate and require revision.
A second is that the program might not be implemented as designed (see next section on process evaluation) despite a sound theoretical and research basis.
A third cause of program failure can be the operation of reactance in program recipients (Rothman, Haddock, & Schwarz, 2001).
Reactance refers to the idea that when a source of influence threatens people’s sense of freedom to think or behave as they see fit, people will act against the influence to protect their freedom (Brehm & Brehm, 1981).
A fourth reason for program failure may stem from an incompatibility between program design and cultural context.
Types of Evaluation
Two main types of program evaluation:
Process Evaluation (Formative Evaluation): Determines whether the program has reached its target audience and whether the program activities have been implemented in the prescribed manner; concerned with whether the program is implemented as planned.
Outcome Evaluation (Summative Evaluation): Assesses how well a program meets its objectives and goals; concerned with whether the hypothesized improvement in functioning occurs among the recipients of the program as a result of exposure to its activities.
The results of a program evaluation will often lead one to revisit and revise the program logic model, and accordingly make changes in the goals, objectives, and activities.
Evaluators also have recognized the important role evaluation can play in supporting the process of innovation and learning within an organization: developmental evaluation (Patton, 1994).
Importance of Research Design in Evaluating Interventions
Experimental designs are the "gold standard" in evaluation as they allow for heightened confidence in internal validity.
Potential program participants are randomly assigned to either the intervention group or a “no intervention” control group.
Experimentation frequently is not feasible for a variety of practical reasons.
Alternative research designs like quasi-experiments can be valuable.
The use of qualitative research methods has also gained momentum in the evaluation field.
Many evaluators use a mixed-method approach that incorporates both quantitative and qualitative methods providing a more complete picture of the intervention and its effects.
Evidence-Based Interventions
Evidence-based interventions or "best practices" are continually increasingly emphasized.
Many books and clearinghouses on evidence-based practices across many disciplines and social problems.
Some organizations have also introduced evidence-based decision-making tools that guide practitioners to critically assess research and apply research findings to their program decisions.
Biglan and colleagues recommended standards that scientific organizations should apply before promoting programs and policies directed toward the prevention of behavior problems in youth (Biglan, Mrazek, Carnine, & Flay, 2003).
An Intervention Example: Reducing Alcohol Problems on Campus
An intervention at Northern Illinois University (NIU) aimed to reduce high-risk drinking among students and reduce the incidence of injuries due to alcohol consumption.
Identifying the Problem
Alcohol consumption is a significant problem on many college campuses.
Harvard School of Public Health survey showed that 44% of students had engaged in "binge drinking" (five or more drinks in a row) during the two weeks prior to the survey.
Traditional methods of intervention, such as alcohol education and awareness campaigns, have little research evidence that education-oriented programs reduce alcohol consumption among college students (Moskowitz, 1989).
NIU found that traditional methods did not reduce drinking rates or alcohol-related injury rates among its students.
Developing a Solution: Forming the Intervention Hypothesis
To reduce alcohol consumption among NIU students, an intervention that represented an application of social norm theory was designed (Berkowitz, 2004; Perkins, 2003).
Fundamental to social norm theory is the central role that norms play in people’s lives.
Norms refer to shared beliefs about which behaviors are acceptable and which behaviors are not acceptable for members of a given group to engage in.
Social norm theory is based on the following three principles:
Individuals tend to conform to what they perceive to be the norm for a particular behavior.
In some instances, individuals may behaviorally conform to misperceived norms.
If misperceptions of norms are corrected, individuals will change their behavior to agree with the corrected perceptions.
Two cognitive errors that can lead people to misperceive norms:
False consensus: The tendency for people to believe that others are like them when in fact they are not (Ross, Green, & House, 1977).
Pluralistic ignorance: Occurs when the majority of individuals incorrectly assume that others’ behavior or thinking is more different from theirs than it actually is (Toch & Klofas, 1984).
Survey studies on a variety of college campuses have shown that these two types of misperceptions, especially pluralistic ignorance, are very common.
Social norm theory leads directly to the hypothesis that if misperceptions of the drinking norm are corrected, levels of high-risk drinking will decline. Therefore, NIU used the following intervention hypothesis for the design of its program: If NIU students are led to perceive the campus norm for drinking levels more accurately (i.e., the levels are lower than believed), drinking levels among students should decrease.
Goal Setting and Designing the Intervention
The main goals of the intervention were to reduce high-risk drinking and alcohol-related injuries among students at NIU.
The objective of the intervention was to reduce the misperception of the amount of drinking on campus.
The campaign plan was to target all students who drank alcohol.
Four rules used to guide the development of the campaign message to the student body were:
Keep it simple
Tell the truth
Be consistent
Highlight the norm of moderate drinking.
The message was that most NIU students (55%) drink five or fewer drinks when they party.
The main program activity would be to use a mass media campaign.
The print media campaign included campus newspaper advertisements, a campus newspaper column, press releases, flyers, and posters.
Implementing the Intervention
The media campaign was first implemented at the beginning of the fall semester in 1989.
The long-term intervention strategy was to initiate the media campaign at the beginning of each fall semester and to keep it highly visible until spring break.
This social norm-based intervention has been ongoing at NIU since its inception during the early 1990s.
Evaluating the Intervention
An outcome evaluation of the social norm media campaign at NIU was conducted to determine whether the intervention was able to reach both its main objective and its two key goals.
The evaluation sought to answer three questions:
Did the perceived rate of high-risk drinking among peers decrease to a more accurate perception?
Did the rate of actual high-risk drinking decrease?
Did the rate of alcohol-related injuries decrease?
Baseline information refers to data that are collected on the target population prior to an intervention (i.e., the pretest) and that are compared with data collected after the intervention has been implemented (i.e., the posttest).
This evaluation study demonstrates a quasi-experimental design because a nonequivalent comparison group of U.S. college students was used.
From 1989 to 1990, after the first year of implementation, researchers found that there was a 12% reduction in perceived high-risk drinking and an 8% reduction in actual high-risk drinking, but no significant reduction in alcohol-related injuries to self or others.
Over a span of 10 years using the social norm media campaign, NIU experienced a 37% reduction in perceived high-risk drinking, a 30% reduction in actual high-risk drinking, and a 20% reduction in alcohol-related injuries to self or others (Haines, 2003).
These data suggest that the NIU social norm intervention has been very successful in reaching its goals.
Incorporating Qualitative Methods to Design and Evaluate Interventions
Qualitative research methods can help design and evaluate interventions.
To illustrate, de Visser and colleagues (2015) conducted an intervention in England to reduce alcohol consumption and alcohol-related harm in young people.
The design of the intervention involved both qualitative and quantitative research methods.
*The researchers created videos promoting the idea of the sweet spot. The student participants felt that the “sweet spot” message was realistic and useful, and they felt that it would help young people resist the pressure to binge drink
Further Applications of Social Norm Theory
The social norm approach has been used to reduce high-risk drinking among students at many other colleges and universities across the United States (Berkowitz, 2004).
Social norm theory also has been applied to other social issues where individuals tend to misperceive the norm, including for example, cigarette smoking, driving while intoxicated, seatbelt use, gambling, tax compliance, risky sexual behavior, and energy conservation.
The application of social norm theory in programmatic interventions that successfully reduce a variety of high-risk behaviors aptly demonstrates the potential for the development of beneficial interventions based on social psychological theory and research.
Other Interventions
Examples of Other Interventions
Among the many excellent examples of how social psychological theory can be applied successfully to the amelioration of social problems is the work of Stice and colleagues on eating disorders.
One of the most effective interventions, developed by Stice and his co-researchers, relies on the application of Festinger’s (1957) cognitive dissonance theory (discussed in Chapter 2).
Another example is from Hansen, Meissler, and Ovens (2000) used aspects
of social learning theory and the fundamental
attribution error to create group play therapy pro-
grams for children with attention deficit/
hyperactivity disorder (ADHD).
Complex Problems Require Complex Solutions
Many of the most serious social issues confronted by social scientists reflect very complex problems that are embedded in a highly interconnected cluster of causal variables, and thus are apt to be extremely challenging to solve.
Highly complex problems tend to require resource-intensive, multifaceted holistic interventions that are able to take into consideration the interconnectedness of multiple key variables and address multiple causes.
Two intervention efforts that have attempted to take a more holistic approach:
Comprehensive community initiatives
Wraparound initiatives
For the Applied Social Psychologist, It Is Not Enough to Know Only That an Intervention Has Worked
The applied social psychologist involved in designing and evaluating interventions may be the only stakeholder who is particularly concerned about understanding the intervening processes that link intervention to outcome.
*As a scientist, it is necessary for the applied social psychologist to avoid giving insufficient attention to understanding why an intervention works and why it does not work.
Influencing Social Policy
This chapter would be remiss if it did not recognize another avenue of application—influencing the development of social policy.
The potential of psychology and other social sciences to contribute to public policy has long been recognized.
Fifteen years ago, two recent American Psychological Association (APA) presidents, DeLeon (2002) and Zimbardo (2002b), challenged psychologists to become more involved in public policy development.
Probably the best-known example of social psychological research having a role in policy formation occurred in 1954 when the U.S. Supreme Court, in its Brown v. Board of Education ruling, struck down the 1892 Plessy v. Ferguson separate but equal doctrine, thereby making racial segregation in public schools unconstitutional (Benjamin & Crouse, 2002).
Another example of social science research contributing to policy development comes from
the criminal justice field in North America,
which during the past 25 years has witnessed a
shift in philosophy and operational strategy.
The difference between
policy and intervention is that policy refers to a
general course of action endorsed by an organi-
zation, whereas an intervention refers to the
specific concrete action(s) (i.e., program activi-
ties) that the organization chooses to take to
implement the policy.
Intervention Issues
Process Issues
All interventions have to operate within constraints, and it is sometimes very challenging to both work within these constraints and maintain the integrity and effectiveness of the intervention. The most prevalent constraint is budgetary. Other constraints are subtler.
Interventions must be paid for, and funding typically is rooted in policy directions that in turn are embedded in the funder’s broader political philosophy or ideology (whether conservative or liberal).
*Intervention design is a collaborative process in which the responsibility always is shared among stakeholders.
Ethical Issues
Ethical issues can arise in both the design and the evaluation of interventions, and they can challenge a practitioner’s professionalism and integrity.
Ethical issues in program intervention can arise when the applied social psychologist is aware of or discovers information that, if divulged, will further the interests of one stakeholder group at the expense of another.
Another approach to preventing or resolving ethical dilemmas is to involve various stakeholders (program recipients, front-line staff, managers, board members, funders, etc.) in making key decisions regarding the intervention design or evaluation project.
The key
variables were measured by interviews and
surveys for each study at pretest and posttest, and
then depending on the study, at follow-ups of one,
three, six, or twelve months. Across the four stud-
ies and all follow-up intervals, compared with
control participants who had also reported eating
disorder concerns, the women in the dissonance
condition showed reductions in the three risk
factors (i.e., thin-ideal internalization, body dis-
satisfaction, and negative affect) and also in the
dieting and eating disorder symptoms. Stice,
Marti, Spoor, Presnell, and Shaw (2008) con-
ducted a 2- and 3-year follow-up of the
intervention reported in Stice and colleagues
(2006) and nota-
bly found that all of the reductions in the key
factors had been sustained for either two or three
years, as well as finding a lower risk of the onset
of an eating disorder at three years. Extrapolating
from the latter result, the researchers estimated
that out of every 100 women who participated in
the intervention, nine fewer would develop an
eating pathology. Needless to say, these results
are remarkably promising, perhaps especially for
a relatively simple 3-hour intervention. Add to
this that Stice and colleagues (2008) cite five other
research teams who also have demonstrated the
efficacy of the dissonance intervention. Clearly
this is an excellent example of an evidence-
based intervention.
It must be remembered that one’s
expertise carries with it the responsibility to
always act in ways that place the interests of
those who are in the greatest need of help above
the self-interests of the professional. This is the
essence of acting ethically.