POLI 351 Final

Lecture 10/11, Pal chps, 5, 7, 8, 2

Canadian Evaluation Society

at its peak in the late 1980s, professionalized evaluation in public + non-profit sector

Why do we evaluate?

improved policy, accountability, program enhancement, fiscal responsibility

evaluation (Patten 2008)

systematic collection of data, used to make judgements about the efficiency and effect of activities, outcomes

What types of data are used in evaluation?

qualitative and quantitative descriptive + demographic data are routinely normatively gathered, mixed-methods approach helps move beyond widget-counting into measuring social impacts

Stages of Policy Evalution

Patton, Sawicki, and Clark (2016); four stages


1. Ex-Ante (formative): do at beginning how to set-up, look at process
2. Policy Maintenance: is this appropriate data, right after implementation
3. Policy Monitoring: operation, process of recording data, performance measures
4. Ex-Post Policy Evaluation: useful in pilots, once program is up & running

What are the central components in policy evaluation?

practice and application of techniques, ultimate goal of improvement, systematic character of evaluation research,

\
principles of neutrality, objectivity, expertise, professional standards

4 key principles of evaluation

1. systematic data-based inquiry
2. competence
3. integrity + honesty
4. respect for people

impact evaluation (Pal)

an analysis of the actual effect or impact of a program on its intended target, along with unintended consequences

impact evaluation (Hall)

core of what we talk about re: evolution, program theory tho draw conclusions; focuses on impacts and outcomes, critical in determining success, challenges with causality, greater reliance on statistical techniques + quasi-experimental design

process evaluation (Pal)

evaluations that monitors an existing program to assess the effort and organizational resources put into it

process evaluation (Hall)

focuses on process by which goals could be met, identifies activities required to implement and assess how each activity is carried out,

\
ex. audit of policies and procedures

efficiency evaluation

less common in govt not looking at social impact; looks at ratio of cost elements to benefits; did govt get most value for their money

\
two techniques: cost benefit + cost effectiveness

cost-benefit analysis (Pal)

the evaluation of a program in terms of it total costs compared to its total benefits

cost-effectiveness analysis (Pal)

comparison of different program alternatives for achieving a given set of goals; it is also applied by considering a fixed budget and choosing alternatives that provide the highest rate of goal achievement

cost-benefit analysis (Hall)

net present value, technical, financial analysis, time & labour, requires expertise, purely qualitative

formative evaluation (Pal)

an evaluation designed to support development and improvement of a program as it is being implemented

formative evaluation (Hall)

checking if policy alternatives are appropriate, feasible, helps designers, allows you to tweak options,

summative evaluation (Pal)

an evaluation undertaken at the end of a program to gauge its success

summative evaluation (Hall)

end of cycle, focus on outcomes + events, should the program be cut

problems in policy evaluation

1. uncertainty over policy goals
2. difficulty in determining causality or level of influence
3. difficulties in data acquisition
4. official resistance, lack of buy-in
5. limited time
6. limited resources
7. lack of influence

steps in the evaluation process

1. define → goals, problem stmt, what type of evalution
2. collect → research design, what data
3. analyse → related to definition, draw conclusions
4. report → write it up!
5. share & inform

What to look for when planning policy research?

what factors are most likely related to your dependent/indepedent variables


1. demographic characteristics (gender, education, age)
2. historical trends
3. political/economic factors (party affiliations, economic sector drivers)

How does budgeting relate to evaluation?

build evaluation components into planning to get budget, then undertake a spending review and revisit the budget preparation process

results orientation (Pal)

a management focus on new forms of review, audit, and evaluation based on results

performance story (Pal)

the reporting of program results in such a way as to highlight both successes and shortcomings, the challenges faced by the organization and what it might do in the future to improve results

Why is communication important?

key to understanding stakeholder, the issue itself and the nature of the problem

policy is made and sold with perspectives and involves large and diverse audiences

How is communication changing?

digital and social media, information flows more quickly, abundance of information, sifting through

issue framing (Pal)

a way of depositing a policy issue or problem in broad understandable if somewhat simplified terms

What is communication?

the act of giving, receiving and sharing information:

talking, writing, speaking, reading, listening

What is the IAP2 spectrum?

a continuum of strategies which reflect the public’s increasing impact on aspects of the decision-making process

levels:


1. informing
2. consultation
3. involvement
4. collaboration
5. empowerment

Informing (IAP2)

govt. provides information only to individuals/groups to make their own. hopefully more informed, decisions; information is balanced and objective

Consultation (IAP2)

govt. will provide information and seek feedback on the information it provides,

listening to and incorporating ideas/info from the public + provide information on how that input has been received and implemented

Involvement (IAP2)

a commitment to work with the public throughout the policy development process + ensure that the publics goals and ideas are consistently reflected in the materials that are produced

Collaboration (IAP2)

represents partnership models, shift in responsibility + ownership, public and other partners recommendations are incorporated as far as it is reasonably possible

Empowerment (IAP2)

suggestion that government fully delegates the responsibility to external public organization or decision-making group and implements fully what is discussed

Dunn’s 4-stage process of policy communication

1. policy analysis (the ‘what’)
2. materials development (the ‘how’)
3. interactive communication (the ‘where’)
4. knowledge utilization (the ‘why’)

5 steps for effective communication planning

1. understand your objectives
2. understand your audiences
3. choose the right channels
4. plan your messaging
5. monitor effectiveness

policy argument

An organized set of claims about a policy problem that includes causality, severity, novelty, crisis, instruments, and solutions

A communications tool used to influence governments and decision makers by getting a problem and/or a desired solution on the agenda

Media Logic

The process through which media transmit and communicates information
4 key ideas:


1. Programmability
2. Popularity
3. Connectivity
4. Datafication

What are the contents of a communication plan template?

\-Title of Action

\-Objectives (short para. - what will it accomplish?)

\-Key Messages/Statement (comms. goals; 2-3 main facts the public needs to know; IAP2)

\-Key Audiences (key stakeholders; others that need to be part of the coms. plan)

\-Action Plan (how the info will be disseminated; pre-plan answers to anticipated questions; have a list of contacts/talking points)

\-Post-Action Outreach/Evaluation (monitoring things afterwards)

What are some key things to look for when understanding your audience?

What __motivates__ them?

What do they __care__ about?

What's __important__ to them?

(Specific info; comments; positions or agendas; mandates; meeting minutes)

Goal of Policy Communication

Informing citizens of the existence of a policy and persuading them to participate in and/or comply with the policy requirements