CHES CH 4

Delimitations

Decisions made by an evaluator or researcher that ought to be mentioned because they are used to help the evaluator identify the parameters and boundaries set for a study. Examples include why some literature is not reviewed, populations are not studied, and certain methods are not used

Evaluation

Series of steps that evaluators use to assess a process or program to provide evidence and feedback about the program

Limitations

Phenomena the evaluator or researcher cannot control that place restrictions on methods and, ultimately, conclusions. Examples include time, nature of data collection, instruments, sample, and analysis

Logic models

Depict aspects of a program such as inputs, outputs, and outcomes. Offer a scaled down, somewhat linear, visual depiction of programs

Research

An organized process in which a research uses the scientific method to generate new knowledge

Reliability

Refers to the consistency, dependability, and stability of the measurement process

Validity

The degree to which a test or assessment measures what it is intended to measure (accuracy)

Variables

Operational forms of a construct. Researchers use these to designate how the construct will be measured in designated scenarios

Unit of Analysis

What or who is being studied or evaluated (the individual, group, organization, or program, etc)

Formative evaluation

Process used to check an ongoing process of the evaluation from planning through implementation

Process evaluation

Any combination of measures that occurs as a program is implemented to assure/improve the quality of performance/delivery

Summative evaluation

• Often associated with measures or judgements that enable the investigator to conclude impact and outcome evaluation

• Includes activities taken to create a judgement on the performance of the program, including whether specific goals and objectives were met

Impact evaluation

The immediate and observable effects of a program leading to desired outcomes

Outcome evaluation

Focused on the ultimate goal, product, or policy and often measured in terms of health status, morbidity, and mortality

What does IRB stand for?

Institutional Review Board

Logic Model: Inputs

Resources needed

Logic model: Outputs

Activities, products, and services that will reach the participants

Logic model: Short-term Outcomes

Changes in knowledge, skills, and access to resources

Logic model: Intermediate outcomes

Changes in behavior or policy

Logic model: Long-term Outcomes

Fundamental changes, morbidity, mortality

Statement of Purpose

Used to identify in detail what health education specialists want to learn throughout evaluation. Usually a sentence or two written with specificity and detail

Goal

Usually long-term and represent a more global vision

Objectives

Define measurable strategies used to attain progress toward a goal

Evaluation model

• Attainment

• Decision-making

• Goal-free

• Naturalistic

• Systems Analysis

• Utilization-focused

Evaluation Model: Attainment

Focused on program objectives and the program goals; serves as standards for evaluation

Evaluation model: Decision-making

Based on four components designed to provide the user with the context, input, processes, and products with which to make decisions

Evaluation model: Goal-free

Not based on goals: evaluator searched for all outcomes including unintended positive and negative side effects

Evaluation model: Naturalistic

Focused on qualitative data and responsive information from participants in a program is used; most concerned with narrative explaining “why” behavior did or did not change

Evaluation model: System analysis

Based on efficiency that cost benefits or cost effectiveness analysis is used to quantify effects of a program

Evaluation model: Utilization-focused

Accomplished for and with a specific population

CDC Framework for Program Evaluation

1. Engage stakeholders: Those involved, affected, primary stakeholders

2. Describe the program: Need, expected effects, activities, resources, stage, context, logic model

3. Focus the evaluation design: Purpose, users, uses, question, methods, agreements

4. Gather credible evidence: Indicators, sources, quality, quantify, logistics

5. Justify conclusions: Standards, analysis/synthesis, interpretation, judgment, recommendations

6. Ensure use and share lessons learned: Design, preparation, feedback, follow-up dissemination

Standards for Effective Evaluation

• Utility

• Feasibility

• Propriety

• Accuracy

Standards for Effective Evaluation: Utility

Serve the information needs of intended users

Standards for Effective Evaluation: Feasibility

Be realistic, prudent, diplomatic, and frugal

Standards for Effective Evaluation: Propriety

Behave legally, ethically, and with due regard for the welfare of those involved and those affected

Standards for Effective Evaluation: Accuracy

Reveal and convey technically accurate information

Probability sampling techniques

Methods in which each member of the priority population has known chance, or probability, of being selected

Probability sampling technique: Simple random sampling

An inclusive list of the priority population is used to randomly (such as with a list of random numbers) select a certain number of potential participants from the list

Probability sampling technique: Systematic random sampling

An inclusive list of the priority population is used, and starting with a random number, every nth potential participant is selected (such as every 14th participant)

Probability sampling technique: Stratified random sampling

The sample is split into groups based on a variable of interest, and an equal number of potential participants from each group are selected randomly

Probability sampling technique: Cluster sampling

When naturally occurring groups (such as schools) are selected instead of individuals

Probability sampling technique: Multistage cluster sampling

In several steps, group are selected using cluster sampling (ex. in a state counties are selected at random, then schools within the county are selected at random)

Probability sampling technique: Stratified multistage cluster sampling

In several steps, a variable of interest is used to split the sample, and then groups are randomly selected from this sample (ex. in a state, counties are selected at random, then an equal number of elementary schools and secondary schools are randomly selected in each country)

Non-probability sampling techniques

Not all units from the priority population have an equal chance of being selected, and thus their representativeness to the population is unknown

Non-probability sampling technique: Convenience

Selection of individuals or groups who are available

Non-probability sampling technique: Purposive

Researcher makes judgements about who to include in the sample based on study needs

Non-probability sampling technique: Quota

Selecting individuals who have a certain characteristics up to a certain number (ex. selecting 50 females from a worksite)

Non-probability sampling technique: Network sampling/Snowball

When respondents identify other potential participants who might have desired characteristics for the study

The goal of data analysis

To use statistical tests to answer the research questions

Health Surveillance Programs: Behavioral Risk Factor Surveillance Survey (BRFSS)

The nation’s premier system of health-related telephone surveys that is used to collect state data about U.S. residents regarding their health-related risk behaviors, chronic health conditions, and use of preventive services

Internal validity

• Refers to the accuracy or truth of the study results which depends on how well the design, implementation, and statistical analysis have minimized or eliminated bias.

• Used to estimate the degree of cause and effect

• If a design has high _____, the researcher has more confidence that changes in dependent variable occurred because of the intervention

External validity

The degree to which the findings are able to be generalized to other groups or populations.