chap 13 stats

Chapter 13 Program Evaluation: Applying Your Skills in the Real World

Program Evaluation Process: Overview

  • Planning: Defining the objectives and scope of the evaluation.

  • Execution: Implementing the evaluation plan.

  • Identify Stakeholders: Recognizing everyone involved and their interests.

  • Gather Data: Collecting information relevant to the evaluation.

  • Describe Program: Summarizing the program's features and goals.

  • Analyze Data: Interpreting the data collected.

  • Clarify Evaluation's Goal: Establishing what the evaluation seeks to achieve.

  • Create Evaluation Plan: Structuring steps and methodologies for the evaluation.

  • Communicate Results: Sharing findings with stakeholders.

  • Interpret Data Analysis: Understanding the implications of the analysis results.

  • Form Conclusions: Drawing insights based on data interpretation.

  • Make Recommendations: Suggesting improvements or changes based on conclusions.

  • Report the Results: Compiling a comprehensive report of findings and recommendations.

Textbook Research Question

  • Research Question: Is the Sanctuary Under the Palms approach an effective addiction treatment?

  • Sanctuary Approach: Aims to enhance personal growth through holistic life changes, emphasizing self-guided discovery.

  • Highlights: Inspirational quotes around the facility encourage reflective thinking, such as "Life can only be understood backwards but must be lived forwards."

  • Programmatic Research: Systematic studies that build on previous findings to gain comprehensive insights. Focus groups will be utilized to gather broad feedback on program goals and features for a subsequent survey.

Focus Groups (Study 1)

  • Discussion Format: 90-minute session with 6-12 stakeholders; includes administration, staff, and past/present clients.

  • Sampling Method: Random sampling from each stratum when possible.

  • Planned Focus Group Questions:

    • What are the program's goals?

    • What program features help achieve these goals?

    • How effective is the program currently?

    • What measures indicate program effectiveness?

    • What are the objectives of this evaluation?

Qualitative Data: Content Analysis of Program Goals and Features

  • Goals:

    • Addiction recovery

    • Feeling empowered

    • Personal growth

    • Life satisfaction

    • Happiness

  • Features:

    • Self-guided discovery

    • Quotes for inspiration

    • Solitary reflection time

    • Smoothie supplements

    • Organic food

    • Hot yoga

Survey (Study 2)

  • Literature Review: Seeking scales to quantify success in meeting treatment goals.

  • Addiction Recovery Scale:

    • Developed by Butler et al. (2005) for self-reporting on addiction severity and effectiveness of recovery.

    • 27 items assessing dependence severity and functioning, using a 4-point scale (1 = not at all true; 4 = very true).

  • Reliability: High alpha (α = .93) confirming scale consistency.

Review of Related Literature

  • Locus of Control Scale: Measures beliefs about control over life outcomes (Rotter, 1966). Items assess internal vs. external control across 29 items, with alpha ranging from .65 to .79.

  • Personal Growth Initiative Scale: 9 items measuring engagement in personal growth activities (Robitschek, 1998) with high reliability (α = .93).

  • Life Satisfaction Scale: 5 item scale measuring overall life satisfaction (Diener et al., 1985); reliability (α = .87) indicates good consistency.

  • Happiness Measure: 4 items assessing current happiness levels (Lyubomirsky & Lepper, 1999) with reliable scores (α = .79 to .94).

Survey Implementation

  • Target Group: Current clients (at least 10 days in the program) and past clients.

  • Survey Components:

    1. Outcome scales for various goals.

    2. Overall effectiveness rating and specific feedback on program features.

    3. Questions on addiction type and demographics.

Study 2 Results

  • Descriptive Statistics:

    • Addiction Recovery: M = 3.61, SD = 0.23, t = 72.21, p < .001

    • Personal Growth: M = 4.56, SD = 0.31, t = 67.75, p < .001

    • Locus of Control: M = 0.48, SD = 0.17, t = -1.01, p = .313

    • Satisfaction with Life: M = 6.20, SD = 0.48, t = 57.49, p < .001

    • Happiness: M = 6.62, SD = 0.53, t = 61.15, p < .001

Study 2 Results: Program Features Evaluation

  • Effectiveness Ratings across Program Features:

    • Self-Guided Discovery: M = 6.50, SD = 0.52

    • Quotes: M = 6.63, SD = 0.50

    • Daily Reflection: M = 6.52, SD = 0.52

    • Smoothies: M = 6.75, SD = 0.51

    • Organic Food: M = 6.49, SD = 0.50

    • Hot Yoga: M = 6.74, SD = 0.54

Follow-up Analyses: Current vs. Former Clients

  • Assessment of Completion Impact: Minimal benefits associated with having completed the program, with former clients having been out of the program for a mean of 13.34 days.

Follow-up Analyses: Addiction Type

  • Effectiveness by Addiction Type: The program shows effectiveness for both alcohol and prescription drug users, with no significant differences noted.

Reporting Results

  • Technical Reporting Style: Outcomes presented by percentage grades, indicating scale mean comparisons:

    • Self-Guided Discovery: 92.86% (A-)

    • Quotes: 94.71% (A)

    • Daily Reflection: 93.00% (A)

    • Overall Effectiveness (Single Item): 95.00% (A+)

Other Possible Analysis

  • Quasi-experimental Design: 2x2 factorial design assessing impacts of client type and addiction type on several dependent variables (e.g., recovery, personal growth).

Example Result: Addiction Recovery

  • Statistical Significance: Interaction effect significant, F(1, 102) = 9.42, p = .003, indicating the program is most effective for past clients with drug addiction.