Chapter 15 Measuring Program Outcomes and Standardized Instruments

Overview of Program Outcome Measurement

• Measuring instruments are selected to evaluate variables, which are typically program objectives within evaluation studies.

• The selection of a specific measuring instrument depends on several situational factors:

  • The specific research or evaluation question being asked.

  • The type of data required (e.g., qualitative words versus quantitative numbers).

  • The characteristics of the selected evaluation participants.

  • The total time available to conduct the study and analyze the data.

  • The financial budget and resources allocated for the evaluation.

Five Practical Types of Data Collection Instruments

Journals and Diaries

• Definition & Context: Journals and diaries are data-collection tools primary used in interpretive studies that gather qualitative data in the form of words. They are generally not used in positivistic studies that focus on numerical data.

• Application Example: An evaluation asking, "What are women's experiences of home birth?" where participants record their personal experiences continuously from early pregnancy through post-delivery.

• Psychometric and Practical Properties:

  • Validity: A journal is valid to the extent that it completely and accurately describes all relevant experiences while omitting irrelevant details. Achieving high validity requires participants to possess adequate language skills, remain focused on the target topic (avoiding tangential entries such as lengthy descriptions of pets or plants), and complete entries on a regular, timely basis.

  • Retrospective Data Caution: Retrospective data relies on human memory, which is subject to embellishment, invention, or distortion. Descriptions recorded immediately after an event are substantially more accurate than those written days or weeks later.

  • Reliability: Reliability depends on whether identical experiences evoke consistent written descriptions over time. Participant fatigue can compromise reliability if subjects change their writing style or alter details to make entries less repetitive.

  • Utility: Highly dependent on whether participants enjoy writing and can sustain a demanding recording schedule. It also depends on whether the evaluator has sufficient time to conduct rigorous qualitative data analysis on extensive text entries.

  • Sensitivity: Refers to the level of detail captured. High sensitivity allows subtle changes in participant experiences to be tracked over time.

  • Reactivity: Highly reactive. The act of self-recording forces reflection, which often generates deeper insights and induces behavioral or emotional changes. While reactivity is typically undesirable in measurement, it can be advantageous in interpretive evaluation seeking to uncover the personal meaning participants attach to their experiences.

Logs

• Definition: Structured variations of journals where participants record specific events, experiences, or behaviors in abbreviated note form using standardized headings.

• Structure: Typically structured around predefined categories such as:

  • The specific event that occurred.

  • When and where the event took place.

  • Who was present during the event.

• Comparison to Journals:

  • Validity: Higher than journals because structured headings prompt participants to focus exclusively on relevant information and eliminate tangential topics.

  • Reliability: Higher than journals because structured formats encourage consistent recording of similar events.

  • Utility: Greater utility for both participants (faster to write) and evaluators (faster to code and analyze).

  • Sensitivity: Lower than journals because brief note-taking captures less contextual detail.

  • Reactivity: Less reactive than journals due to reduced opportunity for deep narrative reflection.

Inventories

• Definition: Participant-completed open lists designed to capture items relevant to a specific variable or condition.

• Format Example: A depression inventory prompting: "List below the things that make you feel depressed: _______________________."

• Psychometric Properties:

  • Validity: Dependent on how completely and truthfully the participant populates the list.

  • Sensitivity: High; additions or omissions of list items over time serve as direct indicators of change.

  • Reliability: High if identical internal experiences consistently prompt the participant to record the same entries.

  • Reactivity: Fairly reactive, as generating list items provokes cognitive reflection on targeted feelings or behaviors.

  • Utility: High, provided participants complete the list thoroughly.

Checklists

• Definition: Instruments containing predefined lists of items prepared in advance by the evaluator, requiring participants to mark applicable items.

• Format Example: A depression checklist prompting: "Check below all the things that you have felt during the past week: [ ] A wish to be alone, [ ] Sadness, [ ] Powerlessness, [ ] Anxiety."

Summative Instruments

• Definition & Distinction:

  • Rating Scales: Single-item questions designed to capture data about a practice or program objective.

  • Summative Scales: Multi-item questionnaires that combine participant responses across all questions into a single composite score indicating the individual's overall level or position regarding the variable being measured.

• Response Formats: Utilize structured Likert-type response options, such as "strongly agree," "agree," "neutral," "disagree," and "strongly disagree."

• Methodological Advantage: Provide substantially higher objectivity and precision in variable measurement compared to journals, logs, inventories, and checklists.

• Scope of Use: Broadly applied to assess individual or family problems, execute community needs assessments, and perform process or outcome evaluations at both individual case and program levels.

Standardized Measuring Instruments

Overview and Core Information Components

• Standardized instruments are pre-tested, high-precision tools widely used in research and practice. All standardized instruments provide documented empirical information across six core dimensions:

  1. Purpose: A clear statement defining exactly what variable or construct the instrument measures.

  2. Description: Structural details including total item count, scale format, target age, reading level, and clinical cutting scores.

  • Clinical Cutting Score: A designated numeric score threshold that separates individuals with clinically significant problems from those without clinically significant problems.

  1. Norms: Detailed demographics of the sample population used to establish and validate the instrument.

  2. Scoring Instructions: Step-by-step procedures for calculating final scores, including directions for handling missing items or reverse-scored items.

  3. Reliability Data: Quantitative evidence of consistency, such as internal consistency estimates (Cronbach's alpha) and test-retest stability coefficients.

  4. Validity Data: Quantitative evidence confirming the instrument accurately measures the intended theoretical construct (e.g., known-groups validity, construct validity).

Standardized Instrument Examples in Service Evaluation

Client Satisfaction Inventory (CSI) - Short Form

• Variable Measured: Client satisfaction with received services (a unidimensional scale containing $9$ items).

• Response Scale: $7$-point frequency scale ranging from $1$ ("None of the time") to $7$ ("All of the time").

• Items included:

  1. People here really seem to care about me.

  2. I would come back here if I need help again.

  3. I would recommend this place to people I care about.

  4. People here really know what they are doing.

  5. I get the kind of help here that I really need.

  6. People here accept me for who I am.

  7. People here seem to understand how I feel.

  8. I feel I can really talk to people here.

  9. The help I get here is better than I expected.

• Mathematical Scoring Protocol:

  • Step 1: Calculate the sum of all valid responses, designated as $\text{SUM}$. Assign a value of $0$ to items left blank or marked with an "x".

  • Step 2: Count the total number of valid responses, designated as $N$. Blank or "x" items are excluded from $N$. If $N < 7$, the form cannot be scored.

  • Step 3: Subtract $N$ from $\text{SUM}$: $(\text{SUM} - N)$.

  • Step 4: Multiply the result by $100$: (\text{SUM} - N) \times 100$.\n - **Step 5**: Divide the Step 4 product by (N \times 6).\n - **Formula**:\n    \text{SCORE} = \frac{(\text{SUM} - N) \times 100}{N \times 6}\n - **Worked Example**: If \text{SUM} = 51$and$N = 8:\n    \text{SCORE} = \frac{(51 - 8) \times 100}{8 \times 6} = \frac{4300}{48} = 89.6\n\n### Client Satisfaction Inventory (CSI) - Long Form\n\n\n\n- **Developer**: Steven L. McMurtry (Copyright 1994).\n- **Variable Measured**: Client satisfaction with services (25 items).\n- **Scale Anchors**: 1$= None of the time to$7$= All of the time;$X = Does not apply.\n- **Reverse Scored Items**: Items 4$,$7$,$16$,$18$, and$22.\n- **Evaluation Application**: Used exclusively in process evaluations, not outcome evaluations, because client satisfaction is an operational output rather than a client outcome objective.\n\n### Reid-Gundlach Social Service Satisfaction Scale\n\n\n\n- **Structure**: A multidimensional summative instrument containing 34$items rated on a$5$-point agreement scale ($1$= Strongly agree to$5 = Strongly disagree).\n- **Subscales**:\n 1. *Relevance of received social services*: Items 1$through$11\n 2. *Extent to which services reduced the problem*: Items 12$through$21\n 3. *Extent to which services enhanced client self-esteem and contributed to a sense of power and integrity*: Items 22$through$34\n\n- **Dimensionality Classification**:\n - **Unidimensional Instruments**: Measure a single unified variable (e.g., CSI Short Form and Long Form).\n - **Multidimensional Instruments**: Measure several distinct but highly correlated sub-variables simultaneously by combining separate subscales into one instrument.\n\n## Hudson's Index of Self-Esteem (ISE)\n\n\n\n\n\n- **Author & Reference**: Walter W. Hudson (Copyright 1993; Primary Reference: Hudson, W.W., 1982, *The clinical measurement package: A field manual*).\n- **Purpose**: Designed specifically to measure the degree, severity, or magnitude of a client's self-esteem problems (evaluative component of self-concept).\n- **Description**: 25$-item self-report instrument using a$7$-point response scale ($1$= None of the time to$7 = All of the time). Part of the nine-scale Clinical Measurement Package.\n- **Clinical Cutting Score**: 30 \ (\pm 5)$. Scores above$30$indicate a clinically significant self-esteem problem; scores below$30 indicate no clinically significant problem.\n- **Normative Sample**: Derived from 1,745$respondents spanning single and married individuals, clinical and nonclinical populations, college students and nonstudents, Caucasians, Japanese Americans, Chinese Americans, and smaller ethnic subgroups. Not recommended for children under age$12\n- **Reverse Scored Items**: Items 3$,$4$,$5$,$6$,$7$,$14$,$15$,$18$,$21$,$22$,$23$, and$25\n- **Psychometric Properties**:\n - **Reliability**: Mean Cronbach's alpha of 0.93$(excellent internal consistency); Standard Error of Measurement (S.E.M.) of$3.70$;$2$-hour test-retest stability correlation of$0.92\n - **Validity**: High known-groups validity (significantly discriminates between clients judged clinically to have self-esteem problems versus non-problematic clients) and high construct validity (correlates predictably with depression, happiness, sense of identity, and Generalized Contentment Scale scores).\n\n## Principles of Matching Norms and Construct Selection\n\n- **Demographic Alignment**: An instrument must only be administered to populations whose demographic characteristics match those of the normative standardization sample. Cultural variations alter how items are interpreted:\n - *Example*: Modesty norms in specific cultures may prevent participants from endorsing item 3$("I feel that I am a beautiful person all of the time") or item$16 ("I feel very self-conscious when I am with strangers").\n - *Example*: Cultural idioms such as "wallflower" (item 19$$) may not be understood by non-native speakers.

• Age Appropriateness: Instruments applied to children must be constructed and normed specifically with child populations.

• Rule of Construct Selection: The target outcome variable must dictate the choice of instrument. Evaluators must never modify or redefine their target variable simply to fit an available instrument (e.g., substituting family cohesion for family coping).

Evaluation Criteria for Standardized Instruments

When evaluating standardized instruments for research or practice, four critical dimensions must be systematically examined:

1. Sample Characteristics:

   • Are the standardization samples representative of pertinent target populations?

   • Are sample sizes sufficiently large?

   • Are samples homogeneous?

   • Are sub-samples relevant to client demographics?

   • Is the normative data up to date?

2. Validity Standards:

   • Is the content domain clearly and specifically defined?

   • Was a logical, systematic procedure used to select items?

   • Is the criterion measure reliable, valid, and direct?

   • Is the theoretical construct accurately specified?

   • Do scores converge with related constructs and discriminate from unrelated variables?

   • Have cross-validation studies replicated these findings?

3. Reliability Standards:

   • Is there sufficient evidence of internal consistency?

   • Is there equivalency across alternate forms?

   • Is there score stability over relevant time intervals?

4. Practicality of Application:

   • Is the length reasonable for the population?

   • Is the item content socially acceptable to respondents?

   • Is completion feasible and direct?

   • Is the instrument useful, nonreactive, and sensitive to clinical change?

   • Is scoring feasible and straightforward?

Advantages and Disadvantages of Standardized Instruments

Advantages

• Readily available and accessible.

• Development work, item selection, and pre-testing are already complete.

• Established reliability and validity coefficients are documented.

• Comparative normative data is available.

• Easy for most clients to complete and simple to score.

• Frequently accessible without licensing fees.

• Often available in multiple translated versions.

• Clear guidelines provided regarding age ranges and required reading levels.

• Time required for administration is pre-determined.

Disadvantages

• Normative samples may not match the targeted client population.

• Language or reading level may be overly difficult.

• The item tone may conflict with program values (e.g., deficit-focused rather than strength-based).

• Target populations may misunderstand translated items.

• Scoring algorithms can be overly complex.

• Proprietary instruments can be expensive.

• Special clinical qualifications or formal certifications may be required for administration.

• Administering the instrument may be too long or time-consuming.

Considerations for Measuring Underserved Populations

Instrument Bias

• Standardized intelligence and psychological tests frequently underestimate the actual capabilities of ethnic minority children.

• Standardization samples historically lacked sufficient ethnic minority representation, invalidating score interpretations for minority groups.

• Intelligence tests often require fluency in dominant European American cultural norms, a requirement not imposed on European American children regarding minority cultures. This default structure inherently treats European American culture as superior.

Measurement Sensitivity

• Culturally insensitive instruments continue to be administered to populations for which they were never validated.

• When ethnic minorities are excluded from instrument development, true measurement validity cannot be established.

• Interpreting minority client scores using insensitive tools generates erroneous conclusions and widespread misrepresentation.

Importance of Variables Measured

• Evaluators must not assume all cultures place equal value on the same variables.

• Example: Spirituality may hold central importance for Native American populations, whereas material acquisition may be secondary. Research must focus on variables valued by the participant community rather than solely by the evaluator.

Language and Translation Barriers

• Assessing non-English speaking clients with English instruments invalidates results.

• Translated instruments frequently lack empirical studies confirming measurement equivalency between translated and original versions.

• The presence of translators or interpreters can systematically alter participant responses, introducing subtle measurement biases.

Observational Misinterpretations

• Qualitative nonverbal observations are highly vulnerable to cultural misinterpretation.

• Example: Direct eye contact from a subordinate to an authority figure signifies respect in European American culture, whereas the same behavior signifies disrespect in specific Native American cultures.

Chapter Summary and Review Questions

Study Questions

1. Before you entered your social work program, and before you read this chapter, how knowledgeable were you about the various types of standardized measuring instruments? Discuss in detail.

2. In your own words, describe measuring instruments. What's so "standardized" about them anyway? Provide social work examples throughout your discussion to illustrate your main points.

3. List and discuss the main advantages of using standardized measuring instruments in social work research and practice. Provide social work examples throughout your discussion to illustrate your main points.

4. What are journal or diaries? How can they be used in social work research and practice? Provide social work examples throughout your discussion to illustrate your main points.

5. What are logs? How can they be used in social work research and practice? Provide social work examples throughout your discussion to illustrate your main points.

6. What are inventories? How can they be used in social work research and practice? Provide social work examples throughout your discussion to illustrate your main points.

7. What are checklists? How can they be used in social work research and practice? Provide social work examples throughout your discussion to illustrate your main points.

8. What are summative instruments? How can they be used in social work research and practice? Provide social work examples throughout your discussion to illustrate your main points.

9. When selecting an instrument to measure a variable of some kind, you need to ask and answer six questions. List each question and discuss why the answer is important for you to know either as a social work researcher or as a social work practitioner. Provide social work examples throughout your discussion to illustrate your main points.

10. When choosing a standardized measuring instrument from the hundreds that exist, you need to ask and answer four questions. List each question and discuss why the answer is important for you to know either as a social work researcher or as a social work practitioner. Provide social work examples throughout your discussion to illustrate your main points.

11. What are nonstandardized measuring instruments? In what research or practice situations would it be appropriate to use them? Provide social work examples throughout your discussion to illustrate your main points.

12. List and then discuss each one of the nine advantages of using standardized measuring instruments either in research situations or practice situations. Provide social work examples throughout your discussion to illustrate your main points.

13. List and then discuss each one of the eight disadvantages of using standardized measuring instruments either in research situations or practice situations. Provide social work examples throughout your discussion to illustrate your main points.

14. Go to the website www.walmyr.com and click on the sample scales tab, then find the short-form scales option:

    • From the numerous variables that come up, pick an instrument that measures a variable you're interested in.

    • Now, with the help of Google, find three other measuring instruments that claim they measure the same variable as the one you picked out from www.walmyr.com.

    • Compare and contrast their features, such as scoring, format, and end results. Discuss what you learned from this exercise.