Chapter 15 Measuring Program Outcomes and Standardized Instruments

Selection Criteria for Evaluation Measuring Instruments

The choice of a measuring instrument to evaluate variables—most commonly program objectives—within an evaluation study depends on several situational factors:

• Research Question: The specific inquiry being addressed by the evaluation.

• Data Requirements: The type and format of data needed to answer the evaluation question (e.g., quantitative versus qualitative).

• Participant Characteristics: The capabilities, literacy levels, cultural background, and demographics of the selected evaluation participants.

• Resource Constraints: The amount of time and funding available to conduct the measurement and data analysis.

Types of Measuring Instruments

Five primary types of measuring instruments are commonly utilized in outcome and process evaluations:

1. Journals and Diaries

2. Logs

3. Inventories

4. Checklists

5. Summative Instruments

Journals and Diaries

Journals and diaries are data-collection tools primarily used in interpretive qualitative studies that gather data in the form of words rather than numbers. They are generally excluded from positivistic studies that rely on numerical data.

• Application Example: Tracking women's personal experiences of home birth from early pregnancy through post-delivery.

• Validity: A journal is valid to the extent that it accurately and completely records relevant experiences while excluding irrelevant information (e.g., avoiding length descriptions of unrelated personal topics such as pets or household plants). Achieving validity requires respondents to possess adequate language skills, maintain focus on the topic, and commit to completing entries regularly.

• Retrospective Data Considerations: Retrospective data relies on human memory, which is susceptible to distortion, embellishment, or subconscious invention. Data accuracy is significantly higher when entries are recorded immediately following an event rather than days or weeks later.

• Reliability: Reliability is achieved when identical experiences produce identical written descriptions. Over time, respondent fatigue can lead to omitted details, deliberate embellishments to avoid repetition, or stylistic changes, all of which compromise both validity and reliability.

• Utility: Dependent on the participant's willingness and ability to maintain a written record without finding the task excessively burdensome. For evaluators, utility depends on having sufficient time and resources to conduct rigorous qualitative data analysis on raw text.

• Sensitivity: Reflects the level of descriptive detail captured. High sensitivity is essential for tracking subtle, incremental changes over the course of an intervention or experience.

• Reactivity: Journals are highly reactive. The reflective process inherent in writing often leads participants to gain deeper insights, prompting spontaneous behavioral or affective changes. While reactivity is advantageous in therapeutic contexts, it is generally undesirable in formal measurement instruments unless the explicit goal of the qualitative evaluation is to explore how self-reflection shapes personal meaning.

Logs

Logs represent a structured form of a journal. Participants record specific events, behaviors, or experiences in concise note form under predefined headings.

• Standard Headings: Typical log entries include: the event itself, the time and location of the event, and the individuals present.

• Validity: Logs offer higher validity than journals because structured headings constrain the participant to record only relevant information, preventing off-topic elaboration.

• Reliability: Standardized headings increase reliability by encouraging consistent recording across similar events over time.

• Utility: Highly practical; logs require less time for participants to complete and less time for evaluators to code and analyze.

• Sensitivity & Reactivity: Logs are generally less sensitive to subtle changes due to their brief, structured format, and they tend to be less reactive than open-ended journals.

Inventories

An inventory is an open-ended list generated and completed directly by the evaluation participant.

• Application Example: A depression inventory asking participants: "List below the things that make you feel depressed: _______________________"

• Psychometric Properties:

  • Validity: Valid to the extent that the participant's self-generated list is complete.

  • Sensitivity: Sensitive if the addition or removal of items across administrations accurately reflects individual change over time.

  • Utility & Reliability: Useful when participants fill them out carefully and honestly; reliable if identical psychological states produce identical entries. Provokes thought, making it moderately reactive.

Checklists

A checklist is a closed-ended list prepared in advance by the evaluator. Participants mark items that apply to their current or recent status.

• Application Example: A depression checklist instructing participants: "Check below all the things that you have felt during the past week:"

  • \_\_\_\_ A wish to be alone

  • \_\_\_\_ Sadness

  • \_\_\_\_ Powerlessness

  • \_\_\_\_ Anxiety

Summative Instruments

Summative instruments provide greater objectivity and precision than journals, logs, inventories, or checklists.

• Rating Scales vs. Summated Scales:

  • Rating Scales: Obtain data from a single question or item regarding a practice or program objective.

  • Summated Scales: Present multiple items related to a single domain. Individual item responses are added together to form a composite score indicating the participant's overall status on the measured variable.

• Format: Typically utilize Likert-style response continuum options (e.g., "strongly agree," "agree," "neutral," "disagree," "strongly disagree").

• Unidimensional vs. Multidimensional Scales:

  • Unidimensional Scales: Measure a single target variable.

  • Multidimensional Scales: Combine multiple related subscales into a single instrument to measure distinct sub-dimensions of a complex construct simultaneously.

Examples of Summative Scales

• Client Satisfaction Inventory (CSI) Short Form: A 9-item unidimensional scale designed by Steven L. McMurtry to assess overall client satisfaction with services.

CSI Short Form Scoring Procedure

The scoring protocol for the 9-item CSI Short Form follows a 5-step mathematical procedure:

1. Step 1: Calculate the sum of all valid response values ($Sum$). Unanswered items or items marked "X" (does not apply) are assigned a value of $0$.

2. Step 2: Count the total number of valid responses ($N$). Items left blank or marked "X" are excluded from $N$. Forms with fewer than 7 answered items ($N < 7$) cannot be scored.

3. Step 3: Subtract $N$ from $Sum$ ($Sum - N$).

4. Step 4: Multiply the result from Step 3 by 100: $(Sum - N) \times 100$

5. Step 5: Divide the product from Step 4 by the product of $N$ and 6: $(N \times 6)$.

$\text{Score} = \frac{(Sum - N) \times 100}{N \times 6}$

• Worked Example: If $Sum = 51$ and $N = 8$:

$\text{Score} = \frac{(51 - 8) \times 100}{8 \times 6} = \frac{4300}{48} = 89.6$

• Client Satisfaction Inventory (Long Form): A 25-item unidimensional version created by Steven L. McMurtry (1994) that measures client satisfaction in greater detail. Note that client satisfaction scale scores are appropriate for process evaluations, but cannot be used for outcome evaluations because client satisfaction is not a clinical program outcome objective.

• Reid-Gundlach Social Service Satisfaction Scale: A 34-item multidimensional summative instrument containing three distinct subscales:

  1. Relevance of received social services: Items 1 through 11.

  2. Extent to which services reduced the problem: Items 12 through 21.

  3. Extent to which services enhanced client self-esteem, power, and integrity: Items 22 through 34.

Standardized Measuring Instruments

Standardized measuring instruments are empirical measurement devices that have undergone rigorous testing and include technical documentation across six core parameters:

1. Purpose: A clear, concise statement specifying the exact variable or construct the instrument is designed to measure.

2. Description: Details structural characteristics, total item count, administration procedures, and the clinical cutting score (if applicable).

   • Clinical Cutting Score: A specific numeric threshold that differentiates individuals with a clinically significant problem from those without. In Walter W. Hudson's Index of Self-Esteem (ISE), the clinical cutting score is $30 \pm 5$. Scores above 30 indicate a clinically significant self-esteem problem; scores below 30 indicate no clinically significant problem.

3. Norms: Details the demographic composition and size of the sample used to standardize the instrument.

   • ISE Norms Example: Tested on $1,745$ respondents, including single and married individuals, clinical and non-clinical populations, college students and non-students, Caucasians, Japanese Americans, Chinese Americans, and smaller ethnic groups. Not recommended for children under age 12.

   • Norm Matching Requirement: Instruments must only be administered to client populations sharing the exact demographic characteristics of the normative standardization sample. Cultural differences influence response styles (e.g., cultural modesty affecting self-ratings of physical beauty, unfamiliarity with colloquialisms like "wallflower," or differing norms regarding self-consciousness around strangers).

4. Scoring: Standardized formulas and directions for scoring items, accounting for positively and negatively worded statements.

5. Reliability: Psychometric documentation of internal consistency and stability over time.

   • ISE Psychometrics: Mean Cronbach's alpha of $0.93$ (excellent internal consistency), Standard Error of Measurement (S.E.M.) of $3.70$, and a 2-hour test-retest stability correlation of $0.92$

6. Validity: Evidence of construct and criterion validity.

   • ISE Validity Evidence: High known-groups validity (significantly differentiates individuals clinically diagnosed with self-esteem problems from those without) and construct validity (correlates predictably with depression, happiness, sense of identity, and the Generalized Contentment Scale).

The Fundamental Selection Rule

The target variable selects the instrument; the instrument never selects the variable. Evaluators must never modify or substitute their evaluation variable simply to fit an available standardized instrument (e.g., substituting family cohesion for family coping).

Evaluating Standardized Measuring Instruments

When evaluating standardized instruments for inclusion in research or evaluation studies, evaluators must systematically assess four major evaluation criteria:

1. Sample Criteria

• Are the standardization samples representative of pertinent target populations?

• Are the sample sizes sufficiently large?

• Are the samples homogeneous?

• Are subsamples pertinent to the target respondents' demographics?

• Are the data obtained from the normative samples up to date?

2. Validity Criteria

• Is the content domain clearly and specifically defined?

• Was a logical, rigorous procedure used to select items?

• Is the criterion measure directly relevant to the instrument?

• Was the criterion measure demonstrated to be reliable and valid?

• Is the underlying theoretical construct clearly and correctly conceptualized?

• Do instrument scores converge with other established measures of the same construct?

• Do instrument scores discriminate effectively from irrelevant constructs?

• Do cross-validation studies confirm these validity parameters?

3. Reliability Criteria

• Is there sufficient empirical evidence of internal consistency?

• Is there demonstrated equivalence between alternate instrument forms?

• Is there stability over relevant time intervals (test-retest reliability)?

4. Practicality of Application

• Is the instrument an appropriate length for the evaluation setting?

• Is the item content socially acceptable to respondents?

• Is the completion process feasible for participants?

• Is the instrument format direct and clear?

• Does the instrument demonstrate high practical utility?

• Is the instrument nonreactive?

• Is the instrument sensitive to measuring meaningful client change?

• Is the scoring procedure simple and feasible?

Advantages and Disadvantages of Standardized Instruments

Judy Krysik outlines the primary advantages and disadvantages associated with standardized measurement instruments:

Advantages

• Readily available and easy to access.

• Extensive developmental and psychometric work is pre-completed.

• Established, documented reliability and validity estimates.

• Normative data are available for population comparisons.

• Most instruments are simple to complete and score.

• Many instruments are available free of charge.

• Instruments are frequently available in multiple languages.

• Target age ranges and required reading levels are explicitly specified.

• Required administration time has been empirically established.

Disadvantages

• Established norms may not apply to the evaluation's target population.

• Reading levels or language complexity may be inappropriate for participants.

• Instrument tone may conflict with program orientation (e.g., deficit-based items versus strengths-based values).

• Target populations may misunderstand translated versions.

• Scoring algorithms can be overly complex.

• Commercial instruments may be cost-prohibitive.

• Special clinical qualifications or training may be required for administration.

• Instruments may be excessively long or time-consuming.

Measuring Underserved Populations

Robin McKinney highlights critical measurement issues and ethical concerns when utilizing standardized instruments with ethnically and culturally diverse populations:

Instrument Bias

Standardized intelligence and psychological tests often underestimate the true abilities of racial and ethnic minority children. The standardization samples historically failed to include sufficient numbers of minority children, rendering normative interpretation invalid. Standardized instruments frequently require fluency in European American cultural norms, while European American children are never required to show proficiency in minority cultures—implicitly establishing European American culture as the default standard.

Measurement Sensitivity

Measuring instruments are routinely deployed with populations for which they were never validated. Because ethnic minorities were excluded during initial instrument development, validities for these groups cannot be assumed. Interpreting data using culturally insensitive instruments leads directly to the misrepresentation and misunderstanding of minority populations.

Importance of Variables Measured

Assuming that all cultural groups attach equal importance to the same variables misuses measurement and asserts cultural dominance. For example, spirituality may hold paramount importance for Native Americans compared to material accumulation. Evaluators must study variables that hold intrinsic value to the group under study rather than imposing variables valued exclusively by the researcher.

Language Barriers

Assessing non-English primary speakers with English instruments introduces severe error. Translations of standard instruments rarely undergo equivalency verification studies, and translated versions may yield results inconsistent with English originals. Furthermore, utilizing interpreters introduces uncontrolled interpersonal dynamics that directly influence participant responses.

Observational Data Misinterpretations

Qualitative observational research is highly vulnerable to cultural misinterpretation. Nonverbal behaviors carry culturally specific meanings: in several Native American cultures, maintaining direct eye contact with an authority figure is considered disrespectful, whereas in European American culture, direct eye contact signifies respect and attentiveness. Unfamiliarity with cultural norms leads directly to incorrect behavioral coding.

Chapter Study Questions

1. Before entering your social work program, how knowledgeable were you about the various types of standardized measuring instruments? Discuss in detail.

2. In your own words, describe measuring instruments. What's so "standardized" about them anyway? Provide social work examples throughout your discussion to illustrate your main points.

3. List and discuss the main advantages of using standardized measuring instruments in social work research and practice. Provide social work examples throughout your discussion to illustrate your main points.

4. What are journals or diaries? How can they be used in social work research and practice? Provide social work examples throughout your discussion to illustrate your main points.

5. What are logs? How can they be used in social work research and practice? Provide social work examples throughout your discussion to illustrate your main points.

6. What are inventories? How can they be used in social work research and practice? Provide social work examples throughout your discussion to illustrate your main points.

7. What are checklists? How can they be used in social work research and practice? Provide social work examples throughout your discussion to illustrate your main points.

8. What are summative instruments? How can they be used in social work research and practice? Provide social work examples throughout your discussion to illustrate your main points.

9. When selecting an instrument to measure a variable of some kind, you need to ask and answer six questions. List each question and discuss why the answer is important for you to know either as a social work researcher or as a social work practitioner. Provide social work examples throughout your discussion to illustrate your main points.

10. When choosing a standardized measuring instrument from the hundreds that exist, you need to ask and answer four questions. List each question and discuss why the answer is important for you to know either as a social work researcher or as a social work practitioner. Provide social work examples throughout your discussion to illustrate your main points.

11. What are nonstandardized measuring instruments? In what research or practice situations would it be appropriate to use them? Provide social work examples throughout your discussion to illustrate your main points.

12. List and then discuss each one of the nine advantages of using standardized measuring instruments either in research situations or practice situations. Provide social work examples throughout your discussion to illustrate your main points.

13. List and then discuss each one of the eight disadvantages of using standardized measuring instruments either in research situations or practice situations. Provide social work examples throughout your discussion to illustrate your main points.

14. Go to the website www.walmyr.com and click on the sample scales tab, then find the short-form scales option.

    • From the numerous variables that come up, pick an instrument that measures a variable you're interested in.

    • Now, with the help of Google, find three other measuring instruments that claim they measure the same variable as the one you picked out from www.walmyr.com.

    • Compare and contrast their features, such as scoring, format, and end results. Discuss what you learned from this exercise.