Carr, J. E., Wilder, D. A., Majdalany, L., Mathisen, D., & Strain, L. A. (2013)

Background and Purpose

• Functional assessment is the standard approach for identifying maintaining variables of problem behaviour in clinical & educational settings (Hanley, Iwata & McCord, $2003$).
• Organisational analogues (performance analysis / performance diagnostics) have existed since the $1970s$ (Gilbert, Daniels, Mager & Pipe).
• Performance Diagnostic Checklist (PDC; Austin, $2000$) is a $20$-item, interview-based tool used to locate environmental determinants of poor employee performance.
• Current article introduces the Performance Diagnostic Checklist – Human Services (PDC-HS) and reports a preliminary predictive-validity study in a university-based Early Intensive Behavioural Intervention (EIBI) centre for children with autism.
• Core aim: evaluate whether a checklist tailored to human-service contexts can (a) isolate performance determinants (training, feedback, resources, etc.) and (b) direct effective, parsimonious interventions.

PDC vs PDC-HS

• PDC was designed mainly for business & industry.
  • Includes questions about equipment reliability, ergonomics, etc. – often irrelevant in care environments.
• PDC-HS revisions:
  • Section titles reordered & re-worded for human‐service relevance.
  • Terminology shifted (e.g., "Equipment & Processes" → "Resources, Materials, & Processes").
  • Items explicitly reference caregiving tasks (treatment implementation, data collection, graph construction, attendance, reporting, etc.).
  • Mixed information sources: $13$ items via supervisor interview, $7$ via direct observation.
  • Output: Any "No" answer tags an intervention opportunity; clusters of "No"s determine priority.

Key Concepts in Performance Analysis

• Behavioural deficits vs behavioural excesses: PDC-HS most useful for deficits (absence of behaviour) frequent in staff performance.
• Four environmental domains assessed:
  1. Training
  2. Task Clarification & Prompting
  3. Resources, Materials, & Processes
  4. Performance Consequences, Effort, & Competition
• Analogy to functional behaviour assessment: align intervention with function (e.g., don’t retrain a skill if the real problem is missing consequences).
• Common human-service problems explicitly considered:
  • Poor treatment integrity
  • Inaccurate data collection
  • Inadequate programme material prep
  • Tardiness / absenteeism
  • Failure to report issues
  • Poor graphing practices

Development of PDC-HS

• Original PDC items were applied to six frequent human-service performance problems to spot mismatches.
• Iterative editing by the authors plus $11$ external behaviour analysts (mean experience $12$ years; $9$ PhDs; $10$ BCBAs).
• Result: $20$-item instrument (see Appendix B) with sample interventions & citations attached.

Structure and Administration of PDC-HS

• Four sections; each contains $4 – 6$ binary (Yes/No) items.
• Mixed-method data collection:
  • Interview supervisor → informant answers.
  • Direct in-situ observation for starred items.
• Interviewer: typically a BCBA supervising the performers.
• Scoring logic:
  • Any "No" → candidate deficit.
  • Multiple "No"s within a domain → high-priority target.
  • Practitioner selects concurrent vs sequential interventions depending on staffing resources.
• Embedded intervention menu (e.g., BST, prompts, feedback, access to materials) with literature pointers.

Study Methodology

• Setting: University-based autism treatment centre (SE USA) delivering EIBI to children aged $3 – 7$ years.
• Physical context: $8$ treatment rooms (≈ $3\,m \times 3\,m$ each).
• Participants: $15$ graduate-student therapists (all female, ages $23 – 27$) – each worked in only one target room.
• Baseline problem: Treatment rooms left unclean after $1.5$-h sessions despite hire-orientation training.
• Materials:
  • Treatment-room cleanliness checklist (Appendix A).
  • Graph posted inside each room for feedback phase.
  • PDC-HS interview form.
  • Cleaning supplies: disinfectant wipes, gloves, tissues, first-aid kit.
  • Video camera (data collection).
• Dependent variable: % of checklist tasks completed correctly.
• Data collection logistics:
  • Observers entered $10 – 15$ min post-session; performers absent.
  • IOA on >$40\%$ of sessions per room; mean agreements $91 – 97\%$ (ranges $75\% – 100\%$).
• Experimental design: Concurrent multiple-baseline across rooms; added non-function-matched comparison.

Baseline and Interventions

1. Baseline
   • No contact/feedback with performers.
2. Function-Matched Intervention (indicated by PDC-HS)
   • Components: Brief re-training + posted, graphed feedback.
     • Retraining: verbal description of each checklist item; demonstration of material locations; request for confidentiality (protect design).
     • Feedback: Data graphed within $5$ min of each session and posted beside checklist; performers saw it next entry.
3. Non-Indicated Intervention (control condition)
   • Applied to Rooms G & H first.
   • Components: Task clarification (checklist posted prominently) + centralised materials placement; NO training, NO feedback.
   • Purpose: Test predictive validity by evaluating an intervention NOT suggested by PDC-HS results.

Results and Interpretation

PDC-HS Interview Outcomes

• All $3$ BCBA supervisors pinpointed deficits in Training and Performance Consequences domains.
  • Respondent 1: $75\%$ of Training items & $80\%$ of Consequence items problematic.
  • Respondent 2: same pattern.
  • Respondent 3: $75\%$ Training; $60\%$ Consequences.
• Fewer issues detected in Task Clarification & Resources domains.

Intervention Efficacy (Fig. 2)

• Rooms with Training + Feedback package:
  • Room A: Baseline $47\%$ → $97\%$.
  • Room B: $26\%$ → $98\%$.
  • Room C: $38\%$ → $96\%$.
  • Room D: $41\%$ → $97\%$.
  • Room E: $27\%$ → $92\%$.
  • Room F: $31\%$ → $92\%$.
  • Room G (after failed control): $25\%$ baseline → $36\%$ (control) → $100\%$ (packaged).
  • Room H: $18\%$ baseline → $12\%$ (control) → $80\%$ (packaged).
• Non-Indicated Control (Task Clarification + Materials Only):
  • Room G: minimal improvement (from $25\%$ to $36\%$).
  • Room H: deterioration (from $18\%$ to $12\%$).
• Conclusion: PDC-HS-aligned package robustly improved performance; non-aligned package ineffective, supporting predictive validity.

Practical Implications for Human-Service Settings

• PDC-HS offers a rapid, structured method to diagnose staff performance problems beyond default "retrain & feedback".
• Aligning interventions with diagnosed variables maximises efficiency and avoids unnecessary components.
• Potential cost savings & improved client outcomes when treatment integrity tasks (cleanliness, data accuracy) remain high.
• Tool usable by practising BCBAs overseeing multiple staff; especially helpful when simple solutions fail.

Ethical and Philosophical Considerations

• Client welfare: Poor staff performance (e.g., unhygienic rooms) directly affects health & learning of vulnerable populations (children with autism).
• Staff dignity & fairness: Diagnosing actual environmental determinants prevents blaming individuals for systemic issues.
• Use of feedback: Graphed public posting must respect confidentiality; only task performance, not personal data.
• Indirect assessment caveat: Reliance on supervisor report may introduce bias; ethical obligation to corroborate with observation.

Limitations & Future Research

• Package intervention confounds: Cannot parse separate effects of training vs feedback.
• Generality: Sample = young, highly educated graduate students; replication needed with paraprofessional direct-care staff, nursing aides, etc.
• Scope: Only Training & Consequence deficits evaluated; future work should test interventions for Clarification or Resource issues.
• IOA on PDC-HS administration unmeasured – need reliability studies.
• Potential alternative non-indicated interventions (goal-setting, supervisor presence) not examined.
• Indirect nature limits certainty; advancement could involve blended checklists with direct behaviour sampling.

References & Foundational Literature (Select)

• Austin (2000); Austin, Carr & Agnew ($1999$) – OBM assessment need.
• Burgio et al. ($1990$) – Staff management for continence.
• Iwata et al. ($1994$) – Predictive-validity logic adopted here.
• Rodriguez et al. ($2005$); Pampino et al. ($2003$) – Prior PDC empirical applications.
• Reid & Parsons ($2000$) – Staff management review in human services.

Appendix Highlights

Treatment-Room Cleanliness Checklist (Safety, Organisation, Floors)

• Safety & Sanitation: Hand-sanitiser pumps, glove boxes in three sizes, Clorox wipes, stocked first-aid kit.
• Panic Button: Present & room-matched, alarms functional.
• Organisation: Essential items only on counters, furniture placement, labelled toy bins, closed cabinets.
• Floors & Tables: No trash left.

PDC-HS Item Examples

• Training #1: "Has the employee received formal training?" (Instruction, Demonstration, Rehearsal tick-boxes).
• Task Clarification #3*: "Is a job aid visibly located in the task area?"
• Resources #2*: "If materials are required, are they readily available?" (List blank lines for $\ge4$ items).
• Consequences #2: "Does the employee receive feedback?" (Specify type, delay, frequency).

Key Statistics and Data Points

• Instrument length: $20$ items, $4$ sections.
• Participants: $n = 15$ therapists; Supervisors interviewed: $n = 3$ (BCBAs).
• IOA means: $91 – 97\%$ across rooms.
• Effect sizes: Performance increases of $+51\%$ to $+75\%$ per room under matched package; $-6\%$ to $+11\%$ under non-matched control.
• Training & Feedback intervention maintained by programme post-study (informal social-validity indicator).