Systematic Protocols for the Visual Analysis of Single-Case Research Data
Abstract
Researchers in applied behavior analysis (ABA) and fields like special education and school psychology utilize single-case designs to evaluate causal relationships between variables and the effectiveness of interventions.
Visual analysis is the leading method to analyze single-case research (SCR) data.
Concerns exist regarding the reliability of visual analysis due to lack of formal guidelines, leading to influence from subjective and idiosyncratic factors.
This paper introduces systematic protocols for visual analysis of A-B-A-B and multiple-baseline designs, resulting in numeric scores for analyses.
The paper details protocol content, application to graphs, a small-scale evaluation study, and future considerations.
Keywords
Visual analysis
Single-case research
Visual inspection
Data analysis
Introduction to Single-Case Research (SCR)
SCR is the main methodology for evaluating causal relations and the effectiveness of interventions in applied behavior analysis, special education, and psychology (Horner et al., 2005; Kazdin, 2011).
Focuses on individual cases as the unit of analysis, enabling detailed examination of intervention effectiveness.
Allows for adaptive phase changes and the identification of optimal treatment components through component analysis (Barton et al., 2016; Horner et al., 2005).
Visual Analysis Methodology
Visual analysis is the primary analysis technique for SCR data, assessing the presence of causal relations (functional relations, experimental control) through examination of graphed data.
Analysts observe changes in:
Level: Amount of behavior on the y-axis, indicating overall behavior occurrence. (Barton et al., 2018)
Trend: Direction over time (increasing, decreasing, flat) (Barton et al., 2018).
Variability: Fluctuation of data around the trend line (Barton et al., 2018).
Identification of basic effects occurs through analysis of level, trend, and variability across adjacent phases.
Causal relationships are established through multiple replications of effects at different times (Kratochwill et al., 2013).
Reliability Issues in Visual Analysis
Concerns about interrater agreement among analysts exist, highlighting inconsistency in evaluating the same graph (e.g., Brossart et al., 2006; DeProspero & Cohen, 1979; Wolfe et al., 2016).
Wolfe et al. (2016) observed minimal interrater agreement (ICC = .601 and .58) on 31 multiple-baseline design graphs analyzed by 52 experts.
Lack of formal guidelines leads to subjectivity in the visual analysis of SCR data within the field, affecting its credibility as a research methodology (Furlong & Wampold, 1982).
Smith (2012) revealed that fewer than half of published SCR studies between 2000-2010 explicitly stated their analytic methods.
Standards and Protocol Development
Recent initiatives (e.g., What Works Clearinghouse Single-Case Design Standards) aim to standardize visual analysis, suggesting a four-step process emphasizing level, trend, variability, immediacy, overlap, and consistency; however, they do not provide detailed protocols for conducting visual analysis.
Maggin et al. (2013) developed a visual analysis protocol that improved agreement among visual analysts but still lacked a way to synthesize multiple data characteristics effectively.
Despite a structured protocol's capabilities, variability in expert requirement for basic effects leads to differences in conclusions about functional relations (Wolfe et al., 2016).
Development of Systematic Protocols for Visual Analysis
Developed web-based protocols for A-B-A-B and multiple-baseline design analysis, guiding analysts through evaluating SCR data with yes/no response options.
Protocols include systematic questions based on recommendations from published standards and textbooks, aiming to enhance the reliability of visual analysis.
Content and Structure of Protocols
Accessibility: Available as web surveys and Google Sheets; each protocol comprises specific questions corresponding to phases and contrasts in SCR.
Questions facilitate systematic evaluation of data stability and predictability within phases, leading to assessments of basic effects and experimental control.
Each question is structured to address six characteristics of data from the WWC standards.
Specific questions categorize data into weighted importance based on their role in demonstrating functional relation.
Scoring Mechanism
Experimental control score for the graph ranges from 0 (no functional relation) to 5 (strong functional relation) based on analysts' categorical responses.
Questions related to basic effects contribute 1 point each, while additional contributing items such as immediacy and non-overlapping data are weighted less (e.g., 0.25 points).
High standard set for confidence in functional relations through need for three basic effects in adjacent phase changes.
Application Examples of Protocols
A-B-A-B Design Protocol
Involves systematic comparison of baseline and treatment phases to identify basic effects and causal relations through a steady routing of questions.
Example involving a graph depicting talk-outs across phases demonstrates utilization of the protocol to evaluate stability and predictability, immediacy of change, and data overlap, with individualized questions for unique characteristics of the data.
Multiple-Baseline Design Protocol
Similar to A-B-A-B but accounts for staggered arrival of treatment and vertical analysis of data across tiers.
Analysts compare data stability across tiers and assess the timing and nature of intervention impacts vertically.
Evaluation of Protocols
Reliability and Validity Evaluation
Initial reliability evaluation utilized 16 participants with graduate-level coursework in SCR, comparing results from a visual analysis rating scale versus the newly developed protocols for 48 SCR graphs.
Interrater agreement (ICC) evaluations resulted in significant improvements for the protocol group at posttest (ICC = .78), rising from fair to excellent agreement; whereas rating group remained stable (ICC = .63).
Majority of protocol group found protocols easy to use and more reflective of standard analysis, enhancing the overall understanding of SCR data analysis.
Validity through Expert Comparison
Expert analysis of the same graphs indicated similar functional relation determinations between protocol scores and expert ratings, exhibiting a strong correlation.
Future Directions
Ongoing refinement of protocols involves increased sample sizes and varied graph types, developing more explicit response instructions, and integration of additional expert recommendations.
The aim is to enhance usability across diverse participant backgrounds and ensure adaptation as research on visual analysis evolves.
Conflict of Interest
All authors declare no conflicts of interest.
Ethical Compliance
Study procedures adhered to ethical standards with the approval of the University of South Carolina Institutional Review Board; informed consent was secured for participant involvement.
References
Comprehensive reference list detailing contributions from the field regarding visual analysis methodology, SCR design standards, and empirical studies underlying the discussed protocols.
Abstract
Researchers in applied behavior analysis (ABA) and fields such as special education and school psychology utilize single-case designs to evaluate causal relationships between variables and the effectiveness of interventions. Visual analysis is the leading method for analyzing single-case research (SCR) data. However, concerns exist regarding the reliability of visual analysis due to the lack of formal guidelines, which leads to influences from subjective and idiosyncratic factors. This paper introduces systematic protocols for visual analysis of A-B-A-B and multiple-baseline designs, resulting in numeric scores for analyses. The paper details protocol content, application to graphs, a small-scale evaluation study, and future considerations.
Keywords
Visual analysis, single-case research, visual inspection, data analysis.
Introduction to Single-Case Research (SCR)
Single-case research (SCR) serves as the main methodology for evaluating causal relations and the effectiveness of interventions in applied behavior analysis, special education, and psychology (Horner et al., 2005; Kazdin, 2011). This methodology focuses on individual cases as the unit of analysis, which allows for a detailed examination of intervention effectiveness. SCR enables adaptive phase changes and helps in identifying optimal treatment components through component analysis (Barton et al., 2016; Horner et al., 2005).
Visual Analysis Methodology
Visual analysis is the primary technique used in SCR data analysis. This method assesses the presence of causal relations (functional relations, experimental control) through the examination of graphed data. Analysts observe three main components: level—indicating the amount of behavior on the y-axis; trend—defined as the direction over time (increasing, decreasing, or flat); and variability—the fluctuation of data around the trend line. Basic effects are identified through the analysis of level, trend, and variability across adjacent phases, while causal relationships are established through multiple replications of effects at different times (Kratochwill et al., 2013).
Reliability Issues in Visual Analysis
Concerns surrounding interrater agreement among analysts highlight inconsistencies in graph evaluations. Wolfe et al. (2016) observed minimal interrater agreement on 31 multiple-baseline design graphs analyzed by 52 experts, with an ICC of .601 and .58. The absence of formal guidelines results in subjectivity during the visual analysis of SCR data, thereby affecting its credibility as a research methodology (Furlong & Wampold, 1982). Furthermore, a study by Smith (2012) revealed that fewer than half of published SCR studies between 2000 and 2010 explicitly stated their analytic methods.
Standards and Protocol Development
Recent initiatives, such as the What Works Clearinghouse Single-Case Design Standards, aim to standardize visual analysis. They propose a four-step process emphasizing level, trend, variability, immediacy, overlap, and consistency, but do not provide detailed protocols for conducting visual analysis. Maggin et al. (2013) developed a visual analysis protocol which improved agreement among visual analysts, yet it still lacked an effective method for synthesizing multiple data characteristics. Even with a structured protocol, variability in expert requirements for basic effects leads to different conclusions regarding functional relations (Wolfe et al., 2016).
Development of Systematic Protocols for Visual Analysis
Web-based protocols for A-B-A-B and multiple-baseline design analysis have been developed to guide analysts in evaluating SCR data through yes/no response options. These protocols include systematic questions based on published standards and textbooks to enhance the reliability of visual analysis.
Content and Structure of Protocols
The protocols are accessible as web surveys and Google Sheets, with each protocol comprising specific questions related to phases and contrasts in SCR. Questions assist in systematically evaluating data stability and predictability within phases, culminating in assessments of basic effects and experimental control. Each question addresses six characteristics of data from the WWC standards, categorizing data into weighted importance based on their role in demonstrating functional relations.
Scoring Mechanism
The score for experimental control reflects the graph's range from 0 (no functional relation) to 5 (strong functional relation) based on analysts' categorical responses. Each question related to basic effects contributes 1 point, while additional contributing items such as immediacy and non-overlapping data are weighted less (for instance, 0.25 points). A high standard is set for confidence in functional relations, requiring three basic effects in adjacent phase changes.
Application Examples of Protocols
The A-B-A-B Design Protocol involves systematic comparison of baseline and treatment phases to identify basic effects and causal relations through a steady route of questioning. For example, a graph depicting talk-outs across phases demonstrates the protocol's utilization to evaluate stability and predictability and assess immediacy of change and data overlap, with individualized questions for the unique characteristics of the data. The Multiple-Baseline Design Protocol functions similarly but accounts for staggered arrival of treatment and vertical analysis across tiers, comparing data stability across tiers and assessing the timing and nature of intervention impacts vertically.
Evaluation of Protocols
Reliability and Validity Evaluation
An initial reliability evaluation utilized 16 participants with graduate-level coursework in SCR, comparing results from a visual analysis rating scale versus the newly developed protocols for 48 SCR graphs. Interrater agreement (ICC) evaluations exhibited significant improvements for the protocol group at posttest (ICC = .78), rising from fair to excellent agreement, while the rating group remained stable (ICC = .63). The majority of the protocol group found the protocols easy to use and more reflective of standard analysis, improving the overall understanding of SCR data analysis.
Validity through Expert Comparison
Expert analysis of the same graphs demonstrated similar determinations of functional relations between protocol scores and expert ratings, showing a strong correlation.
Future Directions
The ongoing refinement of protocols is focused on increasing sample sizes and diverse graph types, developing clearer response instructions, and integrating more expert recommendations. The goal is to enhance usability across different participant backgrounds and ensure adaptation as research on visual analysis evolves.
Conflict of Interest
All authors declare that they have no conflicts of interest.
Ethical Compliance
All study procedures adhered to ethical standards with the approval of the University of South Carolina Institutional Review Board; informed consent was secured for participation.