Data Analysis Notes

Background

  • SCED logic is similar to experimental research in any discipline.

  • Aims to isolate causal relationships between independent variables (IVs) and dependent variables (DVs).

  • Nomothetic/group data aims to ascertain generalizable principles/laws.

  • Idiographic/SCD data aims to draw conclusions at the individual level.

  • Interpretation and analysis involve different criteria.

Validity Reminders

  • External Validity: The degree to which findings can be extended beyond the subject population, settings, measures, etc.

  • Internal Validity: The degree of confidence that the change in the DV was due to the manipulation of the IV and not to another factor.

  • The strength of a design (group or SCD) lies in its ability to eliminate alternative explanations.

  • Alternative explanations are also known as threats to internal validity, confounds, or extraneous variables.

  • A sound study ensures that the IV caused the change in the DV.

Threats to Internal Validity

  • Maturation Effect: Changes in the DV might be attributable to developmental, biological, or psychological processes.

    • Examples include puberty.

  • History Effect: Changes in the DV might be attributable to an event outside of the experimenter’s control.

    • Examples include a tornado or a fire.

  • Regression Toward the Mean: High/low scores that are statistically rare will move toward normative values in the absence of treatment.

Name That Threat!

  • Severe weather: History.

  • Participants getting older during the study: Maturation.

  • Attending support groups: History; maturation if it is over a longer period.

  • California wildfire (e.g., Camp fire): History.

  • Medication change: Maturation.

  • Clinically depressed patients serving as participants: Regression toward the mean.

Combating Threats

  • All studies have limitations.

  • Researchers use the following to control for threats:

    • Replication (both intrasubject and intersubject).

      • At least 3 phase changes. The more phase changes, the more control.

    • Repeated measurement of DV.

      • This helps combat threats to internal validity.

Visual Analysis

  • Data in SCD are mostly presented via line graphs.

  • Visual inspection remains a viable, long-practiced method of analysis.

  • Underlying rationale: Investigators seek variables that attain potent effects that should be obvious from merely inspecting your data.

  • Visual inspection is viewed as a more conservative measure than statistical evaluation in between-groups designs.

    • If the effect isn’t visually obvious, the treatment may not be effective.

    • SCED is more likely to have Type 2 errors.

Visual Inspection Criteria

  • Criteria for visual inspection depend on:

    • Magnitude of changes across phases

      • Mean

      • Level

    • Rate of change

      • Trend/slope

    • Latency

Visual Inspection Criteria: Changes

  • Changes in means: Refers to shifts in the average rate of performance.

  • Changes in level: Refers to the shift or discontinuity of performance from the end of one phase to the beginning of the next phase.

    • Change in level is independent of change in mean.

  • Changes in trend: Refers to the tendency for data to show increases or decreases over time.

  • Latency of the change: Refers to the period between the onset or termination of one condition and changes in performance.

Visual Inspection Criteria: Examples

  • Changes in Means are demonstrated via graph.

  • Changes in Means vs. Level are demonstrated via graph.

  • Changes in Level are demonstrated via graph.

  • Changes in Trend/Slope are demonstrated via graph.

  • Latency of the Change is demonstrated via graph.

  • Weisman, 2006 Increasing % of opportunities of fast-food employees asking customers to “up-size”.

    • Changes in means - Yes

    • Changes in level - Yes

    • Changes in trend/slope - Yes/No

    • Latency of change - Yes for both

Visual Inspection Concerns

  • Beyond previously mentioned criteria, also consider:

    • Variability within and across phases

    • Stability

    • Replication of effects

    • Experimental design used

  • Concerns:

    1. No concrete decision rules (i.e., subjectivity).

    2. Multiple influences in reaching a decision (i.e., no weights assigned to various criteria).

    3. Search for only marked effects (i.e., overlooking weak but reliable effects; Type II errors)

Benefits of Graphic Displays and Visual Analysis

  • Immediate access to the record of behavior

  • Variations prompt exploration

  • Provides judgmental aid

  • Visual analysis takes less time and is relatively easy to learn

  • Imposes no arbitrary level for determining significance

  • Conservative method

  • Encourages independent judgments and interpretations

Quantitative Analysis

  • Many believe visual inspection should be used in conjunction with quantitative analysis

  • Refers to a large collection of mathematical practices:

    1. Mean lines

    2. Trend lines/split-middle line of progress/least squares regression line

    3. Dual-criteria analysis/conservative dual criteria analysis

    4. Effect sizes (e.g., percentage of nonoverlapping data; PND)

    5. Statistical process control (SPC) charts

    6. Tests of statistical significance

Questionable PND

  • PND Interpretation displayed via graph.

    PND=57=71PND = \frac{5}{7} = 71

Statistical Significance Tests

  • Highly controversial!

  • One main assumption needed: “independent observation.”

    • Each observation is independent of every other observation.

    • Easily met in group designs (obs 1 for subject 1, obs 2 for subject 2, etc.).

    • Not the case in SCDs (serial dependency; AKA: autocorrelation).

  • Another large shortcoming of “inferential statistics”:

    • Generally the aim of group designs/nomothetic approach.

    • Not typically the aim of SCDs/idiographic approach.

  • Therefore, much disagreement in this area.

Summary

  • SCD research → causes of bx change across different phases of a study

  • Researchers attempt to control for threats via:

    • Replication

    • Repeated measurement

  • Visual inspection historically used to evaluate effects

    • Large between-phase changes more easily observed

    • Might increase Type II errors

  • Some argue visual inspection is inherently biased

  • Critics → adopt quantitative analysis

  • However, no consensus exists re: which methods are best

  • Authors recommend: both visual and quantitative analysis