Threats to Validity

Axis of error

• alternative interpretations

• competing explanations

• extraneous variables

• confounding variables

Threats to internal validity

• testing effects

• history

• interactions

• mortality

• participant selection

• instrumentation

• regression

• experimenter/subject bias

• maturation 

Maturation threat

• the processes within subjects which act as a function of the passage of time.

• i.e. if the project lasts a few years, most participants may improve their performance regardless of treatment.

History Threat

• the specific events which occur between the first and second measurement.

• i.e. Effectiveness of a stuttering program. Only take pre and post treatment measures (what happened in between measurements period)

Testing Effects

• Also called “practice” effects and “reactive measures”

• The effects of taking a test on the outcomes of taking a second test

Statistical Regression Threat 

• it is also known as regression to the mean.

• This threat is caused by the selection of subjects on the basis of extreme scores or characteristics.

• Give me forty of the worst students and I guarantee that they will show immediate improvement right after my treatment.

Experimenter/Participant Bias

• bias introduced by an experimenter whose 

expectations about the outcome of the experiment can be subtly communicated to the participants in the experiment

• Acting differently with different experimental groups

Instrumental Threat

• The changes in the instrument, observers, or scorers which may produce changes in outcomes.

• Can be an instrument or a formal test

Participation/Selection Threat 

• the biases which may result in selection of comparison groups.

• How to avoid:

  • Establish Selection Criteria

  • Random Assignment

  • Matching on relevant variables across groups

Mortality threat

• loss of subjects

  • attrition

  • death

Threats to External Validity

• multiple TX interference

• reactive testing 

• participant selection bias 

• experimental arrangements 

Participant selection bias

• Were the participants in the investigation similar to the population to which the results can be generalized?

• Participant meets specific criteria (i.e. college student, male, right-handed, no history of head injury, etc.)

Experimental Arrangements

• Research can be done in multiple settings such as in the real environment or in a lab setting

• Ex. Validity can be more effected by an artificial setting.

  • i.e. testing auditory comprehension in a sound proof both compared to a classroom.

Reactive Testing 

reactive testing = Their responses to a test/questionnaire/measurement change their later behavior

• Any task or test given to a participant can influence their performance on later parts of the experiment.

  • Pretests, questionnaires, instructions to participant impacts results- these same instructions, tests may not be given in the outside world

  • Step 1 – Show the video: Participants watch the instructional video.

  • Step 2 – Give a questionnaire: You ask them how they feel about vocal abuse or how often they think they overuse their voice.

  What your teacher means:

  • That questionnaire is part of the “reactive testing”.

  • Why? Because answering it makes participants think differently about their voice.

  • Later, when they actually speak, they might change their speaking behavior, not just because of the video, but because the questionnaire made them more aware.

Multiple TX interactions 

• has multiple treatments are given to the same subjects, it is difficult to control for the effects of prior treatments.

Complex designs

• are logical extensions of simple designs:

• More than two levels of the IV

• More than one IV

• More than one DV

• May combine between-subjects and within-subjects designs (i.e., mixed-model)

Within subjects (repeated measures)

•Definition: same participants are exposed to all levels of the independent variable (IV).

•Each participant serves as their own control, and differences in the dependent variable (DV) are observed across conditions or time points.

•Example: If we want to test the impact of background noise on speech clarity, we could have participants perform a speech task under different levels of noise (no noise, low noise, high noise). Each participant would perform the task in all noise conditions.

Between-Subjects Design (practice with purple chart!!)

•Definition: different groups of participants are exposed to different levels of the independent variable. Each participant is only tested in one condition, and comparisons are made between groups.

•Example: To compare the effects of two types of therapy (e.g., articulation therapy vs. phonological therapy), we would have one group receive articulation therapy and another receive phonological therapy. The speech outcomes of the two groups would then be compared.

Within subjects design advantages/disadvantages

Adv

• reduces variability due to individual differences

• requires fewer participants

• high sensitivity to detecting small effects

Disadv

• potential for carryover effects

• need for counterbalancing options 

Within subjects best for situations where

• want to measure how the SAME participants respond to different conditions over time

between-subjects design best for situations where

• comparing differences between distinct groups, especially when exposure to multiple conditions is not feasible or practical

between subjects advantages/disadvantages

Adv

• avoids carryover effects

• easier to implement when one condition might permanently change participants (irreversible therapy effects) 

Disadvantages

• requires more participants

• may not control for pre-existing differences between groups 

Mixed Design ANOVA

•Example: Studying the impact of therapy type on speech improvement over time.

•IVs:

•Between-Subjects Factor: Therapy Type (articulation, phonological, combined)

•Within-Subjects Factor: Time (baseline, 6 months, 12 months)

•DV: Speech intelligibility score.

•Structure: Different therapy groups are evaluated at three time points to see if improvement differs over time depending on the therapy type.

Understanding Main Effects and Interactions

•What is a Main Effect?

•The main effect is the direct influence of one independent variable on a dependent variable, regardless of other variables.

•In CDS, this could be how a specific treatment method (e.g., voice therapy) affects patient outcomes (e.g., vocal loudness) on its own.

•Example:

•Independent Variable (IV): Treatment type (Voice Therapy vs. No Therapy)

•Dependent Variable (DV): Improvement in vocal loudness

•Main Effect: Voice therapy shows a significant improvement in vocal loudness, irrespective of other variables like age or gender.

Example of Main Effects Scenario 

•Scenario:

•Study Question: Does using an Expiratory Muscle Strength Trainer (EMST) improve swallowing outcomes in patients with dysphagia?

•Variables:

•IV1: Use of EMST (Yes/No)

•IV2: Type of dysphagia (Neurological vs. Age-related)

•DV: Improvement in swallowing function

•Main Effect Example:

•Main Effect of EMST: Regardless of the type of dysphagia, patients using EMST show a greater improvement in swallowing function compared to those not using it.

What is an interaction ?

•Definition:

•An interaction occurs when the effect of one independent variable depends on the level of another independent variable.

•In CDS, this could mean that the effect of a treatment might be different depending on patient characteristics (e.g., age, type of disorder).

•Example:

•IV1: Treatment type (Traditional therapy vs. EMST)

•IV2: Type of dysphagia (Neurological vs. Age-related)

•DV: Improvement in swallowing function

•Interaction: EMST might work better for patients with neurological dysphagia but not for those with age-related dysphagia.

Example of Interaction Study Question:

Study Question: Does the effectiveness of voice therapy depend on the age group of the patient?

•Variables:

•IV1: Therapy type (Voice Therapy vs. No Therapy)

•IV2: Age group (Younger adults vs. Older adults)

•DV: Improvement in vocal loudness

•Interaction Example:

•Interaction Effect: Voice therapy might be more effective in improving vocal loudness for younger adults than for older adults, indicating an interaction between age and therapy type.

Creating a Matrix 

Let’s say the Independent Variables are:

•IV1: Treatment type (Voice Therapy vs. No Therapy)

•IV2: Age Group (Younger Adults vs. Older Adults)

•Dependent Variable: Improvement in Vocal Loudness

Main effect and Interaction Matrix

Main Effects: Look across rows or columns independently. For example:

•In the "Voice Therapy" row, both groups (Younger and Older Adults) show improvement, which is the main effect of therapy.

•In the "Younger Adults" column, both therapy and no therapy show higher improvement, indicating the main effect of age.

•Interactions: Compare how the effect of one variable changes based on the level of the other variable.

•Notice how Voice Therapy works better for Younger Adults than Older Adults (moderate improvement for older adults indicates an interaction).

Explanation of the Matrix

•Main Effect (Voice Therapy):

•Regardless of age, patients who received Voice Therapy show a higher improvement compared to those who did not receive therapy. This indicates a main effect of Voice Therapy.

•Main Effect (Age Group):

•Regardless of whether they received therapy or not, Younger Adults tend to show greater improvement in vocal loudness compared to Older Adults. This indicates a main effect of Age Group.

•Interaction:

•The interaction occurs where the effect of Voice Therapy is more pronounced in Younger Adults compared to Older Adults. Older adults benefit from therapy but not as much as younger adults, suggesting the treatment's effectiveness depends on age.

Main Effect 

The direct influence of an independent variable (IV) on a dependent variable (DV), ignoring other IVs in the study. It occurs when changes in the levels of an IV result in consistent changes in the DV.

•Example (Within-Subjects Design - Noise Levels):

•Independent Variable: Noise Level (no noise, low noise, high noise).

•Dependent Variable: Speech clarity score.

•Main Effect: There is a main effect of noise level if speech clarity scores significantly decrease as noise level increases. This means that the noise level itself, regardless of other factors, directly impacts the speech clarity.

Interaction Effect

•Occurs when the effect of one independent variable on the dependent variable depends on the level of another independent variable. It indicates that the combined influence of two or more IVs on the DV is different from their individual effects.

•Example (Between-Subjects Design - Therapy Type and Noise Level):

•Independent Variables: Therapy Type (articulation, phonological) and Noise Level (no noise, low noise, high noise).

•Dependent Variable: Speech clarity score.

•Interaction Effect: There is an interaction effect if the impact of therapy type on speech clarity varies depending on the level of noise.

•For instance, phonological therapy may improve speech clarity more than articulation therapy under no noise, but under high noise, the benefits of phonological therapy might diminish, and articulation therapy might perform better. This suggests that the effectiveness of therapy type depends on the noise condition.

Graph Interpretation (ADD GRAPHS)

Graph Interpretation:

•Articulation Therapy is represented in light blue, and Phonological Therapy is in light green.

•As the noise level increases, the speech clarity scores decrease for both therapies, but the extent of the decrease differs:

•Phonological Therapy shows higher clarity scores than Articulation Therapy at lower noise levels, but the difference diminishes as noise increases.

This visual representation helps to clearly see how the effectiveness of each therapy type varies across different noise conditions, demonstrating the interaction effect. ​