psyc 217 floor & ceiling effects, strength of independent variable, sensitivity of dependent variable, & designing surveys - oct 6 lecture review

Variables of interest

Terminology used in correlational designs

Dependent variable

Term used in experimental designs

Ways variables can be measured

1. Self report; measures attitudes, beliefs, or mood

2. Behaviours & responses

3. Physiological

Examples of self report measures

Surveys, questionnaires, or interviews

Examples of behaviour & response measures

Scores on tests, reaction time, or memory

Examples of physiological measures

Heart rate, pupil dilation, blood pressure, or brain activity

What needs to be considered when selecting a dependent variable (DV)?

- Ceiling effect

- Floor effect

Ceiling effect

All participants are at maximum performance on the dependent variable (DV); cannot see the effect of the independent variable (IV), scores are the same across all conditions

Cause of ceiling effect

Questions that measure the dependent variable are too easy; all participants answered correctly

Example of ceiling effect

Participants never stop completing math problems, the researchers have to cut them off; the math problems were too easy

Floor effect

All participants are at minimum performance on the dependent variable (DV); cannot see the effect of the independent variable (IV), scores are the same across all conditions

Example of floor effect

All participants perform poorly on math questions with exponents & factorials; the math questions were too hard

Cause of floor effect

Questions that measure the dependent variable are too hard; all participants answered incorrectly

Strength of independent variable (IV)

Refers to how strong or weak a manipulation is

Sensitivity of dependent variable (DV)

Refers to how tough/resistant or sensitive/weak a manipulation is to change; how likely change is

Strong manipulations of the independent variable (IV) result in...?

Bigger or more drastic changes in the dependent variable

Weak manipulations of the independent variable (IV) result in...?

Smaller or subtler changes in the dependent variable

The conditions of a strong manipulation of the independent variable (IV) vary...?

Significantly/drastically

The conditions of a weak manipulation of the independent variable (IV) vary...?

Subtly

Tough/resistant manipulations of the dependent variable (DV) are...?

Unlikely to change

Responses to a tough/resistant manipulation of the dependent variable (DV) are...?

Stronger or lack a range for nuance

Sensitive/weak manipulations of the dependent variable (DV) are...?

Likely to change

Responses to a sensitive/weak manipulation of the dependent variable (DV) are...?

More sensitive to the independent variable (IV), moderate & on a spectrum

When to use a strong independent variable (IV) paired with a sensitive/weak dependent variable (DV)

Good for initial experiments to determine whether an effect exists; if the independent variable (IV) impacts the dependent variable (DV)

When to use a weak independent variable (IV)

Should be used after a strong independent variable (IV) has first been shown to affect the dependent variable (DV); good for follow up experiments

When to use a weak independent variable (IV) paired with a sensitive/weak dependent variable (DV)

Useful when finding small differences; can determine thresholds/minimum amounts or ranges

When to use a tough/resistant dependent variable (DV) paired with a strong independent variable (IV)

Useful to start with to determine whether a dependent variable (DV) is likely to change, is rarely used; e.g for polls

When to use a sensitive/weak dependent variable (DV)

Should be used after a strong independent variable (IV) paired with a tough/resistant dependent variable (DV) have demonstrated that responses tend to be strong & are unlikely to change; good with both types of independent variables (IVs)

Reasons why a dependent variable (DV) is resistant/tough

- The dependent variable (DV) measures strongly held beliefs

- The dependent variable (DV) makes significant demands of participants

If 2 experiments have equally sensitive dependent variables (DVs), how does weakening the independent variable (IV) affect the ability to detect an effect?

It is harder to detect an effect using a weak independent variable (IV); smaller differences across conditions

If 2 experiments have equally strong independent variables (IVs), how does increasing the sensitivity of the dependent variable (DV) affect the ability to detect an effect?

Is easier to detect an effect using a more sensitive/weak dependent variable (DV); results are more likely to change across conditions

How can the likelihood of detecting an effect be increased when using a weak independent variable?

By using a repeated measures/within groups design

Purposes of asking questions on surveys

- To analyze & understand trends

- For correlational research; whether 2 variables of interest are related

- As a measure of the dependent variable (DV) in an experiment

- As a manipulation check in an experiment; whether the prediction aligns with the results

Manipulation check

Questions designed to test whether responses align with expected outcomes/predictions of scores

Considerations when designing questionnaires

- What questions are asked & how they are worded

- How participants will respond & whether their responses will capture what is being measured

- Proofreading & piloting questions if designing questions; whether they are confusing or difficult

Information participants can be asked on questionnaires

- Attitudes & beliefs

- Facts & demographics; e.g age, household income, or nationality

- Behavioural frequencies; e.g how often someone exercises, sleeps, or drinks

Likert scale

Scale from 1-7 where 1 is strongly disagree & 7 is strongly agree; 4/neutral may be omitted

Scale of frequency

Ranges from always, often, sometimes, rarely, or never; can be unclear on the definitions

Scale of 0-100

Where 0 is the complete absence & 100 is the absolute maximum

Why not applicable/I don't know should be included on surveys

If a response does not apply, excluding this option makes answers inaccurate

Wording considerations for questions

- Simplicity

- Double barrelled questions

- Loaded questions

- Negative wording

Simplicity

Questions should avoid jargon, abbreviations, or complexity (e.g vocabulary or technical terms); should be obvious what the question is asking

Solution to simplicity

Avoid using complex vocabulary & technical terms; ask questions in a clear & concise way

Double barrelled questions

Assesses multiple variables of interest at the same time; often uses the word "&"

Issue with double barrelled questions

Using "&" implies all of the scenarios apply; participants may be unsure how to respond if only 1 condition is applicable

Solution to double barrelled questions

Separate out variables into single item questions; avoid using "&"

Loaded questions

Uses adjectives to sway participants towards an opinion or answer

Negative wording

Questions that use "not" or double negatives; can confuse participants & increase recording errors

Solution to negative wording

Only word questions to have positive wording

Question validity

Whether questions assess a measure they intend to

Pre testing

Others answer survey questions to test them before the actual experiment

Benefits of pre testing

- Researchers can ask for feedback

- Unclear or misunderstood questions can be revised before the experiment

Response sets

Tendency of participants to respond to all questions from a particular perspective or in a particular way; instead of answering as themselves

Solutions to response sets

- Questions designed to catch people answering the same way to all questions

- Open ended questions

Examples of designing questions to catch response sets

- Can switch the order of agreeing & disagreeing on adjacent questions; the answers can be conflicting or consistent, reveals whether the participant is answering in a sincere way

- Use of focus checks; participants are told to give a specific answer if they are paying attention

Social desirability/faking good

Tendency of participants to respond in the most socially acceptable way or as most people would

What increases the likelihood of social desirability/faking good

When surveys ask about sensitive or taboo topics

Solution to social desirability/faking good

- Emphasizing to participants that their anonymity will be guaranteed

- Include questions that catch lying; impossible questions that apply to everyone, even participants are reluctant to admit they apply

Example of an impossible question to catch lying

How often do you have racist thoughts?; answers should not be never, everyone has at least some

Pros of open ended questions

- Answers are rich, creative, &/or complex

- Can identify common themes across responses

Cons of open ended questions

- Responses can be ambiguous

- Participants may interpret questions in different ways

- Is time consuming to sort/group & interpret responses