PY2501 Week 9 Item Analysis - Tagged

Page 1

• Title: Item Analysis Lecture

• Institution: Aston University, Birmingham, UK

• Course Code: PY2501 Research Methods & Data Analysis

• Instructor: Dr. Ryan

Page 2

Purpose of Item Analysis

• Psychologists utilize questionnaires to assess psychological constructs, such as:

  • Personality

  • Anxiety

  • Self-esteem

  • Internal motivation

• Questionnaires provide indirect measures of these constructs primarily depending on self-reporting.

• The challenge for psychologists is to ensure that these questionnaires are reliable.

Reliability Examples

• (Very) Unreliable Questionnaire: Lacks consistent responses and accuracy.

• Reliable Questionnaire: Produces consistent responses and accurately measures psychological constructs.

Page 3

Concept of Reliability

• Reliability vs. Validity:

  • Reliability refers to the consistency of the measurement scale.

• Types of Reliability:

  • Internal Reliability (focus of this lecture)

  • Test-Retest Reliability

• Importance of measuring reliability for questionnaire items.

Page 4

Quick Quiz Questions

1. Identify a reverse scored item related to extraversion.

2. Define a questionnaire with good reliability.

3. Define a questionnaire with good validity.

4. Define a questionnaire that exhibits both good validity and reliability.

Page 5

Correlation Recap

• Correlation Coefficient Ranges:

  • Negative relationships:

    • -0.1 = weak

    • -0.3 = moderate

    • -0.5 = strong

    • -1 = perfect

  • Positive relationships:

    • 0.1 = weak

    • 0.3 = moderate

    • 0.5 = strong

    • 1 = perfect

• Correlational analysis measures the relationship between two continuous variables.

• Correlation coefficient (r) values range from -1 to 1, indicating strength based on absolute size.

Page 6

Strength of Correlation

• The strength is determined by the degree of scatter rather than slope.

• More scatter indicates a smaller correlation coefficient.

Types of Relationship Strength

• Perfect Positive: 1

• Strong Positive: 0.5

• Moderate Positive: 0.3

• No Correlation: 0

• Moderate Negative: -0.3

• Strong Negative: -0.5

• Perfect Negative: -1

Page 7

Structure of Lecture

1. Internal Reliability (recap)

2. Item Analysis: Improving Internal Reliability

   • Item-total correlation

   • Cronbach’s alpha if deleted

3. Additional checks:

   • Test-Retest Reliability

   • Validity Evidence

Page 8

Internal Reliability Recap

• Individual items should reflect the same construct.

• High correlation among items allows scores to be summed for a total variable score.

Page 9

Internal Reliability Explained

• Example: Beck's Depression Inventory (BDI)

  • Participants scoring high on one item should score high on related items.

Page 10

Measuring Internal Reliability

1. Split-half Reliability:

   • Randomly split the dataset and check correlation between halves.

   • Correct for reversed questions to ensure accuracy.

   • Issue: Potential variability based on split method.

Page 11

Measuring Internal Reliability with Cronbach’s Alpha

• Overcomes split-half issues.

• Provides an average correlation coefficient across all possible splits.

• Cronbach's alpha (α) value ranges between -1 and +1.

Page 12

Interpretation of Cronbach’s Alpha

• Internal consistency interpretation:

  • α ≥ .9: Excellent

  • .9 > α ≥ .8: Good

  • .8 > α ≥ .7: Acceptable

  • .7 > α ≥ .6: Questionable

  • .6 > α ≥ .5: Poor

  • .5 > α: Unacceptable

Page 13

Quick Quiz Questions on Internal Reliability

1. Best statistic to assess internal reliability?

2. Ideal value for optimal internal reliability?

3. What is indicated by calculating the average of item scores in a ten-item scale?

Page 14

Improving Internal Reliability

Page 15

Strategies to Enhance Internal Reliability

• Remove items reducing reliability.

  1. Item Total Correlation:

     • Check if item score correlates with total score excluding that item.

     • Good items correlate at least .4 with the total score.

  2. Cronbach’s Alpha if Deleted:

     • Check how alpha changes if an item is removed.

     • Goal is to achieve α ~.9.

Page 16

Example of Item Removal

• New 8-item depression scale developed.

• The shorter scale improves efficiency but needs reliability checks.

Page 17

Using Jamovi for Analysis

• Jamovi provides item-total correlations and Cronbach’s α if deleted.

• Each column represents questions; rows represent individual scores.

Page 18

Output Analysis

• Example output shows:

  • Cronbach’s α at .78 with all items included.

  • Identify problematic items with item-total correlation < .4.

Page 19

Item Removal Impact

• Assess effectiveness of removing items one at a time.

• Start with the worst-performing item for analysis.

Page 20

Subsequent Analysis After Item Removal

• After removing Q3, α increased to .81.

• Further examination suggests Q7 should also be removed to improve α to .85.

Page 21

Final Adjustments to Scale

• Final changes yield Cronbach’s α at .85 after removing Q3 and Q7.

• Q6 could also be removed but it has a reasonable correlation with other items.

Page 22

Quick Quiz Questions on Item Analysis

1. Recommendation regarding an item with high correlation and good internal reliability?

2. Decision-making on another item with high alpha if deleted score.

Page 23

Short Break

• Transition to Part 3: Overview of Test-Retest Reliability and Validity.

Page 24

Test-Retest Reliability

• Measures reliability over time.

• Correlation between Time 1 and Time 2 should exceed .7 for consistency.

Page 25

Recap on Validity

• Validity assesses if the questionnaire measures the intended constructs.

• No absolute way to demonstrate validity; evidence is gathered through criteria.

Page 26

Types of Validity

• Content Validity: Range of items evaluated by experts.

• Face Validity: Expert judgment on whether it measures the intended construct.

• Criterion Validity:

  • Concurrent: Correlation with existing measures.

  • Predictive: Ability to make future predictions based on the measure.

Page 27

Conclusion

• Thank you for participation!

• Questions can be directed to: r.blything@aston.ac.uk