PY2501 Items and Scales 2024 - Tagged

Page 1: Introduction

• Course Title: PY2501 Research Methods and Data Analysis - Questionnaire Design: Items and Scales

• Instructor: Prof Andrew Schofield, based on slides by Prof Adrian Burgess

Page 2: Lecture Schedule

• Week 3: Theory of measurement

• Week 8: From items to scales

• Week 9: Assessing quality of scales and their meaning

Page 3: Scale Development Principles

• Don’t reinvent the wheel: Utilize existing reliable and valid scales when possible.

• Reliability and Validity: Critical yet challenging to achieve

Page 4: Psychometrics Overview

• Topics Covered:

  • Transition from items to scales

  • Creation of items

  • Scaling responses

  • Understanding multivariate scales

Page 5: Multidimensional Psychological Traits

• Psychological traits are multidimensional; fear consists of three components:

  • Physiological: Example indicators include GSR (Galvanic Skin Response), ECG (Electrocardiogram)

  • Behavioral: Fight, flight, freeze responses

  • Cognitive/Affective: Subjective emotional states

• Some phenomena require subjective inquiry through direct questions.

Page 6: Item Generation Techniques

• Sources for Item Development:

  1. Psychological Theory: Start with robust theoretical frameworks.

  2. Systematic Search: Includes literature review and expert consultation

Page 7: Deriving Items from Theory

• Define a psychological trait. For example, extroversion:

  • Questions might include:

    • "Do you enjoy going to parties?" (Yes/No)

    • Reversed statements such as "I like to sit quietly and read".

• Ensure all questions align with the trait definition, avoiding irrelevant questions about transient states.

Page 8: Systematic Search for Item Generation

• Semantic Links:

  • Explore related terms in dictionaries or web resources.

• Expert Consultation: Engage focus groups and informants for input.

Page 9: Ensuring Content Validity

• Items should encompass the entire spectrum of the domain (e.g., fear of spiders includes behavioral, physiological, cognitive/affective dimensions).

  • Example items:

    • “I feel an urge to run away at the sight of a spider.”

Page 10: Item Design Considerations

• Single Focus: Avoid composite questions (e.g., "I like parties and dancing").

• Readability: Strive for an 8-year reading age.

• Cultural Sensitivity: Consider potential translation or cultural misunderstandings.

Page 11: Scaling Overview

• Definition: Transform qualitative attributes into quantitative measures.

• Two Approaches to Scaling:

  1. Comparative Scaling

  2. Direct Estimation

Page 12: Comparative Scaling Techniques

• Techniques include:

  • Paired Comparison

  • Thurstone’s Method

  • Steven’s Scaling

  • Guttman Scaling

Page 13: Paired Comparison Technique

• Identifies the key aspects that determine lecturer quality based on preferences.

Page 14: Theoretical Basis for Paired Comparison

• Assumptions about latent psychological measurement scales and normally distributed weighting estimates.

Page 15: Algorithm for Paired Comparison

1. Pick a small set of relevant items.

2. Have judges compare each pair.

3. Convert preferences into z-scores.

4. Average z-scores for final scaling.

Page 16: Step-by-Step Paired Comparison

• Assess pairs of lecturer attributes for importance.

Page 17: Converting Preferences to z-scores

• Example: 60% preference yields a z-score of 0.26.

Page 18: Weight Conversion from z-scores

• Final scores convert z-scores to a scaled weight reflecting importance.

Page 19: Direct Estimation Method

• Participants provide direct ratings of their experiences (e.g., fear levels).

Page 20: Response Formats

• Various ways to quantify fear:

  • Dichotomous

  • Rating scales

  • Visual analogue scales

Page 21: Likert Scale Application

• Common formats to gauge responses to fear levels through various comparative phrases.

Page 22: Frequency, Intensity, Duration

• Considerations for how to ask about fear across different timeframes.

Page 23: Scaling Considerations and FAQ

• Balance between too few and too many response categories.

• Importance of clarity and context for adjectives used.

Page 24: Response Formats Continued

• Understand implications of number choices and consistency.

Page 25: Direct Estimation vs. Comparative Scaling

• Efficiency of direct estimation versus the reliability of comparative scaling.

Page 26: Quiz 1

• Topics covered in the first quiz.

Page 27: Advantages of Scales

• Improves accuracy and reliability, achieving interval level measurement.

Page 28: Classical Test Theory Overview

• Factors affecting observations and reliability calculations.

Page 29: Reliability and Validity Through Multiple Measurements

• Errors can cancel out over repeated measurements.

Page 30: Repeated Measurements Insight

• Challenges of repeated questions in psychological contexts due to memory effects.

Page 31: Classical Test Theory on Errors

• Discusses random error distributions in psychological measurement.

Page 32: Interval Level Assumptions

• Rating scales often viewed as ordinal but can be treated as interval under certain conditions.

Page 33: Justifications for Interval Level Data

• Theory supporting the treatment of ordinal data as interval data.

Page 34: Benefits of Using Interval Level Data

• Enables precise statistical analysis and inferences.

Page 35: Scale Correlation and Justification

• Importance of item correlation for scale validity and internal consistency.

Page 36: Correlation and Shared Variance Concept

• Correlation as a measure of variance and reliability of item sets.

Page 37: Correlation Categories

• Ranges of correlation coefficients indicating strength and shared variance.

Page 38: Internal Consistency Assessment

• Homogeneity in items influences scale reliability.

Page 39: Internal Consistency Reliability Criteria

• Correlated items validate the formation of a scale.

Page 40: Measuring Internal Consistency

• Methods such as item-total correlation, split-half reliability, and Cronbach’s Alpha.

Page 41: Cronbach's Alpha Interpretation

• Thresholds for assessing internal consistency of scales.

Page 42: Quiz 2

• Topics covered in the second quiz.

Page 43: Multivariate Psychological Constructs

• Understanding constructs like personality and intelligence, which consist of multiple factors.

Page 44: Analyzing Correlations in Multivariate Data

• Identifying potential multiple latent factors through correlation inspection.

Page 45: Factor Analysis Methodology

• Statistical approach to explore latent variable representation in scale measurements.