Ch 5 (Identifying Good Measurement) Power Point Notes

MEASUREMENT CONCEPTS

LEVELS OF MEASUREMENT

  • Nominal Measurement

    • Definition: Categorical data that is qualitative in nature.

    • Example: Gender (male, female, non-binary).

  • Ordinal Measurement

    • Definition: Puts data in a ranking order.

    • Characteristic: Intervals are not equal; it is numeric.

    • Example: Ranking in a race - 1st, 2nd, 3rd place.

  • Interval Measurement

    • Definition: Data is organized in order with equal intervals but lacks a true zero point.

    • Example: Fahrenheit temperature scale (arbitrary zero point).

    • Additional Context: Although scales may be debated, they are treated as interval.

  • Ratio Measurement

    • Definition: Data is organized in order with equal intervals and has a true zero point.

    • Example: Income (where zero income means absence of income).

EXAMPLES OF AGE AS A LEVEL OF MEASUREMENT

  • Age Categories:

    • 18-35 years

    • 36-55 years

    • 56 years and above.

RELIABILITY

  • Definition: Consistency with which a measure assesses a given concept.

TYPES OF RELIABILITY

  • Test-Retest Reliability

    • Also known as: Temporal stability.

    • Definition: Ability of a test to produce consistent results when administered on repeated occasions under similar conditions.

    • How to Assess: Use the Test-Retest Method.

  • Internal Consistency

    • Definition: Degree to which components within a measure are related to one another.

    • How to Assess:

    • Split-half method.

    • Cronbach’s alpha, with values ranging between .00 and 1.00.

      • Value of .7 and above is considered acceptable; .8 is even better.

      • However, caution is required with very high values (> .95) which may indicate redundancy.

    • Note: A reliable measure does not necessarily imply a valid measure.

  • Interrater Reliability

    • Definition: The degree of agreement among raters measuring the same construct.

    • How to Measure:

    • Use Cohen’s Kappa or Intraclass Correlation.

    • Aim for at least 80% agreement among raters.

INTERNAL CONSISTENCY MEASUREMENT EXAMPLES

  • Sample Statements to Assess Internal Consistency:

    • "I am satisfied with myself."

    • "I feel like I have good qualities."

    • "I feel like I have worth."

    • "I am able to do things as well as most other people."

DEALING WITH UNSATISFACTORY INTERNAL CONSISTENCY

  • Steps to Enhance Reliability:

    • Inspect item-total correlations.

    • Remove items that do not relate well to others.

    • Ensure items measure common issues without excessive similarity.

    • To boost Cronbach’s alpha:

    • Increase the number of items on the scale.

    • Remove less reliable items.

    • Use established measures, where applicable.

PRACTICE SCENARIOS

  • Study Example by Dr. Valencia:

    • Research Goal: Examine whether narcissistic individuals have poorer social interactions compared to those who are not.

    • Initial Task: Identify participants' narcissistic traits using the Mayo scale.

    • Example Questions from the Mayo Scale:

    1. "I tend not to think about other people as much as I think about myself."

    2. "I do not have a high opinion of myself."

    3. "I think other people think I am really special."

ASSESSING SCENARIO RELIABILITY

  • Test-Retest Reliability Assessment:

    • Dr. Valencia gives the measure to students on Tuesday and again on Thursday to check score consistency.

    • This assessment type is known as Test-Retest Reliability.

  • Internal Reliability Assessment:

    • Analysis of individual questions' responses shows correlations conducive to checking internal consistency.

    • This assessment type is known as Internal Reliability.

  • Correlation Analysis:

    • Findings:

    • Correlation Coefficient between Questions 1 and 2: r=−0.73r = -0.73

    • Correlation Coefficient between Questions 1 and 3: r=0.74r = 0.74

    • Conclusion: There appears to be good internal reliability in the scale.

  • Internal Reliability Measurement Confirmation:

    • Ideal Result: α=0.95\alpha = 0.95 is favorable, while α=0.10\alpha = 0.10 or α=−0.03\alpha = -0.03 are undesirable.

INTERRATER RELIABILITY

  • Definition: Influenced by potential biases or unclear standards that may affect reliability.

  • Measurement: Interrater Reliability examines the degree of agreement among observers.

    • Evaluated using Cohen’s Kappa or Intraclass Correlation, aiming for at least 80% agreement.

CONSTRUCT VALIDITY

  • Definition: The extent to which a measure accurately reflects the theoretical construct it is designed to measure.

TYPES OF CONSTRUCT VALIDITY

  • Face Validity

    • Definition: Measures what it intends to measure based on face-value.

  • Content Validity

    • Concerns whether the content of a measure represents various aspects of the variable under consideration.

  • Criterion Validity

    • Evaluates whether the measure relates to a concrete outcome it should predict.

    • Predictive Validity: Measures the degree to which scores predict behavior in the future and can be assessed using correlation.

  • Convergent Validity

    • Definition: Scores on the measure relate to other measures of the same or related constructs.

    • Example correlations: Extroversion score on Big 5 relates to the user’s extroversion assessment.

  • Discriminant Validity

    • Definition: Scores on the measure should not correlate with unrelated constructs.

RELIABILITY AND VALIDITY RELATIONSHIP

  • Important Note: While a measurement can be reliable but not valid, it cannot be the case that a valid measure lacks reliability.

ADDITIONAL PRACTICE SCENARIOS

  • Case of Dr. Sheffield:

    • Role: Clinical psychologist focusing on pathological gambling.

    • Task: Create a new measure for pathological gambling consisting of 15 questions, taking 20 minutes to complete.

  • If Dr. Sheffield's measure does not accurately reflect pathological gambling, it would lack Validity.

  • After assessment of report, the supervisor concludes that the measure effectively tests components such as feeling restless when trying to quit gambling. This indicates evidence of Content Validity.

  • Comparison between participants of Gamblers Anonymous (GA) and Alcoholics Anonymous (AA) reveals how Dr. Sheffield obtains evidence for Criterion Validity.

  • In pursuing criterion validity, Dr. Sheffield must correlate his measure with known behaviors of the measured population a few weeks later after initial measurement, ensuring dependable evidence of criterion validity.

  • Lastly, Dr. Sheffield finds that his measure has positive associations with other measures, offering evidence for Convergent Validity.

    • Example: Measure correlated with general attitudes towards gambling.

  • To validate his measure further, Dr. Sheffield checks Discriminant Validity among GA participants and non-participants with the expectation that non-participants will not show an association with pathological gambling measures.

    • Ideal Results to Support Discriminant Validity: Non-GA attendees scoring lower than GA participants on the measure, confirming that no significant association exists between gambling measures and unrelated constructs like life satisfaction.