Ch 5 (Identifying Good Measurement) Power Point Notes
MEASUREMENT CONCEPTS
LEVELS OF MEASUREMENT
Nominal Measurement
Definition: Categorical data that is qualitative in nature.
Example: Gender (male, female, non-binary).
Ordinal Measurement
Definition: Puts data in a ranking order.
Characteristic: Intervals are not equal; it is numeric.
Example: Ranking in a race - 1st, 2nd, 3rd place.
Interval Measurement
Definition: Data is organized in order with equal intervals but lacks a true zero point.
Example: Fahrenheit temperature scale (arbitrary zero point).
Additional Context: Although scales may be debated, they are treated as interval.
Ratio Measurement
Definition: Data is organized in order with equal intervals and has a true zero point.
Example: Income (where zero income means absence of income).
EXAMPLES OF AGE AS A LEVEL OF MEASUREMENT
Age Categories:
18-35 years
36-55 years
56 years and above.
RELIABILITY
Definition: Consistency with which a measure assesses a given concept.
TYPES OF RELIABILITY
Test-Retest Reliability
Also known as: Temporal stability.
Definition: Ability of a test to produce consistent results when administered on repeated occasions under similar conditions.
How to Assess: Use the Test-Retest Method.
Internal Consistency
Definition: Degree to which components within a measure are related to one another.
How to Assess:
Split-half method.
Cronbach’s alpha, with values ranging between .00 and 1.00.
Value of .7 and above is considered acceptable; .8 is even better.
However, caution is required with very high values (> .95) which may indicate redundancy.
Note: A reliable measure does not necessarily imply a valid measure.
Interrater Reliability
Definition: The degree of agreement among raters measuring the same construct.
How to Measure:
Use Cohen’s Kappa or Intraclass Correlation.
Aim for at least 80% agreement among raters.
INTERNAL CONSISTENCY MEASUREMENT EXAMPLES
Sample Statements to Assess Internal Consistency:
"I am satisfied with myself."
"I feel like I have good qualities."
"I feel like I have worth."
"I am able to do things as well as most other people."
DEALING WITH UNSATISFACTORY INTERNAL CONSISTENCY
Steps to Enhance Reliability:
Inspect item-total correlations.
Remove items that do not relate well to others.
Ensure items measure common issues without excessive similarity.
To boost Cronbach’s alpha:
Increase the number of items on the scale.
Remove less reliable items.
Use established measures, where applicable.
PRACTICE SCENARIOS
Study Example by Dr. Valencia:
Research Goal: Examine whether narcissistic individuals have poorer social interactions compared to those who are not.
Initial Task: Identify participants' narcissistic traits using the Mayo scale.
Example Questions from the Mayo Scale:
"I tend not to think about other people as much as I think about myself."
"I do not have a high opinion of myself."
"I think other people think I am really special."
ASSESSING SCENARIO RELIABILITY
Test-Retest Reliability Assessment:
Dr. Valencia gives the measure to students on Tuesday and again on Thursday to check score consistency.
This assessment type is known as Test-Retest Reliability.
Internal Reliability Assessment:
Analysis of individual questions' responses shows correlations conducive to checking internal consistency.
This assessment type is known as Internal Reliability.
Correlation Analysis:
Findings:
Correlation Coefficient between Questions 1 and 2:
Correlation Coefficient between Questions 1 and 3:
Conclusion: There appears to be good internal reliability in the scale.
Internal Reliability Measurement Confirmation:
Ideal Result: is favorable, while or are undesirable.
INTERRATER RELIABILITY
Definition: Influenced by potential biases or unclear standards that may affect reliability.
Measurement: Interrater Reliability examines the degree of agreement among observers.
Evaluated using Cohen’s Kappa or Intraclass Correlation, aiming for at least 80% agreement.
CONSTRUCT VALIDITY
Definition: The extent to which a measure accurately reflects the theoretical construct it is designed to measure.
TYPES OF CONSTRUCT VALIDITY
Face Validity
Definition: Measures what it intends to measure based on face-value.
Content Validity
Concerns whether the content of a measure represents various aspects of the variable under consideration.
Criterion Validity
Evaluates whether the measure relates to a concrete outcome it should predict.
Predictive Validity: Measures the degree to which scores predict behavior in the future and can be assessed using correlation.
Convergent Validity
Definition: Scores on the measure relate to other measures of the same or related constructs.
Example correlations: Extroversion score on Big 5 relates to the user’s extroversion assessment.
Discriminant Validity
Definition: Scores on the measure should not correlate with unrelated constructs.
RELIABILITY AND VALIDITY RELATIONSHIP
Important Note: While a measurement can be reliable but not valid, it cannot be the case that a valid measure lacks reliability.
ADDITIONAL PRACTICE SCENARIOS
Case of Dr. Sheffield:
Role: Clinical psychologist focusing on pathological gambling.
Task: Create a new measure for pathological gambling consisting of 15 questions, taking 20 minutes to complete.
If Dr. Sheffield's measure does not accurately reflect pathological gambling, it would lack Validity.
After assessment of report, the supervisor concludes that the measure effectively tests components such as feeling restless when trying to quit gambling. This indicates evidence of Content Validity.
Comparison between participants of Gamblers Anonymous (GA) and Alcoholics Anonymous (AA) reveals how Dr. Sheffield obtains evidence for Criterion Validity.
In pursuing criterion validity, Dr. Sheffield must correlate his measure with known behaviors of the measured population a few weeks later after initial measurement, ensuring dependable evidence of criterion validity.
Lastly, Dr. Sheffield finds that his measure has positive associations with other measures, offering evidence for Convergent Validity.
Example: Measure correlated with general attitudes towards gambling.
To validate his measure further, Dr. Sheffield checks Discriminant Validity among GA participants and non-participants with the expectation that non-participants will not show an association with pathological gambling measures.
Ideal Results to Support Discriminant Validity: Non-GA attendees scoring lower than GA participants on the measure, confirming that no significant association exists between gambling measures and unrelated constructs like life satisfaction.