CHD 4630: Lesson 4 Pt.2

Measurement (continued)

What are the 3 components of scores?

1. True score

2. Systematic error

3. Random error

Systematic Error

Reflects influences from other constructs/systematic influences besides the desired one intended to measure

Random Error (idiosyncratic error)

Reflects nonsystematic influences on the score

• Affects individuals in a unique way that is unlikely to be repeated in the same way

• Ex. Errors in data entry, lost data, feelings triggered by wording of questions, mood differences….

Halo Effect

Cognitive bias in which our overall impression of a person, company, brand, or product (figurative halo) influences our judgement of the character of that person or the properties of that company, brand, or product

Response style: midpoint

Tendency to choose moderate categories rather than extreme categories on rating scales

Response style: extreme

Tendency to choose extreme categories rather than moderate categories

Response style: acquiescence

Tendency to say yes, agree, or “true”

Response style: disacquiescence

Tendency to say no, disagree, or “false”

Response style: social desirability

Tendency to give answers that are considered desirable by social consensus

Generosity Error (leniency error)

Situation in which a rater gives more positive/higher ratings due to that rater’s general tendency to be lenient or insufficiently critical

Contrast Errors

Error during a performance appraisal where evaluation of a person is impacted by previously evaluated or appraised individuals

• Ex. 1st person in row sets certain benchmark, affecting rating of others

Reliability

The extent to which a measure is free from random error

• Prerequisite for measurement validity (addressed before validity and easier to address than validity)

What makes a measure have high reliability?

• Results are free from random error

• Consistent scores when phenomenon being measured is not changing

Test-retest Reliability

Correlation between scores on the same measure administered on two separate occasions (often within 2-3 weeks)

• Reliable = scores correlate perfectly over short period of time

Internal Consistency Reliability

Measure of how well the items on a test measure the same underlying construct or idea

• Measure of internal consistency = Cronbach’s alpha

Cronbach’s Alpha (Alpha Coefficient)

Measure of internal consistence of a scale/questionnaire consisting of 2 or more items

• Measures (0-1):

  • How closely related a set of items are as a group

  • The extent to which a scale is a consistent measure of an underlying construct

What makes up a “good” alpha coefficient score?

• Minimum between 0.70 and 0.80

• Less tha 0.50 = unacceptable

Validity

The conclusions researchers reach about the quality of their research methodology

Validity: Construct validity

Degree to which a set of variables accurately reflects or measures the construct of interest

Construct validity: face validity

Extent to which a test is subjectively viewed as measuring the construct it purports to measure

• Transparency of a test

• LEAST scientific means of evaluating construct validity because of its subjectivity

Construct validity: criterion validity

The extent to which a measure is related to an outcome

Criterion validity: concurrent validity

The extent to which scores from a measure relate to an outcome that was measured at the same point in time

• Cross-sectional design

• Ex. Linking scores of IQ test with grades measured at same time

Criterion validity: predictive validity

The extent to which scores from a measure predict a variable that occur in the future

• Longitudinal design

• Ex. Videotape newly weds to predict divorce 3-6 years later

Construct validity: convergent validity

The extent to which 2 measures that are conceptually related are in fact empirically related

• Overlap between alternative measures

• Highly indicates that correlated measures are picking up common

Construct validity: discriminant validity (divergent)

Assesses whether constructs that are not supposed to be related are in fact unrelated to one another

• Ex. Low correlation between conscientiousness and social desirability

Problem: construct of disinterest

Variables measure almost always not only the construct of interest, but also irrelevant characteristics

Problem: random errors

Variables contain random errors of measurement

• Ex. Recording errors, grading errors, guessing the right answer…

Internal Validity

The extent to which conclusions can be drawn about the CAUSAL effect on one variable on another variable

• X ——> Y

• When high internal validity, can argue that the associations/effects are causal

What must conclusions require to be deemed causal?

• Causal variable and dependent variable are associated (there is an effect)

• Causal variable preceded dependent variable in time

• There are no other explanations for effects between variables

External Validity

The extent to which results can be generalized to other populations, settings, places, or times of interest

What are some threats to validity?

• Small sample size

• Inappropriate operationalization of variables

• Data coded incorrectly

• Misinterpretation of results

• Inappropriate statistical methods

• Incorrect conclusions