Module 3: Reliability and Validity

Assumptions in Test Development

Serves as a guide for psychometricians when developing psychological tests

1st Assumption

Traits and States Exist

2nd Assumption

Traits and States Can Be Measured

3rd Assumption

Test-Related Behavior predicts Non-Test Related Behavior

4th Assumption

Test and Other Measurement Techniques Have Strengths and Weaknesses

5th Assumption

Various Sources of Error Are Part of the Assessment Process

6th Assumption

Testing and Assessment Can Be Conducted in a Fair and Unbiased Manner

7th Assumption

Testing and Assessment Benefit Society

Trait

Enduring, stable personality characteristic that persists across time and situations (e.g., extroverted)

State

Temporary emotional or behavioral condition triggered by a specific situation or environment (e.g., emotions)

Constructs

Psychological Concepts (e.g., shyness, self-esteem)

Measurement Error

Errors within the test itself; errors that affect test reliability

Extraneous Variables

Any factor outside of your independent variable that has the potential to influence the results of your study or experiment (e.g., noise)

What is a good test?

A good test is reliable and valid

Reliability

Refers to the consistency of the test across different contexts, situations, cultures, and time

Validity

Refers to the judgment of how well a test measures what it intends to measure (accuracy)

Types of Measurement Error

Random and Systematic

Random Errors

External errors; unavoidable, unpredictable fluctuations in measurements or processes that occur by chance that affects the test’s reliability.

Test Administration Random Error

How a test is administered is a frequent source of random error; Gender of administrator, overall mood, temperature, situation/s

Systematic Errors

A consistent, predictable shift in measurement or data that deviates from the true value in one specific direction. Also known as bias.

Sources of Systematic Errors

Test Construction & Test Scoring and Interpretation

Test Construction

How items are made and its variations with another test item; Ex: phrasing, wording, sentence construction

Test Scoring and Interpretation

Qualifications of the test administrator, how scores are computed: Manual vs. Computerized?; format of test: Objective vs. Projective?

Assessing Test Reliability

1. Test Retest Reliability Estimates

2. Parallel-Forms and Alternate Forms Reliability Estimates

3. Split Half Reliability Estimates

4. Internal Consistency

5. Inter-scorer Reliability

Test Retest Reliability Estimates

The test is conducted on a pool of respondents at one point. After some time, they are tasked to answer the test again.Scores between the two tests are evaluated if they produced consistent results. More appropriate with static constructs.

Internal Consistency

Chronbach’s α (alpha). Most used method in measuring reliability by determining the tests reliability coefficient by using software and statistics.

Alpha Value 0.90++

Excellent Reliability Level

Alpha Value 0.80 - 0.89

Good Reliability Level

Alpha Value 0.70 - 0.79

Acceptable Reliability Level

Alpha Value 0.60 - 0.69

Questionable Reliability Level

Alpha Value 0.59 and below

Poor Reliability Level

Inter-Scorer Reliability

Test is shown to SMEs or subject matter experts. Then, we evaluate the consistency of scores between different experts. Usually, three (3) experts are chosen to prevent a “tie”

Homogeneous Test

A reliable test is homogeneous. This means that the test items are uniform in though.

Heterogeneous Test

Items are not uniform in thought and are varied

Dynamic Construct

Construct is changing rapidly overtime

Static Construct

Construct is consistent and stable

Assessing Test Validity

Test Validation, Face Validity, Content Validity, Criterion-related Validity

Test Validation

Process of gathering evidence about one’s validity

Face Validity

The degree to which a test, survey, or assessment appears to measure what it claims to measure at "face value”

3 Categories of Validity

Content Validity, Criterion-related Validity, Construct Validity

Content Validity

The extent to which a measurement instrument (like a test or survey) thoroughly covers all relevant facets of the theoretical concept or construct it aims to measure

Criterion-related Validity

Evaluates how accurately a test or measurement predicts an outcome by comparing it to an established external benchmark

Types of Criterion-Related Validity

Concurrent Validity and Predictive Validity

Concurrent Validity

How a test correlates to a “gold standard” test at the same point in time; We use an existing and psychometrically sound test and correlate it with our own existing test at the same time. (correlated = high in concurrent validity)

Predictive Validity

Refers to how well does your test predict future behavior; “Can this test predict something that will happen in the future?”

Construct Validity

Refers to how the test truly measures the theoretical construct of framework of the test; ensures a test accurately measures

Types of Construct Validity

Convergent Validity and Discriminant Validity

Convergent Validity

Your construct is positively correlated with another closely related construct; “Does this test agree with other similar measures?”

Discriminant Validity

This correlation indicates that the construct in your test is negatively correlated with another construct. This proves that your construct is distinct

Test Bias

Inherent factors in a test that interferes with the accuracy of the results (in the context of psychometrics)

Types of Biases in Psychological tests

Leniency Error, Severity Error, Central Tendency, Halo Effect, Horn Effect

Leniency Error

Type of error in which the rater has a tendency to be lenient in scoring; “Okay na ‘to”, “medyo tama naman”

Severity Error

Type of error in which the rater has a tendency to scrutinize the individual too much; “Nitpicking”

Central Tendency Error

Type of error in which the rater has a tendency to stay in the neutral or “safe” zone

Halo Effect

Tendency to give a particular ratee a higher score because they appear “nice”, attractive, pleasant

Horn Effect

Tendency to give a particular ratee a lower score because they appear unpleasant, unattractive, etc.