Reliability and Validity

Reliability

Consistency of test scores across occasions.

Measurement Error

Inaccuracy in test scores due to various factors.

True Score Variance

Variance attributable to actual ability or characteristic.

Observed Score Variance

Variance observed in test scores including errors.

Error of Measurement

Estimation of inaccuracies in a single score.

Chance Errors

Random fluctuations affecting test scores.

Error Variance

Variance irrelevant to the test's purpose.

Test Construction

Designing tests to minimize measurement errors.

Item Sampling

Selection of test items affecting score reliability.

Test Administration

Conditions under which a test is given.

Test-taker Variables

Personal factors affecting test performance.

Examiner-related Variables

Influences from the test administrator's behavior.

Hand-scoring

Manual evaluation of test responses.

Machine Scoring

Automated evaluation of test responses.

Objective Scoring

Scoring based on clear criteria without bias.

Subjective Scoring

Scoring influenced by personal judgment.

Correlation Coefficient

Statistical measure of score agreement.

True Score

Theoretical score without measurement errors.

Observed Score

Actual score obtained from testing.

Measurement Error Formula

X = T + E, where E is error.

Domain Sampling Model

Using limited items to represent broader constructs.

Psychological Testing

Assessing mental functions through standardized tests.

Test Reliability

Extent of true differences reflected in scores.

Sample Size

Number of items used in a test.

Reliability Analysis

Estimates error in test score predictions.

Test-Retest Reliability

Consistency of scores over two test administrations.

Time Sampling

Evaluating reliability by retesting over time.

Random Sampling

Items drawn randomly from a larger domain.

Sampling Error

Variability in estimates due to sample differences.

Coefficient of Correlation

Statistical measure of score relationship.

Constant Characteristics

Traits that remain stable over time.

Changing Characteristics

Traits that can vary over time.

Interval Length

Time between test administrations for retesting.

Generalizability

Extent to which scores apply across situations.

Unbiased Estimate

Score that accurately reflects true ability.

Test Manual

Document detailing test procedures and reliability.

Reliability Coefficient

Numerical value indicating test reliability.

Higher Reliability

Indicates less score variability over time.

Carryover Effect

First test influences second test scores.

Systematic Carryover

Uniform score changes across all test-takers.

Random Carryover

Unpredictable changes affecting some test-takers.

Practice Effects

Improvement due to prior test experience.

Time Interval Selection

Careful evaluation of testing session gaps.

Intervening Factors

Other influences affecting scores over time.

Alternate Form Reliability

Uses different test versions for reliability.

Equivalent Forms Reliability

Parallel forms yield comparable test results.

Temporal Stability

Consistency of scores over different times.

Content Sampling Error

Variability due to non-equivalent test forms.

Test Variability

Differences in scores across testing conditions.

Test Administration Interval

Gap between two test sessions.

Test-Taker Influence

Factors affecting individual test performance.

Skill Improvement

Enhancement of abilities through practice.

Testing Session

Occasion when a test is administered.

Evaluation of Tests

Assessing reliability and validity of assessments.

Reliability Assessment

Measuring consistency and stability of test scores.

Split-Half Reliability

Test divided into two halves for scoring.

Random Division

Method to split test items into halves.

Odd-Even System

Technique to divide tests into two parts.

Correlation Underestimation

Split-half results yield lower reliability estimates.

Spearman-Brown Formula

Estimates reliability for unequal test halves.

Cronbach's Coefficient Alpha

General reliability coefficient for non-dichotomous items.

KR20 Formula

Reliability estimate for dichotomous test items.

KR20 Formula Calculation

KR20 = N/N-1 {(s2 - ∑pq)/s2}.

Reliability Estimate (KR20)

Indicates test reliability based on item variance.

N in KR20

Total number of items on the test.

s2 in KR20

Variance of total test scores.

p in KR20

Proportion of correct responses for each item.

q in KR20

Proportion of incorrect responses for each item.

Coefficient Alpha Purpose

Estimates internal consistency of test items.

Good Reliability Range

.70 to .80 is acceptable for research.

High Reliability Requirement

.90 to .95 is crucial in clinical settings.

Increase Item Count

Method to improve low reliability estimates.

Factor Analysis

Examines item correlation with total test score.

Discriminability Analysis

Identifies items measuring different constructs.

Correction for Attenuation

Adjusts correlations for measurement error effects.

Validity

Extent to which a test measures its intended construct.

Test Scores Meaning

Interpretation of scores based on validity.

Agreement in Validity

Alignment between test scores and measured quality.

Test Measurement Question

Asks if the test measures what it claims.

Appropriate Inferences

Valid tests yield meaningful and useful conclusions.

Validity Degree

Validity exists on a continuum from weak to strong.

Statistical Summaries Limitation

Validity cannot be fully captured by statistics.

Performance Relationship

Links test performance to observable behaviors.

Type of Validity

Depends on measurement purposes and consequences.

Content Validity

Degree items represent the behavior universe sampled.

Sampling Issue in Content Validity

Focuses on adequacy of test item representation.

Expert Panel Review

Experts assess items for content validity.

Judging Test Items

Rate relevance based on domain specification match.

Behavior Domain Analysis

Ensure all major aspects are covered by items.

Overgeneralization Risk

Avoid assuming broader skills from specific tests.

Multiple Choice Spelling Test Limitation

Measures recognition, not dictation spelling ability.

Content Validation Procedures

Involves item choice and test specifications.

Test Specifications Drawing

Outline content areas and importance of topics.

Domain-Referenced Tests

Performance interpreted based on content meaning.

Occupational Tests

Designed for employee selection and classification.

Rating Scale for Content Validity

1-4 scale assesses item relevance to domain.

Systematic Analysis Requirement

Ensure comprehensive coverage of behavior domain.

Relative Importance of Topics

Indicates priority of content areas in tests.

Content Validation

Tests actual job skills and knowledge requirements.

Job Analysis

Assessment to ensure test resembles job activities.