Chapter 1: Test Reliability and Validity Concepts

These flashcards cover key concepts related to reliability, validity, and measurement in psychometrics, as well as the properties and methods associated with effective testing.

Reliability

Dependability or consistency of a test's scores.

Reliability Coefficient

An index that indicates the ratio between true score variance and total variance.

Classical Test Theory

Theory stating a score on a test reflects both true ability and error.

Measurement Error

Factors in measuring a variable that are unrelated to the variable of interest.

Type I Error

Rejecting a true null hypothesis, also known as a false positive.

Type II Error

Failing to reject a false null hypothesis, also known as a false negative.

Variance

A measure of the dispersion of scores on a test.

Reliability Coefficient Interpretation

A higher coefficient indicates greater reliability.

Random Error

Error caused by unpredictable fluctuations during measurement.

Systematic Error

Error that is constant or proportional to the true value being measured.

Item Sampling

Variation among items within a test affecting test-taker's score.

Test Administration Variables

Factors like motivation or environment that can influence test scores.

Test-Retest Reliability

Reliability estimated by correlating scores from two test administrations.

Parallel Forms Reliability

Reliability assessed using different forms of the test that measure the same ability.

Factorial Analysis

Analysis used to identify underlying factors or traits from test scores.

Split-Half Reliability

Reliability obtained by correlating scores from two halves of a single test.

Spearman-Brown Formula

Used to estimate internal consistency reliability from two test halves.

Inter-item Consistency

Degree of correlation among all items on a scale.

Coefficient Alpha

A measure used for internal consistency reliability in non-dichotomous tests.

Inter-scorer Reliability

Degree of agreement between scores assigned by different raters.

Kappa Statistic

Used to measure agreement between two or more raters.

Static Traits

Traits observed as relatively unchanging over time.

Dynamic Traits

Traits prone to change due to situational factors.

Criterion-Referenced Tests

Tests designed to indicate an individual's standing relative to a criterion.

Standard Error of Measurement (SEM)

Estimate of the amount of error in an observed test score.

Confidence Interval

Range of test scores likely to contain the true score.

Content Validity

Assessment of how adequately a test samples the behavior it is designed to measure.

Criterion-Related Validity

Judgment of a test's ability to predict expected outcomes on a criterion.

Convergent Evidence

High correlation between test scores and similar constructs.

Discriminant Evidence

Low correlation between test scores and unrelated constructs.

Face Validity

Extent to which a test appears to measure what it claims to measure.

Incremental Validity

The additional predictive power of a new test over existing tests.

Item Difficulty Index

Reflects the proportion of test-takers who answered an item correctly.

Item Discrimination Index

Measures how well an item differentiates between high and low scores.

Item Response Theory (IRT)

Models the relationship between test-taker ability and probability of correct answers.

Item Characteristic Curve

Graph plotting the probability of a correct response against latent trait levels.

Rasch Model

A one-parameter model in IRT focusing on item difficulty.

Utility

Practical value of a test in improving decision-making.

Expectancy Data

Tables indicating likely scores of test-takers based on prior data.

Selection Ratio

Ratio reflecting the number of people to hire versus available applicants.

Cut Score

Judgment-based reference point dividing data into classifications.

Multiple Hurdle Method

Multi-stage selection with cut scores for each predictor.

Angoff Method

Method for setting cut scores with panels of expert judges.

Known Groups Method

Collection of data to determine cut scores based on known traits.

Discriminant Analysis

Technique analyzing relationships between variables and group membership.

Validity Coefficient

Correlation measuring the relationship between test scores and criterion.

Test Blueprint

Plan detailing item coverage and organization in a test.

Homogeneity

Extent to which a test contains items measuring a single trait.

Heterogeneity

Degree to which a test measures multiple traits.

Leniency Error

Rater's tendency to score more generously.

Central Tendency Error

Tendency of raters to avoid extreme scores.

Halo Effect

Bias where overall impression affects specific ratings.

Test Reliability

Consistency of test scores across different occasions.

Observed Score

The actual score obtained in a test.

True Score

The score that reflects an individual's actual ability.

Errors of Measurement

Factors causing discrepancies between observed and true scores.

Criterion Contamination

When criterion measures include irrelevant factors.

Base Rate

Percentage of the population exhibiting a specific attribute.

Hit Rate

Proportion of correct identifications of a trait by a test.

Miss Rate

Rate at which a test fails to identify a specific trait.

False Positive

Incorrect identification of a trait when it is not present.

False Negative

Failure to identify a trait that is actually present.

Dynamic Assessment

Assessment considering the variability of traits over time.

Test Administration Procedures

Standardized conditions to minimize variability in test scores.

Criteria for Good Tests

Purpose, content, administration, and scoring standards.

Test Specification

Detailed description of what elements a test will assess.

Operational Definition

Clear description of the constructs being measured.

Validity and Bias

Assessment of fairness and accuracy in measuring constructs.

Psychometric Properties

Key characteristics that ensure a test's effectiveness.

Cost-Benefit Analysis in Testing

Financial and practical evaluation of testing methods.

Test Length Impact

Longer tests tend to have higher reliability.

Internal Consistency Reliability

Measure of item intercorrelation within a single test.

Test Performance Stability

Reliability of a test over time for stable traits.

Item Content Sampling

Ensuring item relevance to the construct being measured.

Aptitude Test

Test designed to predict an individual's potential for success.

Personality Test

Assessment of character traits and behavior patterns.

Achievement Test

Evaluation of knowledge or skills in a particular area.

Cohen's Kappa

Statistic for measuring inter-rater agreement, correcting for chance.

Fleiss Kappa

Kappa statistic for agreement across multiple raters.

Graphical Representation of Test Scores

Visual tools to interpret performance data in assessments.

Behavioral Observation

Process of tracking actions or responses in a test setting.

Test Revision Procedures

Methods for improving test items based on item analysis.

Extreme Group Method

Comparative analysis using separate groups based on test performance.

Reliability in Psychological Measurement

Accuracy and consistency of psychological evaluations.

Latent Trait Theory

Theoretical framework explaining unobservable characteristics.

Assessment of Test Quality

Evaluation criteria to ensure effective testing.

Implementation of IRT

Integrating item response theory into testing procedures.

Decision Study

Analysis focused on how test scores aid decision-making.

Test User Guidelines

Standards for administering and interpreting test results.

Multivariate Analysis Techniques

Complex statistical methods for analyzing multiple variables.