Lecture Notes on Reliability and Validity

Flashcards on Reliability and Validity

Reliability

The consistency in measurement.

Reliability coefficient

A statistic that quantifies reliability (0-1), where 0 is not reliable at all and 1 is perfectly reliable.

Measurement error

The inherent uncertainty associated with any measurement, even after care has been taken to minimize preventable mistakes.

Random error

Unpredictable fluctuations and inconsistencies of other variables in the measurement process.

Systematic error

Typically proportionate to what is presumed to be the true value of the variable being measured.

True score

The measurement of quantity if there were no measurement error at all.

Construct score

Persons standing on a theoretical variable that is independent of any particular measurement.

Observed score

Actual measurement obtained when assessing a person.

True score

Long-term average of measurement scores.

Test construction

Variation may exist within items in a test or between tests (item sampling or content sampling).

Test administration

Test taker variables include: pressing emotional problems, physical discomfort, lack of sleep, effects of drugs or medication. Examiner-related variables include: physical appearance and demeanor may play a role.

Test scoring and interpretation

Computer testing reduces error in test scoring, but many tests still require expert interpretation. Example: tests of personality, tests of creativity, various behavioral measures, etc.

Sampling error

The extent to which the population of voters in the study actually was representative of voters in the election.

Methodological error

Result of interviewers not been trained properly, ambiguous wording in the questionnaire, or the presence of biased in the questionnaire.

Test-retest reliability

An estimate of reliability obtained by correlating pairs of scores from the same people on two different administrations of the same test. Only useful when assessing traits that do not change overtime.

COEFFICIENT OF STABILITY

With intervals greater than 6 months, the estimate of test-retest reliability.

Parallel forms reliability

Degree of the relationship between various forms of a test can be evaluated by means of a parallel-forms coefficient of reliability.

Parallel-forms coefficient of reliability

Is often termed COEFFICIENTS OF EQUIVALENCE.

Parallel forms reliability

Comparing scores on two different measures of the same quality.

Split-half reliability

An estimate of internal consistency of a test obtained by correlating two pairs of scores obtained from equivalent halves of a singular test administered once.

Inter-item consistency

To the degree of correlation among all the items on a scale.

Inter-rater reliability

Degree of agreement or consistency between two or more scorers (or rates) with regard to a particular measure.

Coefficient of inter-scorer reliability

Degree of consistency among scorers in a scoring of a test.

Kappa statistic

Statistical measure of inter-rater reliability. Used for more than 2 raters. Used in nominal (categorical) data.

Face validity

The extent to which a test appears to measure to the person being tested, as compared to what the test actually measures.

Content validity

Judgement of how adequately a test samples behavior representative of the universe of behavior that the test was designed to sample.

Test blueprint

A plan regarding the types of information to be covered by the items, the number of items tapping each component, and the organisation of the test.

Criterion

The standard against which a test or a test score is evaluated. Example: psychiatric diagnosis, index of alcohol intoxication.

Criterion validity

Judgement of how adequately a test score can be used to infer an individuals most probable standing on a criterion.

Concurrent validity

An index of the degree to which a test score is related to some criterion measure obtained at the same time (concurrently).

Predictive validity

The extent to which a given test can anticipate future occurrences.

Validity coefficient

A correlation coefficient that provides a measure of the relationship between test scores and the scores on teh criterion measure. It allows one to tell the extent to which the test is valid for making statements about the criterion.

Construct validity

Judgement about the appropriateness of inferences drawn from test scores regarding individual standings on a construct.

Construct

An informed, scientific idea developed or hypothesized to describe or explain behavior.

Evidence of HOMOGENEITY

How uniform a test is in measuring a single concept.

Evidence from DISTINCT GROUPS

Scores on a test vary in a predictable way as a function of membership in some group.

Convergent validity

Scores on the test undergoing construct validation tend to correlate highly in the predicted direction with scores on older, more established tests designed to measure the same (or similar) construct.

Discriminant evidence

Validity coefficient showing little relationship between test scores and/or other variables with which scores on the test should not theoretically be correlated.