Comprehensive Guide to Psychological Assessment and Testing

Educational Assessment

Use of tests and other tools to evaluate abilities and skills relevant to success or failure in a pre-/school context.

Retrospective Assessment

Use of evaluative tools to conclude psychological aspects of a person as they existed at some point in time before the assessment.

Remote Assessment

Use of psychological tools to gather data and draw conclusions about subjects who are not in physical proximity to the evaluator.

Ecological Momentary Assessment

Evaluation of specific problems and related cognitive and behavioral variables at the very time and place that they occur.

Psychological Testing

Process of measuring psychological-related variables by means of devices or procedures designed to obtain a sample of behavior.

Psychological Assessment

Gathering and integration of psychology-related data for the purpose of making a psychological evaluation.

Psychological Test

A device or procedure designed to measure variables related to psychology.

Dynamic Psychological Assessment

Interactive approach; evaluation-intervention-evaluation ("sandwich method").

Collaborative Psychological Assessment

Assessor and assessee may work as partners from the initial contact to the final feedback.

Therapeutic Psychological Assessment

Therapeutic self-discovery and new understandings throughout the process.

Cut Score

A reference point, usually numerical, derived by judgment and used to divide a set of data into two or more classifications.

Psychometric Soundness

Technical quality of psychological tests and assessments.

Psychometric

The science of psychological measurements.

Psychometrician

A professional who uses, analyzes, and interprets psychological test data.

Classical Test Theory

A score on an ability is presumed to reflect not only the test taker's true score but also the error.

Error

The component of observed test score that does not have to do with the test taker's ability.

Variance

Useful in describing sources of test score variability.

True Variance

Variance from true difference.

Error Variance

Variance from irrelevant random sources.

Measurement Error

All of the factors associated with the process of measuring some variables, other than variables being measured.

Reliability

Consistency in measurement.

Reliability Coefficient

An index of reliability, a proportion that indicates the ratio between the true score variance on a test and the total score.

Random Error

Source of error in measuring a targeted variable caused by unpredictable fluctuations.

Systematic Error

Source of error in measuring a variable that is typically constant or proportionate to what is presumed to be the true value.

True Score formula

𝑇𝑟𝑢𝑒 𝑆𝑐𝑜𝑟𝑒= 𝑅𝑥𝑥(𝑥−𝑥) + 𝑥

Item Sampling/Content Sampling

Variation among items within a test as well as to variation among items between tests.

Practice Effect

Scores on the second session are higher due to their experience of the first session of testing.

Test Sophistication

Items are remembered by the test takers, especially the difficult ones/items that we got highlighted as confused.

Test Wiseness

Might inflate the abilities of test takers.

Mortality

Problems in absence in second session (just remove the first test of the absents).

Test Environment

Room temperature, level of lighting, and the amount of ventilation and noise.

Test Taker Variable

Emotional problems, physical discomfort, lack of sleep, effects of drug, formal learning experiences, casual life experiences, therapy, illness, and changes in mood or mental state.

Examiner-related Variables

Examiner's physical appearance and demeanor, nonverbal gestures, and professionalism.

Parallel Forms Reliability

Item sampling and other errors have affected scores on versions of the same test.

Alternate Forms Reliability

An estimate of the extent to which these different forms of the same test have been affected by item sampling error, or other errors.

Test-Retest Reliability

Obtained by correlating pairs of scores from the sample people on two different administrations.

Counterbalancing

A technique to avoid carryover effects for parallel forms, by using different sequence for groups.

Coefficient of Stability

The longer the time that passes, the greater the likelihood that the reliability coefficient will be lower.

Pearson Product-Moment Correlation Coefficient (r)

Typically calculated between the scores from the two administrations.

Carryover Effects

Happens when the test-retest interval is short, wherein the second test is influenced by the first test.

Internal Consistency (Inter-Item Consistency)

Degree of correlation among all the items on a scale.

Homogeneity

Single factor test measure.

Heterogeneity

Multiple factor test measure.

Kuder-Richardson Formula 20

Used exclusively for tests where items are dichotomously scored.

Rulon's Formula

A method for assessing the consistency of a test by comparing the scores on two halves of the test.

Odd-Even Reliability

Assign odd-numbered items to one half of the test and even-numbered items to the other half.

Interrater Reliability

Degree of agreement or consistency between two or more scorers with regards to particular measure.

KR21

Dichotomous items; all the test items have approximately the same degree of difficulty.

Cronbach's Alpha (α)

Arguably the most widely used and reported measure of internal consistency.

McDonald's Omega (ω)

A measure of internal consistency reliability, similar to Cronbach's Alpha.

Average Proportional Distance

A measure used to evaluate internal consistency of a test.

Split-Half Reliability

Correlating two pairs of scores obtained from equivalent halves of a single test administered once.

Coefficient of Interrater Reliability

Way of determining the degree of consistency among scorers in the scoring of a test.

Kappa Statistics

Used formula for nominal data.

Spearman-Brown Formula

Estimate internal consistency reliability from a correlation of two halves of a test.

Fleiss Kappa

Agreement between multiple raters (three or more).

Cohen's Kappa

Agreement between two raters.

Kendal's W

Used for rankings or ordinal data.

Domain Sampling Theory

Seek to estimate the extent to which specific sources of variation under defined conditions are contributing to the test score.

Generalizability Theory

A person's test scores vary from testing to testing because of variables in the testing situation.

Dynamic Characteristics

A trait, state, or ability presumed to be ever-changing as function of situational and cognitive experience.

Static Characteristics

Presumed to be relatively unchanging.

Restriction vs. Inflation of Range

Restricted means lower correlation coefficient; Inflated means higher correlation coefficient.

Universe

Describe the details of the particular test situation.

Facets

The number of items in the test, the amount of training the test scores have had, and the purpose of test administration.

Universe Score

The exact same conditions of all the facets in the universe, the exact same test score should be obtained.

Generalizability Study

Examines how generalizable scores from a particular test are if the test is administered in different situations.

Decision Study

Developers examine the usefulness of test scores in helping the test user make decisions.

Item-Response Theory (Latent-Trait Theory)

Provide a theory way to model the probability of that a person with X ability will be able to perform a level of Y.

Power Test

When the time limit is long enough to allow test takers to attempt all items and if some items are so difficult that no test takers can obtain a perfect score.

Speed Test

Contains items of uniform level of difficulty so that, when given generous time limits, all test takers should be able to complete all the test items correctly.

Criterion-Referenced Test

Assigned to provide an indication of where a test taker stands with respect to some variable or criterion.

Latent trait

Unobservable.

Manifestation trait

Observable.

Discrimination

An item differentiates among people with higher or lower levels of the trait, ability, etc., that is being measured.

Dichotomous Test Items

Only two possible responses.

Polytomous Test Items

More than two responses.

Standard Error of Measurement

Provide an estimate of the amount of error inherent in an observed score of measurement.

Standard Error of Score

Index of the extent to which one individual's score vary.

True Score

A value that, according to CTT, genuinely reflects an individual's ability (or trait) level as measured by a particular test.

Confidence Interval

A range or band of the test scores that is likely to contain the true score.

Validity

A test measures what it purports to measure in a particular context.

Validity Coefficient

Correlation coefficient that provides a measure of the relationship between test scores and scores on the criterion measure.

Inference

Is a logical result or deduction.

Validation

Process of gathering and evaluating evidence about validity.

Ecological Validity

Refers to a judgment regarding how well a test measures what it purports to measure at the time and place that the variable being measured is actually emitted.

Internal Validity

Degree of control among variables in the study (increased through random assignment)

External Validity

Generalizability of the research results (increased through random selection)

Conceptual Validity

Focuses on individuals with their unique histories and behaviors

Face Validity

A test appears to measure what it claims to measure (e.g., intelligence, depression, anxiety)

Construct Underrepresentation

Failure to capture important components of a construct

Construct-Irrelevant Variance

Happens when scores are influenced by factors irrelevant to the construct

Content Validity

An evaluation of the subjects, topics, or content covered by the item in the test

Relevance

Are directly pertinent to the construct being measured

Representativeness

Adequately sample all important sub-domains or facets of the construct

Criterion-Related Validity

Evaluating the relationship of scores obtained on the test to scores on other tests or measures

Importance of Content Validity

Ensures that the instrument is indeed measuring what it claims to measure, leading to more accurate results

High Content Validity

Enhances the credibility and trustworthiness of the assessment process and its outcomes

Content Validity as a Psychometric Property

Addresses the extent to which a measurement instrument adequately represents all relevant facets of the construct

Test Blueprint

A detailed plan of the content, organization, and quantity of the items that a test will contain