md3 lecture 11

What is a typical performance test?

A test that assesses an individual's typical preferences, or how someone normally behaves or performs.

What are examples of typical performance tests?

Personality inventories and attitude surveys.

What is a maximal performance test?

A test that assesses what an individual knows or can do when showing maximum effort. Each item has a known correct answer.

What are examples of maximal performance tests?

Aptitude tests and achievement tests.

What is a speed test?

A test that provides many relatively easy items.

What is a power test?

A test that allows plenty of time for completion and contains items that vary more in difficulty.

What are closed-ended items?

Items with predefined response options, such as multiple-choice, true–false, and matching questions.

What are open-ended items?

Items requiring answers ranging from a few words to several pages, such as short-answer or essay questions.

What is the main difference between open-ended and closed-ended items in development and evaluation?

Open-ended items are generally easier to create but much harder to evaluate than closed-ended items.

What should subject matter experts (SMEs) evaluate during item development?

Whether items assess the intended construct, whether instructions are clear, and whether common item-design pitfalls are avoided.

What is item difficulty in a maximal performance test?

The number of people who answer an item correctly, expressed as a p-value or the percentage of test takers answering correctly.

What p-value is generally preferred for item difficulty, according to the lecture?

Around 50%, to optimally discriminate among test takers.

Why does item difficulty depend on the test-taking population?

An item may be too easy for one population but appropriate for another.

What is an item-test (item-total) correlation?

The correlation between an item and the entire test score, including that item.

What is an item-rest correlation?

The correlation between an item and the total test score excluding that item.

What does the item-rest correlation indicate?

The extent to which an item measures the same construct as the other test items.

How are item-rest correlations interpreted in the lecture?

At least .30 = good; .20–.29 = sufficient; .19 or lower = insufficient.

What is a point-biserial correlation?

A correlation between a dichotomous item score, such as correct/incorrect, and a continuous test score.

What is a biserial correlation?

A correlation that corrects for the artificial dichotomy of an item score and is always larger than the corresponding point-biserial correlation.

How does an item's standard deviation and item-test correlation contribute to reliability?

The item's contribution to reliability is calculated as SD × rit. Higher SD and higher item-test correlation increase its contribution.

How does an item's standard deviation and item validity contribute to criterion validity?

The item's contribution to validity is calculated as SD × rxy. Higher SD and higher item validity increase its contribution.

What is norm-referenced testing?

Testing designed to maximally differentiate between test takers and determine a person's position within a particular population.

What is criterion-referenced testing?

Testing designed to establish whether test takers have achieved a certain level of competence.

What is the Angoff method?

A sample of SMEs estimates the probability that a minimally competent person will answer each item correctly. The probabilities are averaged across SMEs and the item probabilities are added up.

What is the contrasting-groups method for setting cutoff scores?

Identify two groups, such as successful and unsuccessful individuals, plot their score distributions, and set the cutoff at the intersection.

What is forward/reverse regression in setting cutoff scores?

They determine the test score corresponding to the minimum acceptable job performance: forward regression predicts performance from test scores, while reverse regression predicts test scores from performance.

What is Classical Test Theory (CTT)?

An approach to test evaluation that focuses on item difficulty and item discrimination and estimates a person's true score by summing the number of correct items.

What is the basic CTT equation?

X = T + E, where X is the observed score, T is the true score, and E is measurement error.

What is Item Response Theory (IRT)?

An approach that focuses on individual items and estimates the likelihood that a person with a given ability level will answer a particular item correctly.

What is theta (θ) in IRT?

The person's ability level.

What assumptions about item scoring and dimensionality are used in this lecture's IRT discussion?

Dichotomous item scoring (incorrect or correct) and unidimensionality.

What is an item characteristic curve (ICC)?

describing the relationship between a person's ability level and the probability of answering an individual item correctly.

What does parameter a represent in the three-parameter logistic IRT model?

Item discrimination: how well an item distinguishes between individuals with different ability levels.

What does parameter b represent in the three-parameter logistic IRT model?

Item difficulty.

What does parameter c represent in the three-parameter logistic IRT model?

The chance of guessing the correct response.

What is the three-parameter logistic IRT model?

A model that uses all three item parameters: discrimination (a), difficulty (b), and guessing (c).

What is the two-parameter IRT model?

A model in which the guessing parameter c is assumed to be zero.

What is the one-parameter IRT model, also called the Rasch model?

A model in which c is zero and discrimination (a) is assumed to be constant across items.

How does reliability differ between CTT and IRT?

CTT applies the concept of reliability to the entire test, whereas IRT assesses the information provided by each item.

How does the standard error of measurement differ between CTT and IRT?

CTT assumes the standard error is the same across ability levels, whereas IRT can estimate it at different ability levels.

What is an Item Information Function (IIF)?

A function indicating how much information an item contributes to a test's measurement precision.

What is computer-adaptive testing (CAT)?

Testing in which the test is tailored to the test taker's ability.

How does computer-adaptive testing work?

The test begins with an item of an estimated difficulty level. Correct or incorrect responses determine whether subsequent items become harder or easier.

When does a computer-adaptive test end, according to the lecture?

When the maximum number of items has been administered or a specified standard error of measurement has been achieved.

What are the advantages of computer-adaptive testing?

Reduced test length, less item leakage, and higher reliability.

What are the disadvantages of computer-adaptive testing?

It requires strong psychometric knowledge and many items.

What is differential item functioning (DIF)?

An item displays DIF when individuals from different groups with the same ability level have different probabilities of answering it correctly.

What does DIF indicate?

Potential measurement bias.

How does the Mantel–Haenszel technique examine DIF?

It compares item endorsement probabilities between groups for individuals who have the same score on the remaining test items.

How can IRT be used to examine DIF?

By visually inspecting differences between groups' item response functions and using more complex analyses.

What is configural (construct) measurement invariance?

A test measures the same construct in all groups, with the same number of factors.

What is metric (measurement) invariance?

A test measures the same construct across groups using the same response scale, with the same factor loadings.

What is scalar (score) invariance?

A test measures the same construct across groups using the same response scale and measurement units, with the same intercepts.

What did the lecture report about ChatGPT's performance in selection assessments?

GPT-crafted cover letters received more positive ratings than human-only letters; GPT outscored 94% of human test takers on a verbal reasoning test and completed a forced-choice personality test more optimally. Its superiority depended on the instrument and context.

What is an asynchronous video interview (AVI)?

An applicant responds to interview questions displayed on a screen, and the response and video are recorded for later evaluation by humans or algorithms.

What are the three forms of behaviour measured in AVIs?

Verbal: what someone says; paraverbal/audio: how someone says it, such as pitch and volume; nonverbal/visual: appearance, such as facial expression and eye contact.

What did the lecture report about using AVIs to measure general mental ability (GMA)?

GMA related strongly to verbal components but not to paraverbal or nonverbal components. Reliability and convergent validity were acceptable, but divergent validity was poor.

What did the lecture report about algorithmic personality assessment in AVIs?

Algorithmic assessments explained more variance in observer reports than in self-reports. Verbal cues were crucial, while paraverbal and nonverbal cues were not; reliability and validity were sometimes questionable.

What standards must automated assessments meet?

They need sufficient reliability and validity and an absence of measurement and predictive bias, regardless of whether humans or algorithms score them.

Why are training data important for algorithmic assessments?

Algorithms trained on poor-quality data may produce poor-quality assessments: garbage in, garbage out.

What is one concern about transparency in algorithmic assessment?

Transparency can be low, although human judgment can also lack transparency even if it feels more transparent.

What are potential advantages and concerns of AVIs?

they can be more resource-efficient, but criterion-related validity evidence is mostly lacking and negative applicant reactions are a concern.

What is gamification in personnel selection?

The integration of game elements into traditional non-gamified selection instruments.

What is the promise of gamification in personnel selection?

Improved applicant reactions.

What does the lecture say about evidence for gamification?

Findings are mixed, with both more positive and more negative applicant reactions.

What is game-based assessment?

An assessment method in which job candidates participate as players in a core gameplay loop while trait information is inferred.

What is the dilution effect in the context of games and selection?

Catchy or narrative predictors can distract decision makers from valid information and reduce its influence on decisions.

What should researchers and practitioners keep in mind when evaluating games and other selection instruments?

Games must meet psychometric standards. Avoid being influenced by shiny or mysterious predictors, explain assessments to improve applicant reactions, recognize that evidence in high-stakes settings is limited, and apply COTAN standards.