PSYC3020 L10 Extreme Psychometrics

Recognition memory tasks involve people making discrimination judgements.

True

Signal Detection Theory was originally developed for use in recognition memory tasks.

False

Imagine you are given a test of hearing, where you are played a series of beeps of different volumes over headphones. Some of these beeps are too quiet to hear: the test is designed to find the quietest volume you can hear. Your task is to indicate when you can hear a tone. If you want to try and pretend your hearing is better than it might be and you know that the test is NOT employing Signal Detection Theory, then you should adopt a liberal criterion for responding.

True

Someone who has a conservative response bias in a recognition memory task will be less likely to remember the task items than someone who has a liberal response bias.

False

Signal Detection Theory allows you to bypass the usual statistical analysis of data by calculating measures such as d' (d prime).

False. You still have to do the usual stats

When scoring memory accuracy in a recognition memory task, we need to measure both the correct hit rate and the false positive rate.

True

To make a diagnostic test provide a greater number of correct positive diagnoses of a disease, we could use a more liberal response bias.

True

The implications of positive results from diagnostic tests are often misinterpreted because people do not tend to take into account the pre-test probability (i.e., base rate).

True

The specificity of a diagnostic test is the proportion of all people with a negative test result who are correctly indicated as not having the disorder under investigation.

False. All people who don't have disease who are correctly identified as not having it.

The sensitivity of a diagnostic test is the proportion of all people with the disorder who correctly obtain a positive test result.

True

In Item Response Theory (IRT), Theta is a measure of a test taker's overall level of ability.

True

When using Item Response Theory (IRT), it is possible that two people could get the same number of questions correct in the same aptitude test, but nonetheless end up with completely different test scores.

True

Item characteristic curves are a plot of ability versus the item discrimination index.

False. Item difficulty index

In Item Response Theory, Theta is a measure of a test taker's overall level of ability (assuming they're completing some sort of ability test).

True

Applying Classical Test Theory generally involves assuming linear relationships between different variables to evaluate reliability and validity.

True

A key advantage of Item Response Theory is that it uses equations for curves rather than assuming a linear relationship.

True

When applying Item Response Theory, "goodness of fit" tests are required to calculate test scores.

False. Used for estimating best item characteristic curve

In the three-parameter model of an item characteristic curve, level of guessing is represented by the area under the curve.

False. Y-intercept

Item Response Theory (IRT) involves modelling test data by analysing the linear parameters of item characteristic curves.

False. Does not have to be linear

When using Item Response Theory, tests are analysed at the aggregate level, rather than at the level of individual items.

False

Under what circumstances would we need to use Signal Detection Theory?

If you need to discriminate between stimuli

In a recognition memory task, what are the four possible outcomes?

1. Correct hit

2. Correct miss

3. False positive

4. False negative

If we did not use signal detection theory, how could someone cheat a recognition memory test?

"I recognise them all!!!"

What are sensitivity and response bias, in the context of signal detection theory?

- Sensitivity: a person's ability to discriminate between signal and noise

- Response bias: how willing a person is to report the signal (liberal or conservative)

How does signal detection theory "un-confound" sensitivity and response bias?

- We can now measure both the correct hits and false positives so that we can work out a person's sensitivity and response bias

What software do you need to calculate signal detection theory variables?

- Can use Excel + SPSS

The probability a woman (aged 40-50, in a region with routine screening) having breast cancer is 0.8%. If a woman has breast cancer, the probability is 90% that she will have a positive mammogram. If a woman does not have breast cancer, the probability is 7% that she will still have a positive mammogram. What is the probability that she does have breast cancer if she has a positive mammogram?

9%

What are sensitivity, specificity, and pre-test probability in the context of the 2x2 contingency table analysis?

Sensitivity: % of correct hits

Specificity: % of correct misses

A man is given a test for schizophrenia. 1% of men in his age group have schizophrenia. If he has schizophrenia, then the test will give him a positive result 85% of the time. If he does not have schizophrenia, then the test will give him a negative result 90% of the time. What is the percentage chance the man has schizophrenia if he gets a positive test result?

8%

How could you design a diagnostic test that would correctly diagnose every person who had a disease as having the disease?

Diagnose everyone as having the disease (extreme liberal)

Why might you want a test to have a liberal response bias?

When false negatives are costly e.g. in a hospital setting

If a test for a disease yields a high proportion of correct diagnoses, why might this not necessarily mean it's any good at diagnosing the disease?

It may also have a high number of false positives

Why might the false positives yielded by disease screening tests be a bad thing?

- It could lead to a person worrying, having necessary surgery or even suicide

What did Gigerenzer (1998) find that AIDS counsellors believed regarding the predictive certainty of a positive HIV test?

Close to 100%

The probability of a person with no known risk behaviour having HIV is 0.01%. If a person has HIV, the probability is 99.98% that they will test positive. If a person does not have HIV, the probability is 99.99% that they will correctly test negative. What is the probability that a person does have HIV if they receive a positive test?

50%

Why is it important that health professionals understand the calculations described in this presentation?

- So that they can give patients the full picture so that they don't excessively worry or worse e.g. suicide

What does Item Response Theory involve?

Having items that measures an individual's ability (theta). Theta will vary from person to person

What is Latent Trait Theory?

Item response theory

What are Item Characteristic Curves and what do they have to do with Item Difficulty Indices?

A curve for each item that measures how probability of getting an item right (item difficulty index) based on the ability (theta)

What is the link between Item Response Theory and Item Characteristic Curves?

IRT is the theory that uses ICCs to measure ability, which varies by item difficulty

Detail the steps you would go through to find the best equation to represent your item characteristic curve.

1. Choose a curve based on best guess

2. Use software to adjust the parameters of the curve so that it fits the data

3. Check for "goodness of fit" to see how well the curve fits the data statistically

4. If poor fit, then repeat with another curve

What is a logistic function?

A non-linear curve used as a starting point for choosing a ICC for an item

What are the parameters in the three parameter model used in Item Response Theory to estimate the probability of getting a particular question correct for a certain level of ability?

- "a": Discrimination (steepest point of curve)

- "b": Difficulty (point where people's ability gets the item right 50% of the time)

- "c": Guessing (y-intercept)

Once we have modelled an item characteristic curve then what can we use it to do?

Measure's people's individual ability by getting them to respond to the item

Give three advantages of item response theory over traditional psychometric methods.

1. Shorter test

2. Can use non-linear curves

3. Can easily compare the ability of 2 people even if they have not completed the same questions.

Give 3 disadvantages of item response theory compared with traditional psychometric methods.

1. Difficult to understand and implement

2. Requires large samples to estimate ICCs

3. Requires more assumptions than CTT