PSYC 406: Lecture 14

Standardization, evidence, informal, practitioners, traitions, exner, structured, scoring, reference, R PAS, performance, administration, evidence, consistent, evidence,

The Rorschach scoring systems evolved

• ___ and ___ continue to change

1. Earlier approaches

• ___ interpretation varied among ___ and ___

2. ___’s comprehensive system

• introduced ___ procedures, ___ variables, and ____ data

3. ___-___: later standardized ___ based framework emphasizing ___, norms and ___

Standardized scoring make Rorschach results more ___, but we still need scientific ___ that the scores are reliable and valid for their intended use

Counts, ratios, derived, thinking, depression, coping, risk pattern, vigilance, processing, observations, diagnosis

Inside a Rorschach scoring summary

• a fictional example shows how responses are tallied and combined

• ___, ___ and ___ scores across 10 cards.

Composite indices

1. PTI: ___

2. DEPI: ___

3. CDI: ___

4. S-CON: ___ ___

5. HVI: ___

6. OBS: ___

The summary organized coded ___; it does not by itself establish a ___.

Unusual, norm group, representativeness, false positives, norms, scrutiny, interpretation

Norms can change clinical conclusions

• a response is ___ only relative to an appropriate reference

1. The ___ ___: whose responses were used to define the typical distribution ?

2. Historical concern: some earlier reference samples prompted concerns about ____ and ___ ___

3. Contemporary issues

• updated ___ help, but population, langage and context of test administration still require ___

A dated reference group can distort ___ without changing the response.

Interrater, test retest, internal consistency, every, generalizability, reliability, exner, reliability, separately

Different reliability questions matter

• the right statistic depends on what is being reproduced

1. ___: do trained scorers assign the same codes to the same response ? Example: both code movement

2. ___ ___: does a score stay similar when a person is retested and the trait is ___? Example: similar scores two weeks later. A little bit less established

3. ___ ___: if a score combines several responses, do they measure the same characteristics? Not relevant for ___ Rorschach score

4. ___: how muc of the score reflects the person rather than the card, rater or occasion? Example: would a different rater change it ?

• overarching ___ model

The ___ comprehensive system needs to be used to produce reliable results.

There is no single Rorschach ___ result: evaluate each score ___

• lengthy training to learn how to code responses reliability

Differences, variables, score, criterion, use, score, what, evidence, use

Is the Rorschach valid.

• a global yes or no judgement hides major ___ among ___

1. ___-specific: which coded variable is under evaluation?

2. ___-specific: ones it predict an external outcome or self report rating? Those can differ.

3. ___-specific: is the evidence sufficient for screening, formulation, or high stakes decision?

To evaluate the Rorschach’s validity, ask: which ___? ___ does it tell us? What ___ supports using it for a particular ___?

Mihura, 65, .27, .08, 13, 17, perception, thinking, perceptual thinking index, synthesized, 10, 13, 12, egocentricity, isolation,

Which Rorschach scores have validity evidence?

• ___et al. Examined ___ comprehensive system variables. Meta analysis of meta analysis

1. Across studies

• average validity correlations:

• Observer ratings/diagnoses: r = ___ (r between indices)

• Self report criteria: r = ___ (r between scores and self report personality measures)

Stronger support

• ___ variables: excellent support

• ___ variables: good support

• Best supported areas included ___ and ___; for instance, the ___ ___ ___ and the ___ Response.

Limited or missing

• ___ variables: modest support, ___ variables: little or none, ___: no relevant validity studies.

• Weakly supported examples: ___ and ___ indices

Ask: which Rorschach score, for which outcome and with what evidence ?

Standardized, reproducible, variables, meanings, utility, high stakes, system, criterion, population, level

Rorschach: a balanced conclusion

• scientific support is neither uniform nor nonexistent

What can work:

• ___ administration and ___ coding are possible

• Some ___ have useful evidence

What remains limited

• other proposed ___, norms, ___ and ___-___ applications need caution

• Research supports some variables over others

Responsible use

• specificy ___, variable, ___, ___ and intended conclusion. Application

Evaluate claims at the ___ at which their evidence exists.

Supported, ambiguity, unconscious, variable, scoring, criteria, uses, comparative, validity, reliability, feasibility, value, evidence

3 takeaways about projective methods

• focus on what can be ___, not what sounds psychologically revealing

1. ___ is not evidence: different responses do not automatically reveal ___ personality

2. Support is ___-specific: evidence differs across ___ systems, variables, ___ and ___.

3. Assessment is ___: choose tools based on ___, ___, fairness, ___ and incremental ___.

• is it worth it to employ this labour intensive test to get the info were searching for ?

Strong claims about psychological assessment require strong ___

Ok

CHECK THE MISSING SLIDES AND THE RECORDING !!!!!!!!

Volume, complexity, volume, variety, velocity, personality

What counts as “big data ?”

• big data refers to ___ and ___: large volume of data

1. ___: many observations. Example: hundreds of posts per person

2. ___ : different formats. Text, location, activity, or interaction records.

3. ___: repeated or continuous collection: phone usage everyday

Having more data does not necessarily mean having better information about ___

Social media, text, langage, smartphones, wearables, web, audio, video, patterns, personality

Where do digital footprints come from?

• each source reveals some behaviour and misses other behaviour

1. ___ ___: likes, networks, shared content

2. ___ and ___: posts, essays, messages

3. ___: mobility, calls, screen activity

4. ___: activity, sleep, physiological signals

5. ___ behaviour: search and interaction patterns

6. ___/____: speech, movement, expression

Phones, wearables, and online activity can reveal ___ of behaviour, but not automatically someone’s ___

Trait, behavioural, observed, behavioural, explanation, psychological, context, what

A digital trace is not the ___ itself

• before measuring personality, ask what each ___ clue can represent

1. ___ trace: revel posts after midnight in one week. This is a ___ description

2. Proposed ___: this person is high in N. That is a ___ inference

3. Other explanations: night shifts, exams, caregiving, travel, platform habits. ___ matters

Observing a behavior is not the same as knowing ___ it tells us about someone’s personality

Response, measurement, statements, algorithm, behaviour, interaction, observed,

From questionnaires to behavioural traces

• the ___ process changes; the ___ questions do not

• Ask the model to examine what the person does

Traditional self report

• respondent evaluates ___

• Example: i enjoy meeting new ppl

Computational approach

• ___ estimates traits from ___

• Example: ___ patterns in messages

Common challenge

• does the ___ response actually represent E, and for whom ?

Questionnaires and digital behavior provide different info about a person; we cannot assume they measure the same thing.

traits, behavior, traits, behavior, behaviour, personality, why

A new direction of prediction

• traditional research relates ___ to ___; ML (machine learning) often predicts ___ from ___

• Here, ___predicts ___ → predict what they will do on social media

Predicting personality from someone’s behavior does not explain ___ they behave that way

Data, flexible, essays, conversation, terrible, language, facebook, likes, small, posts, .29, new, predictions

How did personality computing develop?

• the same research question evolved with better ___ and more ___ models

• methods have evolved, but they are still young

1. 2000s-2010

• mairesse et al (2007): BFT cues in ___ and ___

• Yarkoni (2010): 694 blogs; ___ linked to N. ___ used can predict personality.

2. 2010s

• goldbeck et al. (2011): ___ profiles predicted BFT scores

• Kosinski et al. (2013): 58,000+ users’ ___ predicted traits. ___ behaviours predict personality above chance levels

3. 2020s

• peter and Matz (2024): LLMS estimated BFT scores from facebook ___ (mean r = ___). ___ data: photos, voice, phone sensors

As AI methods became more sophisticated, researchers still needed to establish whether their personality ___ were accurate

Likes, patterns, questionnaire, predictive, trait, data, psychological, traits, outcomes

Kosinski et al.: facebook ___ offered early evidence

• digital choices predicted some traits better than others

The early finding

• ___ of facebook likes predicted ___ scores and other attributes

• The graph shows different correlations for different outcome s

• ___ performance depended on the ___ and available ___.

• For some personality characteristics, likes really predicted well.

Patterns of facebook likes can predict some ___ characteristics, but accuracy differs across ___ and ___

RECORDING QUICKLY HERE

Examples, new, collect, train, test, evaluate, errors, biases,

How is a personality model built ?

• the algorithm leans from ___ and must be checked on ___ people

1. ___: gather digital behavior and appropriate reference scores

2. ___: learn associations between extracted features and the target.

3. ___: predict scores for ppl not used during training.

4. ___: check accuracy, reliability, fairness and usefulness

AI can learn the same ___ and ___ found in the questionnaire scores used to train it

Prediction, cross validation, overfitting, leakage,

Models must work on new people

• training performance can look impressive even when ___ fails in a new sample

• need ___ ____ among samples

The danger:

• ___: learning chance quirks of the training sample instead of s a stable pattern

• ___: information from the test set influences training

A model that works well on its training data may perform poorly when predicting personality for new people.

Explanation, association, construct, new, behavioural, trait, posting, explaining, trait

Prediction is not ___

• a useful clue does ont always reveal and psychological mechanism

• Maps the ___ between behaviour a personality but does not explain why. There is no ___ explanation. → good at exploiting association s but not at explaining them

Prediction

• late night posting helps estimate a questionnaire sore. Does the pattern hold for ___ users?

Explanation

• why does this person post at night? Sleep, work schedule, age, and context should matter

Psychometric question

• does this ___ indicator actually reflect the personality ___ we want to measure, or could it reflect something else?

A ___ habit may help predict personality without ___ the behavior or accurately measuring the ___

Technology, word dictionaries, supervised, LLM, zero shot, infer predictions, predictions, examples

3 ways algorithms analyze language

• the ___ differs; every approach still needs validation

1. ___ __: count language categories (e.g.: emotion words)

2. ___ models: learn fro textsparied with personality ratings

3. ___ / ___-___: prompt a pretrained language model to ___ traits

• has not seen what you are asking: new task. Not seen similar tasks before. Learn by itself

Zero shot means asking an existing AI model to make ___ without giving it ___ of previously scored cases.

Status, questionnaire, zero shot, 1000, 200, 20, 3, averaged, 0.31, .27, moderate, high

Can GPT models estimate BFT ?

• peter and Matz: facebook ___ updates versus ___ scores

• Used ___ ___ GPT ratings.

Research design and findings

• sample: ___ adults, 100 item BFT questionnaire and ___ Facebook posts each

• Method: GPT 3.5 and GPT 4 rated ___post batches ___ times each; ratings were ___

• Result: mean agreement with self report: GPT 4 r = ___, GPT 3.5 r = ___.

• Not too low. Differences among demographic groups and some traits vs others

GPT 4’s personality estimates showed ___ agreement with questionnaires, not enough to describe an individual with ___ confidence

Assistant, big five, personality, content, 1, 5, 20, 5, 3, average, 10, average,

How did the researchers prompt GPT-4 ?

System prompt: you are a helpful ___

Personality rating instruction:

• rated te text on the ___ ___ personality dimensions

• Pay attention to how people’s ___ might be reflected in the ___ they post online. Return ___-___ ratings for O, C, E, A, and N, numbers only

• Didn’t give the model self reported scores

From posts to scores

• ___ facebook posts: ___ ratings

• Repeat ___ times then ___ the ratings

• Repeat across ___ batches; then ___ to obtain each person’s scores.

Getting five personality ratings from AI does not mean those ratings accurately describe the person