PS Exam 2

What is a psychological test?

A standardized series of problems or questions that assess a particular individual characteristic

What are the three critical components of a psych test?

Perform a behavior

Make an inference

Measuring a psychological Construct

What is behavior?

observable actions/reactions

What is a psychological construct?

A theoretical concept that can not be directly observed

What is an inference?

A conclusion/assumption made based on evidence

Inferences should only be about the _______

construct

In selection, inferences drive ___________

decisions

What are the 3 characteristics of a good psych test?

Representatively sample behaviors thought to measure a KSAO

Have standardized conditions

Have clear rules for scoring

Why should a test include multiple items?

increase reliability

What are the four different types/characteristics of tests?

Group vs Individually Administered

Closed-Ended vs Open-Ended

Paper and Pencil

Power vs Speed

What are the 4 assumptions of psych tests?

Individuals' scores on ability and personality tests will remain stable over time

Individuals understand items the same way

Individuals will report accurately about themselves

Individuals will report honestly about themselves

What is an ability?

Person's capacity to do/learn a particular task

What are CATs?

Tests that measure mental reasoning, intelligence, etc

What is Personnel Tests for Industry?

A test designed to assess mathematical and verbal ability

What does research show about the validity of cognitive ability tests?

They are valid predictors of job performance across a large number of different kinds of jobs

How do I/O psychologists generally feel about the use of cognitive ability tests?

Fair and valid

What are psychomotor ability tests?

Tests that assess things as ability to manipulate objects and use tools

What is the Pegboard Dexterity Test?

A test that assess the ability to use simple tools to manipulate small objects

What is a knowledge/skill test?

Achievement test, designed to assess a person's present level of proficiency

What is personality?

Personality is a predisposition/tendency to behave in a particular way across different situations

What is the International Personality Item Pool?

A personality test that provides more than 2,000 personality items can be used to assess a wide variety of traits

What are some major concerns with the use of personality tests for employee selection?

Faking, lack of face validity

Faking and personality testing article (Hogan et al., 2007)

Purpose

Hypotheses

Method

Results

Conclusions

What is the purpose of the article?

Determine whether faking is an actual threat in assessment

What are the hypotheses in the article?

Scores won't improve on retesting and scores would increase as often as they decreased

What is the method of the article?

Repeated measures design

What are the results of the article?

Scores did not improve in a statistically significant manner and increased as often as they decreased

Applicants with high social desirability scores improved most on Emotional Stability and Conscientiousness

Applicants with lower integrity scores are more likely to change their scores on Emotional Stability, Conscientiousness, and Openness.

What are the conclusions of the article?

The findings support the Impression Management Theory and champion the use of personality tests in assessments.

Can people fake their answers on personality tests to increase their desirability to companies (i.e., put themselves in a good light)? Provide research evidence.

People can fake when instructed to do so

Applicants fake more than general population

Faking is especially likely when socially desirable response is easily discerned

Faking does not markedly affect the criterion related validities of personality scales

How does faking impact criterion-related validity (i.e., extent to which personality tests predict job performance)? Why?

It does not

If everyone fakes, everyone gets a score bump, so rank order of people is not affected

Those who can fake exhibit self presentation skills. (what most employers may want)

What is emotional intelligence?

EI is people's ability to control and recognize emotions in themselves and in others.

What are the two types of emotional intelligence tests?

Trait- measures EI as a personality characteristic that is accessible to the person's awareness

Ability- MC format with one right answer per question, based on the assumption that people can not directly report their EI because it is not accessible to awareness

How do scores on emotional intelligence tests relate to job performance?

Small-moderate correlation with job performance

What is a concern with emotional intelligence tests?

Construct validity, possible overlap with measures of cognitive ability and personality

What is an integrity test?

A test designed to predict whether an employee will engage in counterproductive or dishonest behavior on the job

What are the two types of integrity tests?

Overt- assesses a person's attitudes and prior behavior; obvious assessment of integrity/honesty

Personality- assesses personality characteristics that have been found to predict overall counterproductive behavior; hidden in their purpose

How do scores on integrity tests relate to job performance?

Integrity tests can predict general CWBs and job performance, but there is debate on how well they do so

What is a vocational interest test?

A test that matches either the interests/personality of the test taker to those in a variety of different occupations and occupational categories

What is one of the most popular vocational interest tests?

Self Directed Search

Why do some employers use credit checking for selection?

Widespread beliefs that poor credit history is an indicator of a lack of conscientiousness and integrity

What are the major problems with credit checking for selection?

Little evidence for the validity of credit information in predicting workplace behavior

In the US, ethnic minorities tend to score worse on credit checks

Applicants tend to view the use of credit checking as unfair

When may drug testing be appropriate?

Jobs that are safety sensitive where impaired performance could lead to accidents or injury

What are 4 ways of acquiring psych tests?

Develop entirely in house

Hire a consultant

Purchase off the shelf assessments

Contract with an online assessment company

What existing psychological tests can you use? Which are freely available?

General Aptitude Test Battery

International Personality Item Pool (free)

Interest Item Pool (free)

When might you need to make your own psychological test?

When you have a highly niche situation or do not have the financial resources to pay for another option

What are the basic steps for developing a psychological test?

Define the specific KSAO of interest

Write items intended to reflect the KSAO of interest

Administer the items to a sample

Conduct a statistical analysis of the responses to pick the best ones

Administer the new assessment to a sample of employees with a measure of job performance

Conduct a statistical analysis to show that the assessment can predict job performance

What factors should you consider when deciding how to choose assessments?

How many people you will hire per year

If you have in house expertise

If off the shelf assessments are available

What is biographical information (biodata)? What are some examples?

Information that would be provided through a CV or resume. Ex. Name, address, work experience

What is a biographical (i.e., biodata) inventory? What are some examples?

Asks about specific experiences at school or work (or even other areas of life)

What was your GPA in college? Did you enjoy college?

What are the two types of biographical inventories?

Empirical- developed by administering many potential items to a group of employees in a particular job (ask a bunch of info and let computers tell you what is most important and how to score it)

Rational-developed by beginning with an analysis of KSAO requirements and then devising items that reflect those KSAOs (info that is job relevant, scored based on perceived importance)

What are the pros and cons of empirical biographical inventories?

Pros: specifically designed to maximize the prediction for performance

Cons: not always clearly job relevant

What are the pros and cons of rational biographical inventories?

Pros: job relevance

Cons: time consuming, lower validity than empirical inventories

How do biographical inventories relate to job performance?

They have been shown to predict job performance

Biographical inventory article (Mount et al., 2000)

Type of biographical inventory examined

Purpose

Hypotheses

Method

Results

Conclusions

What type of biographical inventory did this article examine?

Quantity/ quality of work

Problem solving

Interpersonal facilitation

Retention probability

What is the purpose of the article?

Aimed to address whether the extent of overlap among the personality, GMA, and biodata domains is so great that there is no incremental gain in using biodata with the other two predictors

What are the hypotheses in the article?

Empirically keyed biodata scales will account for incremental validity in predicting job performance outcomes beyond what is captured by the FFM and GMA

What is the method of the article?

Concurrent validation

What are the results of the article?

Biodata scales accounted for variance in ther performance criteria beyond that accounted for by FFM and GMA

What are the conclusions of the article?

Aimed to address whether the extent of overlap among the personality, GMA, and biodata domains is so great that there is no incremental gain in using biodata with the other two predictors

Biodata scales tap into aspects of the criteria not covered by internal measures

Use of these 3 methods is better than what we have now

What are interviews designed to assess?

KSAOs

How popular are interviews, and how are they perceived?

Most used selection device

What are the two primary types of interviews? How about the third? Define them

Unstructured: the interviewer asks whatever questions come to mind

Structured: the interviewer has a preplanned series of questions that are asked of every person who is interviewed

Semi-structured: pre-planned questions with freecom to deviate/ask followups

Interview questions can ask about what 4 things?

Attitudes, job experiences, personal background, preferences

What are 4 types of interview questions?

Situational, past behavior, background, job knowledge

What are some advantages of structured interviews?

Better assessing of job relevant KSAOs

Easier to compare candidates when everyone is asked the same questions

Reduces bias

Higher criterion related validity

What are 14 ways to add structure to an interview?

Base questions on a job analysis

Ask the exact same questions of all candidates

Limit prompting

Use better types of questions

Use longer interviews/more questions

Control ancillary information

Do not allow candidate questions until the end

Rate each answer

Use detailed BARS

Take detailed notes

Use multiple interviewers

Use the same interviewers across candidates

Do not discuss candidates/answers between interviews

Provide interviewing training

Structured interview article (Campion et al., 1997)

Purpose

Hypotheses

Method

Results/Conclusions

What is the purpose of the article?

Review the research literature in order to describe and evaluate the many ways interviews can be structured

What are the hypotheses in the article, if applicable?

NA

What is the method of the article?

Literature review, reviewed 15 components of structure and evaluated them based on reliability, validity, and user reactions

What are the results and conclusions of the article?

It is a good idea to:

Base questions on a job analysis

Ask the exact same questions of all candidates

Limit prompting

Use better types of questions

Use longer interviews/more questions

Control ancillary information

Do not allow candidate questions until the end

Rate each answer

Use detailed BARS

Take detailed notes

Use multiple interviewers

Use the same interviewers across candidates

Do not discuss candidates/answers between interviews

Provide interviewing training

What is a behaviorally anchored rating scale (BARS)? How should scale points be written?

A scale that assesses the performance of an interview based on well-defined behavioral patterns

They should align on content but at different levels

What is the purpose of BARS?

Improve accuracy of interview ratings

What three types of information affect interviewers' judgements?

Job related content that is revealed about the interviewee

The interviewee's performance

Apparent personal attributes of the interviewee

What are some biases that may affect interviewers' judgements?

Attraction, race, gender

How does the validity of structured interviews compare with the validity of unstructured interviews?

The validity of structured interviews is significantly higher than unstructured interviews

What did Schmidt and Hunter report as the validity of interviews?

.51 (structured interviews) vs .38 (unstructured interviews)

What are some key pros and cons of structured interviews?

Pros: clearly job relevant when questions are based on job analysis, good predictor of performance, perceived as fair by applicants

Cons: time consuming, interviewees may desire to ask follow-up questions/be more conversational, interviewer biases may come into play (mitigated by BARS and rater training)

What is a work sample?

An assessment device that requires a person to demonstrate how well they can perform the tasks involved in a job under standardized conditions

What are some examples of work samples?

Medical simulation, computer science project, draft an email, physical driving test, teach a class

What are the key characteristics of a good work sample?

Clear, detailed instructions/procedure for applicants

Explain how they will be evaluated

BARS for rating

Work sample article (Robertson & Kandola, 1982)

Purpose

Hypotheses

Methods

Results/Conclusions

What is the purpose of the article?

Review the research literature on work sample tests with a focus on the validity, adverse impact, and applicant reactions

What are the hypotheses in the article, if applicable?

NA

What is the method of the article?

literature review, reviewed literature on 4 types of sample tests (psychomotos, individual decision making, job related info, group discussions/decision making)

What are the results and conclusions of the article?

Work samples can have high criterion-related validity, reduce adverse impact (racial disparities), and produce favorable applicant reactions

What is the criterion-related validity of work samples? (slides)

.33 (most recent)

Why may the criterion-related validity of work samples be difficult to measure and inflated?

If the predictor and criterion are measured the same way, it is not surprising that work samples relate to job performance. You have to capture the facets

What are some key pros and cons of work samples?

Pros: clearly job relevant, good predictor of performance, may have fewer racial difference than GCA tests, perceived as fair

Cons: time consuming, if unpaid this is unethical, complex jobs have many tasks that may not be completely demonstrated

What is an assessment center?

A series of simulation exercises to measure some job relevant abilities/skills that a person has

Must include:

Multiple trained assessors

Multiple assessment components

A process for observing, recording, classifying, and integrating behaviors based on defined behavioral constructs

What type of position/skills are most assessment centers designed to assess?

Leadership/management

How long do assessment centers often take to complete?

several days

What is an assessment center exercise?

Simulations of actual tasks included on the job

What is an AC dimension?

Areas scored in an AC

What is an in-basket exercise?

A simulation exercise that asks the assessee to pretend that it is the first day of a new job and they have found a series of items in their basket and they must deal with each item in an appropriate manner

How could you evaluate someone's performance on an in-basket exercise? What dimensions might be relevant?

oral communication/comprehensions, problem solving, interpersonal relations, coaching, planning, written communication

What is a leaderless group discussion?

Several assessees are given a problem to solve together or generate a solution to an organizational problem

What are two types of leaderless group discussions?

Problem solving and Role play

How could you evaluate someone's performance in a leaderless group discussion? What dimensions might be relevant?

BARS- communication, problem solving, influencing others, consideration/awareness of others

What is a problem-solving simulation?

Assessee is given a problem and is asked to come up with a solution

What is a role play exercise?

Requires the assessee to pretend to be a particular person in a specific organizational role, tasked with handling a problem or situation

How do assessors rate assessees?

On each dimension by observing behavior and reviewing materials produced during the exercises

What is the relationship between assessment center performance and job performance? Provide correlations from existing research.

Scores in an assessment center given to students upon admission predicted their job performance as teachers from 2-5 years after graduation with a correlation of .27

Police officer correlations range from .30 to .21

Explain concerns that have been raised regarding the construct validity of assessment centers

We are not certain that the intended dimensions of the assessment measures are being measured

Provide explanations for why assessment centers may have low construct validity.

The various dimension scores given to the assessees within an exercise are too highly correlated

Assessment Center Guidelines article (Rupp et al., 2015)

Purpose

Hypotheses

Method

Results/Conclusion

What is the purpose of the article?

Provide professional guidelines and ethical considerations for users of the assessment center method

What are the hypotheses in the article, if applicable?

NA

What is the method of the article?

Delegates gathered and drafted this edition

What are the results and conclusions of the article?

ACs should involve the following:

Systematic Analysis to determine job relevant behavioral constructs

Behavioral classification of observed behaviors

Multiple components

Linkages between constructs and assessment components

Simulation exercises requiring the assessee to overtly display behaviors

Multiple assessors

Assessor training

Systematic recording and scoring

Data integration

Standardization

Adherence with Assessment Center Guidelines article (Eurich et al., 2009)

Purpose

Hypotheses

Method

Results/Conclusion

What is the purpose of the article?

Review current AC practices in the US by evaluating whether they follow the guidelines and ethical considerations

What are the hypotheses in the article, if applicable?

NA

What is the method of the article?

Data was collected via an online survey completed by individuals from 54 HR departments of Fortune 500 organizations across the US

What are the results and conclusions of the article?

Companies are doing a lot of things well. Many are aware of the guidelines, conduct job analyses, train assessors, and use 4-5 exercises per AC. Yet, improvements can be made by assessing less dimensions, testing exercises before implementing, and providing feedback to assessees.

Police Candidate Assessment Center article (Dayan et al., 2002)

Purpose

Hypotheses

Method

Results/Conclusion

What is the purpose of the article?

To examine the validity of the assessment center as a selection process for police and its unique value beyond cognitive ability tests

What are the hypotheses in the article, if applicable?

Hypothesized that the overall scores provided by the AC assessors and the AC peer evals will offer additional predictive validity beyond tests of cognitive ability, and peer evaluation measures gathered in the AC will provide incremental validity over AC simulation scores

What is the method of the article?

414 entry level Israeli police candidates participated in an AC. Data was collected on their AC performance and job performance

What are the results and conclusions of the article?

AC performance predicts job performance for police. Peer evals were predictive of job performance beyond assessor ratings and cognitive ability tests. ACs are useful, valid predictors, and peer ratings are valuable

What are some key pros and cons of assessment centers?

Pros: Clearly job relevant, good predictor of job performance, perceived as fair

Cons: time consuming and expensive, best suited for leadership positions

In what ways is an assessment center like a work sample?

Simulation

Job relevant

Performance evaluated using BARS

Structure is good

In what ways is an assessment center different from a work sample?

ACs involve multiple exercises, WS involves one task

ACs are evaluated live, WS is often completed out of office

ACs are often used for leadership, WS has a wider application

ACs are costly (time and money)