PSY 121 Quiz on Reliability and Variability

Operationalizing the variable

Defining and measuring a variable in a way that allows it to be observed or measured

Self-report

A mode of measurement in which participants report information about themselves

Observation/behavioral (other report)

A mode of measurement in which behavior is observed or reported by someone other than the participant

Physiological measurement

Measurement of a physiological symptom or bodily response

Simple observation

Observation in which nothing about the environment or situation is changed

Contrived observation

Observation in which the researcher changes something about the environment or does something to the participant

Naturalistic observation

Observation in which you record what is naturally going on

Systematic observation

Observation in which you have a system for noting or recording what participants are doing

Trace

A type of observation that examines what is left behind after a participant leaves

Qualitative

A type of measurement using open-ended responses

Quantitative

A type of measurement using fixed-choice responses or numerical information

Categorical (nominal)

Different categories in which the choices have no required order

Rank order (ordinal)

A measurement in which categories are ordered

Forced choice

A measurement in which a participant must choose between similar or similarly attractive choices

Interval

A measurement scale in which the intervals between values are assumed to be equally distant

Ratio

A measurement scale with an absolute zero that actually exists

Reliability

How consistent and dependable a measure is

Validity

How well a measure actually measures what it purports to measure

Face validity

Whether a measure appears to measure what it says it measures

Content validity

Whether a measure covers the full content of the construct it is intended to measure

Convergent validity

Whether a measure has similar scores to other measures of similar constructs

Discriminant validity

Whether a measure can distinguish between a construct and something similar but not exactly the same

Criterion validity

Whether a measure predicts or corresponds to the criterion being evaluated

Predictive validity

Whether a measure predicts a future criterion or behavior

Concurrent validity

Whether a measure corresponds to a criterion assessed at the same time

Construct validity

The overall validity of a measurement based on the relevant forms of validity

Measurement error

The difference or inaccuracy between a measured score and the hypothesized true score

True score

The hypothesized actual score that a measurement attempts to get as close to as possible

Context effect

An effect in which being reminded of other life circumstances can influence measurement results

Correlation

The relationship between variables

Correlation direction

Whether a relationship between variables is positive or negative

Correlation strength

The strength of a relationship, determined by the absolute value of the correlation

Alternate forms reliability

Reliability assessed by comparing scores from two different forms of a test

Test-retest reliability

Reliability assessed by giving the same test several times and comparing the scores

Test-retest alternate form reliability

Reliability assessed by using a different test form each time and comparing scores

Internal consistency (internal reliability)

Reliability assessed by comparing how consistently items or trials within the same measure relate to one another

Split-half reliability

Internal consistency assessed by dividing a measure in half and comparing the scores from the two halves

Item-to-item reliability (average intercorrelation)

Internal consistency assessed by examining how highly each item correlates with the other items

Cronbach's alpha (α)

A measure of internal reliability

Item-total r

The correlation of each item with the total score of all items

Interrater reliability

The consistency of ratings or coding between different raters

Example of self-report

Asking how many passes occurred between people in white or whether the participant saw the gorilla

Example of physiological measurement

Measuring saliva or blood pressure as physiological symptoms

Example of observational/behavioral measurement

A participant noticed whether the person asking for directions changed after two people moving a door separated them and changed the person

Example of observational/behavioral response

The response was whether the participant noticed or did not notice the change

Example of observational measurement with babies or young children

Observing babies or young children because they do not have the language level needed for self-report

Example of simple observation

Watching behavior without changing anything in the environment

Example of contrived observation

Using letters and doors to change something about the environment in which the participant is behaving

Example of contrived observation setup

Setting up a situation that the participant was going to encounter

Example of naturalistic observation

A child wants a lightsaber to bloom and is upset while another child wants to sword fight, and the behavior is recorded as it naturally occurs

Example of systematic observation

Using a system for noting or recording what is going on or what participants are doing

Example of sense of self in observation

Counting school clothes and noticing that when the football team won, people tended to wear school clothes more

Example of sense of self after a football win

Students sitting in the student center referred to the team as “WE” when the football team won

Example of sense of self after a football loss

Students referred to the team as “THEY” when the team lost

Example of trace observation

Looking at the amount of garbage left on seats after the team won or lost

Example of trace operationalization

Using the amount of garbage left behind as the operationalization of whether people were helpful or not

Example of trace measurement in archaeology

Studying remains left behind to learn about the past

Example of projective test

Asking ambiguous things so that deeper thoughts a person does not want others to know may be revealed

Example of IAT

Testing associations such as white and good or black and bad

Example of quantitative self-report

A participant gives a fixed-choice or numerical answer about themselves

Example of quantitative observational measurement

Researchers count or record observable behavior numerically

Example of quantitative physiological measurement

Researchers measure a physiological response such as blood pressure numerically

Example of quantitative physical trace measurement

Researchers count or measure something left behind by participants

Example of systematic quantitative football experiment

Counting whether people wore school gear after the football team won or lost makes the observation systematic and quantitative

Example of a quantitative yes response

“Yes, they are wearing school gear”

Example of a quantitative no response

“No, they aren’t wearing school gear”

Example of categorical (nominal) measurement

Choosing what you are going to be for Halloween

Example of categorical (nominal) question

“Are you happy or sad?”

Example of rank order measurement

Ranking the emotions happy, calm, and sad in order of your current mood

Example of forced choice

Choosing between a vacation to Hawaii and a vacation to Lake Michigan when both choices are similarly attractive

Example of forced choice game

A Would You Rather game where you have to pick one option

Example of interval measurement

Rating how happy you are on a scale from 1 to 5

Example of interval scale

Assuming that 1 is equally distant from 2 and 2 is equally distant from 3

Example of ratio measurement

Counting how many people there are, where zero people is a meaningful absolute zero

Example of quantitative nominal question

“Are you ready or not ready for fall break?”

Example of rank order question

“Rank colors from favorite to least favorite”

Example of forced choice question

“Would you rather be rich and sad or poor and happy?”

Example of interval question

“How many hours did you sleep?”

Example of ratio question

“How many pets do you have?”

Example of qualitative question

“Why did you choose TCNJ over other schools?”

Example of ratio measurement with a baby

Measuring how long a baby played with a toy, how many times the baby tossed it to a parent, or how many times the baby pressed a button

Example of unreliable or invalid thermometer

A thermometer always measures the temperature as 60 degrees outside even though the actual temperature is 62 degrees

Example of reliability

A friend who is stranded at the airport at 2 AM needs a friend who will reliably come get them

Example of validity

The friend must not get lost along the way and must actually reach the airport to pick them up

Example of face validity

A measure says it is a measure of happiness and the items appear to be about happiness when you look at them

Example of content validity

If happiness is the construct, compare the items on the measure with the definition of happiness

Example of insufficient content validity

A measure of happiness that does not encompass all aspects of the construct does not have content validity

Example of content validity error

A test says it covers chapters 1 and 2 but instead includes material from chapter 3

Example of happiness construct operationalization

Defining happiness using the prevalence of positive emotions and cognitive evaluation of life

Example of convergent validity

A score on a satisfaction-with-life measure is similar to a score on the Oxford Happiness Index

Example of discriminant validity

Distinguishing satisfaction with life from something in the same general ballpark but not exactly the same

Example of discriminant validity with languages

Using Spanish and Italian to illustrate two similar Romance languages that can still be distinguished

Example of criterion validity

Testing whether a measure of happiness corresponds to the number of times somebody smiles

Example of predictive validity

Giving a person a measure of happiness now and asking whether it predicts whether they will smile a lot three months from now

Example of predictive validity with studying abroad

Testing whether doing well on a language test predicts whether someone will be able to converse well when studying abroad in Madrid

Example of concurrent validity

Giving a measure of happiness now and assessing how many times the person smiles now

Example of concurrent validity with Spanish

Testing whether a high score predicts whether someone can go to a restaurant for lunch and order in Spanish well

Example of a straightforward physical construct

Height can be measured even if the measurement scale is slightly off, because you can still distinguish tall from short

Example of a psychological construct

Self-esteem cannot be visually observed directly