Chapter 3: Evaluating Research and Assessment

Psychometrics

the technology of psychological measurement

Two questions of Psychometrics

1. Are the data reliable?

2. Are the data valid?

Combined question of Psychometrics

Are the data generalizable?

Psychometric Reliability

in measurement, the tendency of an instrument to provide the same comparative information on repeated occasions

Measurement Error (Error Variance)

the difference between the observed score and the true score, which can affect reliability and validity

State

a temporary psychological event, such as emotion, thought, or perception

Trait

a relatively stable and long-lasting attribute of personality

Why is trait harder to learn than state?

because traits are more stable and consistent over time, which require more time to observe and measure

Four factors that undermine reliability

Low precision, poor participant state, poor experimenter state, and minor environmental variations

Techniques to improve reliability

Care with research procedure, standardized research protocol, measure something important rather than trivial, aggregation

Aggregation

the process of combining multiple measurements and observations to increase reliability and accuracy

Spearman-Brown Formula

a mathematical formula used to predict the degree that the reliability of a can be improved

Psychometric Validity

the degree to which a measurement actually measures what it is supposed to measure

Constructs

something that cannot be directly seen or touched but affects and helps to explain things that are visible

Construct Validation

the strategy of establishing the validity of a measure by comparing it with a wide range of other measures

Generalizability

a more sophisticated view that sees reliability and validity as aspects of a single, broader concept

An effective way to make a measure more reliable is to…

Use procedures that are as similar as possible for all participants

When the results of a study can be applied to many different kinds of people, situations, parts of the world, and methods of gathering data, then…

generalizability is high

Types of Research Designs

Case method, Experimental Method, Correlational Method

Case Method

a research technique that involves closely studying a particular event or person in order to find out as much as possible

Experimental Method

A research technique that establishes a relationship between an independent variable (x) and dependent variable (y), assign them to random groups, and measure the average behavior

Correlational Method

focus on examining the relationships between two or more variables to determine if they are connected

Null-Hypothesis Significance Testing (NHST)

can be used to determine how likely a result is if there is no relationship between two variables

Null Hypothesis

the claim in scientific research that the effect being studied does not exist.

P-level

In null hypothesis statistical testing, the calculated probability that an effect of the size (or larger) obtained by a study would have been found if the actual effect in the population were zero

Type I Error

In research, the mistake of thinking that one variable has an effect on, or relationship with, another variable, when really it does not

Type II Error

In research, the mistake of thinking that one variable does not have an effect on or relationship with another, when really it does.

Effect Size

a number that reflects the degree to which one variable affects, or is related to, another variable

Correlation Coefficient

used to measure how strong a relationship is between two variables

Confidence Interval

An estimate of the range within which the true value of a statistic probably lies

Replication

Doing a research study again to see if the results hold up

Publication Bias

The tendency of scientific journals to mainly publish studies with strong results

A good way to interpret an effect size is to

compare it to a rule of thumb

Replication and open science

are useful for determining whether a finding is dependable, meaning that other researchers are also likely to find the same effect.

Open Science

A set of emerging principles intended to improve the transparency of scientific research and that encourage fully reporting all methods and variables used in a study, reporting studies that failed as well as succeeded, and sharing data among scientists

Personality test are useful when

used by employers to select the best candidate for the job.

We should think about how representative researchers are of the general population because

a person’s background and interests influence what topics they choose to study

If measurement errors are truly random, then they should…

Sum to Zero

In general, questions about reliability concern__, whereas questions about validity concern__

Consistency; Accuracy

Validity is the degree to which a measurement…

Actually reflects or measures what you think it does

At the heart of aggregation is the idea that…

Random errors cancel each other out

A research strategy that involves gathering as many different measurements as you can of. particular construct and determining if those measurements correlate is called…

Construct Validation

Many psychologists tend to use college students as participants in their research and then assume that what they learn applies to people in general. However, this common practice may limit the…

generalizability of their findings

Reliability and validity are actually both aspects of a broader concept called…

Generalizability

All of these are questions asked of generalizability EXCEPT which option?

Does your result cause harm to participants?

The major difference between the experimental and correlational methods is that in the experimental method, the presumed causal variable is___, whereas in the correlational method, the same variable is ___

manipulated; measured

Random assignment allows researchers to…

assume that groups of participants are more or less equivalent on preexisting conditions