Psychological Measurement and Research Methods

Measurement

An assignment of scores to individuals that represent a characteristic of the individual (Ex: Weight, temp, IQ)

Psychometrics

Psychological Measurement

Example of Psychometrics

Testing working memory w/ Digit Span test

Psychological Constructs

Variables of interest that we can not assess by looking at the person

Conceptual Definitions

describes behaviors/internal processes that make up a construct and how it relates to other

Example of Conceptual Definitions

Neuroticism: people's tendency to experience negative emotions such as anxiety, anger, or sadness in a variety of situations

Operational Definitions

A definition of a variable in terms of how precisely it is measured

Three Categories of Operational Definitions

Self Report Measures, Behavioral Measures, Physiological Measurement

Self Report Measures

Participant reports their thoughts, feeling, and actions

Behavioral Measures

Participant's behavior is observed by a researcher

Physiological Measurement

Participants' physiological processes such as heart rate, urine, and blood glucose levels are measured

Converging Operations

When psychologists use multiple definitions of the same construct

Levels of Measurement

Nominal. Ordinal, Interval, Ratio

Nominal

Categorical Variables - Categories are coded (Ex: Gender, Sex, Religion)

Ordinal

Rank Order - Higher or Lower (Ex: Birth order, times in mario kart)

Issue with Ordinal

Can't assume equal distances (differences) in the order

Interval

Using a numerical scale in which the intervals have the same distance between them (Ex: Fahrenheit or Celsius)(No true zero)

Ratio

Using a numerical scale in which the intervals have the same distance and there is a true zero (Ex: Money, height, weight)

Reliability

Consistency of a measurement

Test-retest reliability

Consistent across time and space (Ex: Measuring same group at two different times, if correlation is above r = .80 then it is good)

Internal Consistency

Consistency of participants' responses across the items on a multiple-item measure. (Split-half correlation & Cronbach's Alpha)

Split-half correlation splitting the items into two sets and then examining the relationship between them

Cronbach's Alpha

Is a mean of the split-half correlation for a set of items (most common measure of internal consistency)

Inter-rater reliability

The extent to which different observers (researchers) are consistent in their judgements of the same person

Validity

The extent to which the scores of a measure are accurately measuring what it is supposed to

Types of Validity

Face Validity, Content Validity, Criterion Validity, Convergent Validity, Discriminant Validity

Face Validity

The extent to which a measurement appears "on its face" to measure the construct of interest

Content Validity

The extent to which a measure "covers" the construct of interest

Criterion Validity

The extent to which people's scores on a measure correlate with other variables (known as criteria) that one would expect them to correlate

Concurrent Validity

When the criterion is measured at the same time

Predictive Validity

The criterion is measured at some point in the future (after the construct has been measured)

Convergent Validity

Positive correlation with established measures of the same construct

Discriminant Validity

Lack of correlation with conceptually distinct variables

Conceptual Definition

Crucial for good measurement, helps in making sound decisions about to measure the construct

Using an Operational Definition

Preferable due to time-saving, existing, validity evidence, and ease of comparison. Many are free to use with proper citation, may need to choose among alternatives based on factors like commonality, validity evidence, or ease of use

Creating an Operational Definition

Necessary when no suitable existing measure exists or to evaluate convergent validity

Creating own Measure

Strive for simplicity, include practice items & opportunities for questions, keep measures brief, use multiple items rather than single items for better content validity and reliability, Ensure multiple items can be combined into a single score, test measure on small group beforehand

Demand Characteristics

Subtle cues that reveal how the researcher expects participants to behave and makes participants change behavior

When Implementing Measures

Test parti. under similar, quiet, distraction free condition

Evaluating measure

Assess reliability and validity even for established measures with: Test-retest (when possible), Internal consistency (for multiple-item measures), Convergent and discriminant validity, Criterion validity through successful experimental manipulation

If data casts doubts on reliability/validity, possible cause:

Issue with measure or administration of measure, problem w/ conceptual definition, failed experimental manipulation

Purpose of experiments

Study designed specifically to answer if there is a causal relationship between two variables, the IV and the DV

IV

Changes level systematically (Ex: Different groups get different levels or conditions of the IV/Or same group gets different levels or conditions of the IV over time)

Manipulating Variable

Requires researcher action, not same as comparing pre-existing groups

Ways to Manipulate IV

Two Conditions - Single Factor & Multi-level

Single Factors Design

Two level: One variable, two conditions

Multi-level Design

One variable, more than two conditions

Control Groups

Reduce noise in the data

Extraneous Variables

Make noise in data

Confounding Variables

Extraneous variable that differs on average across levels of IV

Between-Subjects Experiment

Each participant is tested in only one condition (Ex: 50 Vers. A, 50 Vers. B, 50 Vers. C)

Random Assignment

Primary way to control for extraneous variables across conditions. Method of assigning participants to condition groups.

Criteria for Random Assignment

Each participant has as equal chance of being in each condition and Each participant is assigned to a condition independently of other participants

Modified random assignment

Block randomization - All conditions occur once before they are repeated

Matched-Groups Design

Alternative to random assignment. Participants matched across conditions on key variables. Differences in groups are due to manipulation not pre-existing factors (Ex: ADHD users, ADHD non-users, without ADHD users, without ADHD non-users)

Within-subjects design

Each participant experiences all conditions.

Advantages for within-subjects design

Controls for participant variables and Can use statistical methods to reduce noise, more powerful results

Disadvantage for within-subjects design

Participants may guess hypothesis, May affect behavior/judgements, Not suitable for all experiments

Carryover Effects

Primary disadvantage of within-subjects design.

Practice effects(carryover)

Improvement over time due to exposure

Fatigue effect(carryover)

worse performance over time

Context effects(carryover)

earlier conditions influence later conditions

Solution to carryover

Counterbalancing - Test participants in different orders (ABC, ACB, BAC, etc)

Difference between between and within

Btwn: Simpler, less time per participant, avoids carryover effects Wtn: Controls extraneous variables, offers more statistical power (use if possible)

Internal Validity:

Shows causation, not just correlation. Very high in experiments with controlled variables

External Validity:

Generalizability to real world situations. Improved by realistic settings and diverse participants

Construct Validity:

Quality of experiment's operationalization. How well variables represent concepts being studied

Statistical Validity

Proper use of statistical tests. Adequate sample size and power

Recruiting Participants

Consider pop. you want to target, subject pool, non-subject pool (ads,flyering)

Treatment

Intervention (condition) to change behavior

Control:

no intervention. Includes no-treatment and placebo

Placebo

Simulated treatment without active ingredients. Can produce positive effects (placebo effect)

Control Condition types

Placebo & Waitlist

Placebo Control

Stimulates treatment without active elements. Helps distinguish treatment from placebo effects

Waitlist Control:

Participant await treatment. Maintains expectations of involvement

Alt approach to control

Compare new treatment to best available treatment

Standardizing the procedure

Critical to minimize extraneous variables and prevent confounding results

Manipulation Check

A manipulation check is a separate measure of the construct the researcher is trying to manipulate. Verifies independent variable was successfully manipulated, Important for constructs manipulated indirectly, Crucial when results are null , Usually done at end of the procedure

Pilot Testing

A test run of the study with mock participants. Answers important questions: Clarify of instructions, Participant engagement, Effectiveness of manipulations, Presence of demand characteristics, Timing and technical issues

Correlational Research

Assesses the statistical relationship (aka correlation) between two variables of interest, Little to no effort to control extraneous variables, Describe the relationship and predictability

Data collection

Interview Style or Online survey

Displaying the Data

Scatterplots: each point represents a person's data point. Positive = Both increase or both decrease together, Negative = When one increase, the other decreases (or vice versa). Measured using a Pearson's r value: Range from +1.0 to -1.0

Directionality problem

What came first the chicken (variable X) or the egg (variable Y)

Third variable problem

Another variable (Z) could cause both X and Y

Spurious Correlations

Often the result of a third variable

Factor Analysis

technique used to study relationships among many variables. Organizes variables into clusters (factors) that are strongly correlated internally but weakly correlated with other clusters. Each cluster represents an underlying construct measured by multiple variables

Correlation Matrix

A table showing correlation coefficients between pairs of variables

Partial correlation

a statistical technique used to control for potential third variables in research. Little change = the relationship is independent of the controlled variable. A significant drop = the controlled variable is a third variable explaining the relationship. A partial drop = the controlled variable accounts for some, but not all, of the relationship

Regression

a statistical technique used to predict on variable (outcome) based on the other variable (predictor)

Multiple Regression

Shows the predictors contribution while controlling for other variables

Qualitative Research Characteristics

Less focused research question. Large collection of "unfiltered data" from a small amount of individuals. Uses non-statistical techniques for data description. Focuses on understanding participants' experiences in detail. Less concerned with drawing general conclusions about human behavior more concerned with trying to understand these experiences have

Strengths Qualitative Research

Can lead to new hypotheses for quantitative research. Offers "rich description" of human behavior in real-world contexts. Conveys a sense of what it's like to be in a particular situation

Data collection Qualitative

Varied approaches including naturalistic observation, participant observation, archival data, artwork

Quantitative Research

Starts with focused question/hypotheses, collects limited data from many individuals, uses statistical techniques for analysis, aims to draw general conclusions about larger pop.

Qualitative Research

Generates novel and interesting research questions, Provides detailed descriptions of behavior in specific contexts, Communicates the "lived experience" of research participants

Grounded theory (quantitative research)

researchers start with the data and develop a theory or an interpretation that is "grounded in" those data. Done in stages: Identify ideas that are repeated throughout the data, Then organize ideas into a smaller number of broad themes, Finally, they write a theoretical narrative: an interpretation of the themes

Main Criticism Qualitative:

Subjective, hard to validate, limited generalizability

Main Criticism Quantitative

oversimplifies human behavior

Mixed Methods Research

Combines quantitative and qualitative. Qualitative for hypothesis generation, quantitative for testing.

Naturalistic observation

Observing behavior in its natural environment (like being a fly on a wall) Can be disguised or undisguised (reduces reactivity)