Chapter 4 - Psychological Measurements

Measurement

Systematic assignment of values to characteristics of people/objects

-Using clearly defined rules, values meaingfully represent attribute

Construct

Abstract, unobservable concept

-Measurement helps us represent these via measurable variable

Psychological Measurement Path

Conceptual Definition (Conceptualization) → Operational Definition (How We Measure) → Observations/Data (values we obtain)
Conceptualization - Defining what we mean by an abstract concept (what it does/doesn’t encompass)
Operationalization - Translating an abstract concept into something that can be measured

Good Measurement Should

1. Difference in scores reflect real difference in what we intend to measure

2. Capture what we intend to measure, minimal influence from other factors

3. Produce scores that can be meaningfully interpreted, and used to evaluate hypotheses

Poor Measurement Leads to

1. More noise, less precision - Scores reflect random fluctuations rather than meaingful differences

2. Misleading conclusions - we may think we’re measuringone thing, but we’re actually measuring something else

3. Wanted Research Efforts - Even a large, representative sample and good study design can’t compensate for fundamentally flawed measurements

Categories of Variable Measurement

Self-Report Measures - Participants report their own thoughts, feelings and actions

Behavioural Measures - Other aspect of a participant’s behaviour is observed/recorded

-Broad category, includes observation of people’s behaviour in structured laboratory tasks or in a natural setting

Physiological Measures - Involve recording of a wide variety of physiological process (heart rate, blood pressure, electrical activity, blood flow)

Self-Report Measure

Participants report their own thoughts, feelings and actions

Advantage - Efficient/Easy to administer, can measure internal experience not directly observable, inexpensive/scalable

Disadvantage - Vulnerable to response bias, depends on self-awareness/honesty

Behavioural Measure

Construct inferred from observable action/performance recorded using clear rules

Advantage - Capture behaviour in real context (sometimes), less vulnerable to response bias

Disadvantage - Behaviour may be influenced by self-presentation, observation, or context

Physiological Measure

Construct inferred from biological or bodily response linked to psychological process

Advantage - Not dependent on self report/awareness

Disadvantage - Expensive/invasive, required specialized equipment

-Signals also reflect multiple constructs

Reliability

Extent to which a measurement produces consistent scores under conditions where characteristics being measured is expected to remain the same

-Measure is reliable if:

1. Scores are similar across repeated measurements/observers/components when similarity expected

2. Scores relatively unaffected by random factors (distractions, accidental guessing)

Types of reliability: Test-retest, internal consistency, interrater

Converging Operations

Multiple operational definitions of the same construct, either within or across studies

Measurement Error

Random fluctuations in score that are not due to real changes in the construct or the individual being measured

Classical Test Theory (CTT)

Every measurement value/score includes two conceptual components:

1. True score component

2. Measurement error component

Internal Consistency

Consistency across parts of the same measure (items/questions on a test/trial)

-Score of +0.80 is good internal consistency

Cronbach’s alpha/McDonald’s Omega - Indices reflect how consistently items function together as a scale

Test-Retest Reliability

Consistency across time

Ex; Giving an IQ test to someone now and in 3 days should yield consistent scores

-Assumes characteristic being measured is relatively stable over measurement interval

Inter-rater Reliability

Consistency across different observers evaluating the same behaviour

-More relevant when measurement involves judgement or coding

Ordinal Level

Assigning scores so that they represent the rank order of the individuals

-ranks communicate whether one individual is higher or lower on that variable

Problem: difference between adjacent scale values don’t represent equal intervals on the underlying scale

Interval Level

Assigning scores using numerical scales in which intervals have the same interpretation throughout

-Ex; IQ

-Lack a “true zero” point, no “complete absence” of something (ex; IQ = 0 doesn’t mean you lack intelligence)

Ratio Level

Involves assigning scores in such a way that there is a true zero point that represents complete absence of the quantity

-Ex; Height measured in meters, amount of money

Money has a “true zero”, so 25 cents is double that of 50 cents, but 0 cents means you have no money at all

Validity

Extent to which scores from a measure represent the variable they are intended to, three types of validity;

1. Face validity

2. Content validity

3. Criterion validity

4. Discriminant validity

Face Validity

Extent to which a measurement method appears, on superficial examination, to measure the construct of interest

-Weak kind of evidence since it’s based on human intuition

Content Validity

Extent to which a measure reflects all aspects of the construct of interest

Criterion Validity

Extent to which people’s score on a measure are correlated with other variables (known as criteria) that one would expect them to be correlated with

-Criterion - Variable that theoretically could be correlated with the construct being measured

Further split into Concurrent, Predictive, Convergent validity

Concurrent, Predictive and Convergent Validity

Concurrent Validity - Criterion is measured at the same time (concurrently) as the construct

Predictive Validity - Criterion is measured at some point in the future after the construct has been measured

Convergent Validity - New measures are correlated with existing measures of the same construct

Discriminant Validity

Extent to which scores on a measure of a construct are not correlated with measures of other, conceptually distinct, constructs and thus discriminate between them

Four Steps in Measurement Process

1. Conceptually defining the construct

2. Operationally defining the construct

• May involve using an existing measure or creating your own measure

• Existing measures save time, have evidence if it is valid/relible, and results can easily be compared with/combined against previous results

• Creating measures can involve variations of existng measures, should be simplistic, and ensure it is of an appropriate length/sufficient to test your construct

3. Implementing the Measure

• Must be aware of participant bias, researcher bias, reactivity, etc

4. Evaluating the Measure

• Done via Criterion validity, assessing test-retest reliability, etc

Cautions in Implementing a Measure

Socially Desirable Responding - When participants respond in ways that they think are socially acceptable

Demand Characteristics - Subtle cues that reveal to participants how the researcher expects them to respond in the experiment

-These can be minimized by making procedure clear/brief as possible, guarantee anonymity, testing in properly space groups, using the same writing utensil, etc

Target Population

Group of individuals/units researcher is interested in studying and wish to generalize findings to

-Population chosen is defined by characteristics that make someone eligible (age, role, location) → characteristics relate/affect what you study

Sample and Sampling

Sample - Subset of target population that is actually studied

-Inferences about target population made from sample

Sampling - Process of selecting individuals from target population

-Ideally, makes sample representative of the population

Sampling Frame

Individuals within population that can be accessed/available to researcher, and from which the sample is actually selected

Why Carefully Define Target Population

1. Different populations behave differently

• Age, education, culture can change results/conclusions

2. Every study makes a claim about who it applies to

• If we don’t know who, conclusion/claim is incomplete

3. Prevents overgeneralization

• Conclusion shouldn’t extend beyond population it is designed to represent

4. Population affects study design/sampling

• Clear definition needed to build sampling frame/decide who is eligible

5. Judging representativeness

• Can only evaluate representativeness if we know what it’s supposed to represent

6. Replication and Comparison

• Other researchers need to know who we intended to study to repeat the study

Categories of Techniques for Sampling

Sampling technique has implications for conclusions you can/cannot draw

1. Probability Sampling - Each member of the population has known probability of being selected, strengthening representativeness and generalization to the population

2. Non-Probability Sampling - Probability of selection is unknown, limits generalizability

• Despite limitations, it is most common/necessary due to practical constraints

Probability Sampling Techniques

1. Simple Random Sampling

2. Stratified Random Sampling

3. Cluster Sampling

Simple Random Sampling

Each member of target population has equal probability of being included in the sample

Advantage: Increases representativeness, minimizes systematic bias, conceptually simple

Disadvantage: unfeasible (dont have list of entire population) and very costly

Stratified Random Sampling

Population divided into meaningful subgroups (strata), random sample drawn from each stratum

Advantage: Representative key subgroups, increases precision of estimates

Disadvantage: More complex to design/implement, many factors in dividing population

Cluster Sampling

Population divided into clusters, random sample of clusters selected

-All, or some, members within selected clusters are studied

Advantage: Cost-effective for large/geographically spread-out populations

Disadvantage: Less precise, higher risk of bias if clusters differ systematically

Non-Probability Sampling Techniques

1. Convenience Sampling

2. Purposive Sampling

3. Quota Sampling

Convenience Sampling

Participants selected based on ease of access/availability

Advantage: Fast/cheap/easier to implement

Disadvantage: high risk of sampling bias, limited ability to generalize to population

Purposive/Judgmental Sampling

Participants intentionally selected because they possess specific characteristics relevant to research question

Advantage: Useful for studying specialized populations, allows focused/theory-driven sampling

Disadvantage: Researcher’s judgement may introduce bias, doesn’t always support generalization to broader population

Quota Sampling

Participants selected to match predefined proportions of certain characteristics in the population

-Selection within each category is non-random

“Non-probability” version of stratified sampling

Advantage - Ensures representation of key subgroups, more structured than simple convenience sampling

Disadvantage - Non-probability sampling, uknown selection probabilities limit generalizability

Sampling Bias

Selection of participants makes sample systematically different from population based on relevant characteristics

-Results in some groups being over/under represented, reducing representativeness/systematically distorts estimates relevant to the study

Conclusion may not accurately reflect what we’re studying

Coverage (Sampling Frame) Bias

Some members of population excluded from sampling frame

-Sample systematically over-represents individuals included in the frame, under-represents those who are excluded

Leads to biased estimates of population characteristics

Nonresponse (Response Rate) Bias

Contacted individuals of a study do not response/participate

-Responders may differ systematically from nonresponses on variables relevant to the study, causing sample estimates to be biased

Self-Selection (Response Rate) Bias

Individuals choose to participate based on interest/motivation/personal relevance

-Sample over-represents individuals for whom topic is salient

-Inflates/deflates estimates of attitudes or behaviours

External Validity

Degree to which a study results generalize beyond specific sample, setting, time, and procedures in the study

-Representativeness is one fact that contributes to external validity

More representative sample strengthens ability to generalize results to the target population

External Validity Depends on

1. Setting/immediate context

2. Place/time periods

3. Operational definition