Scale Construction Part 1; Rational/Deductive Approach

Reliability

refers to the consistency of test scores - if a scale is reliable, this means that scores are “repeatable”

Validity

refers to the meaning of test scores - if a scale is valid, this means that test scores assess the construct we intended

High reliability

tells us that the scale measures something

we can say that the scale is prescise in assessing something

High validity

tells us what the scale is assessing

Good scale construction

Carefully developing a scale increases the likelihood that it will be reliable and valid

Develop the scale using proper construction practices

Conduct research to test whether the scale is reliable and valid

A completed scale (measure)

1. definition of the construct (conceptual definition)

2. items tapping the construct

3. response categories to answer the items

4. instruction set

5. if applicable, interpretation aids, scording criteria, norms, standards for interpreting test scores

General steps in the process of scale construction

1. examine literature and previous tests

2. identify the primary purpose for which the test be used

3. define the construct of interest

4. define the domain (can prepare a table of test specifications to aid this step)

Construct-driven (rational/deductive) approach

start with a theory, use it to define the construct, and then develop items that measure that construct

order: theory → construct definition → items

step 1: examine literature and past scales

• review what is already known about the construct

• understand how it relates to and differs from other constructs

• examine how it has been measured previously

• explain how your measure improves on past approaches and why you chose to measure it this way

helps develop the theory explaining what the construct is and why it matters

Step 2: identify primary purpose

determine what the scale will be used for and be specific

purpose - research or placement/diagnostic decisions

specific decisions - what decisions will the measure help make?

research - what type of research will it support (experiemental or correlational)

target population

step 3: construct defintion

extremely important

articulate what is meant by your construct and what sort of things are not part of the construct

indicate what high and low levels of the construct mean

specify how your construct is similar to, and different from, other related constructs

nomological network

which helps to define your construct

tests whether a construct relates to other constructs as theory predicts

helps establish construct validity

step 4: define the “domain” of the construct

how it manifests - how the construct appears in behaviours, thoughts, or feelings

content - what aspects of the construct should be measured

test specification table

test specification table

organizes the different manifestations (behaviours thoughts feelings) and content to ensure the scale measures all important aspects of it

helps ensure content validity by checking that the scale covers the construct

content-specific (domain specific)

measures a construct in a specific situation (feeling useless as a student)

General (non domain specific)

measures a construct overall, without a specific situations (e.g., generally feeling useless)

Scale breadth

the question of how narrow or broad your test should be

broad = items are heterogeneous (items are different from each other)

narrow = items are homogeneous (items are more similar to each other)

• also non domain specific

why is broad heterogenous items are useful

example “want to capture stressful life experiences in different ways

Homogenous items (narrow measure)

responses will be more highly positively correlated

higher inter-item correlations

heterogeneous items (broad measure)

responses will be less highly positively correlated

lower inter-item correlations

inter-item correlations

how strongly the questions on a scale are related to each other

high correlations suggest the items may be measuring the same thing

Correlation

the most used statistic

measure of the association (relation) between two variables (direction, magnitude)

ranges from 0 to 1

can be + or -

Positive Correlation

As variable X increases, so does Variable Y

Negative correlation

As variable y increases, variable x decreases

Inter-item correlations (rij)

measure how strongly the items/questions on a scale relate to each other

Narrow scale (inter-item correlations)

items measure a very similar concept, so their correlations are higher (e.g., .40-.70)

Broad scale (inter-item correlations)

items measure different aspects of a larger construct, so their correlations are lower (.01-.15)

Pro of Narrow

narrow scales have higher internal consistency (Cronbach’s alpha) because the items measure similar things

similar items have higher inter-item correlations, relate more strongly to each other

Cronbach’s alpha

Depends on the average correlation between items and the number of items in the scale

estimates reliability

Pros of Broad

Measure more aspects of a construct, predict a wider range of attitudes and behaviours

cover more aspects → may correlate with more variables in future research

but lower cronbach alpha

Bandwidth-Fidelity Trade-Off

narrow scale: measure a specific aspect of a construct → predict outcomes related to that aspect well but may not relate to other variables greater fidelity

broad scale: measures more aspects of a construct → may correlate with and predict a wider range of variables greater bandwidth

Bandwidth-Fidelity Trade-Off Example

academic self esteem may relate to academic performance but not necessarily leadership work (measure a specific aspect of self esteem)

global self esteem measures self esteem more broadly, it may relate to leadership in uni and at work (measure g.s.e. across more areas

Multidimensional

measures multiple aspects of a construct

separate subscales, each measuring a specific aspect with its own defintion

homogeneous items - items within each subscale measure the same specific aspect

Unidimensional

measures only one underlying construct or dimension

clusters of correlated items

some items correlate highly with each other because they measure the same dimension, but correlate less with items measuring other dimensions