Methods Exam 2

Conceptualization

Our process for clarifying the meaning of the concepts we use in our research questions

Operationalization

Specifying the measurement procedures

Empirical indicator

Observable characteristic of a concept

Operational Definition

Specific procedures, actions, or

processes used to measure an empirical indicator

archival records

Pre-existing data collected for some purpose other than your study

Verbal report or self-report

often used to measure subjective

experiences

Steps

Reliability

consistency and stability

of our operational definitions

Test-Retest reliability

Test operational definition with the same

people/different time points

Scores can range from 0 (no correlation) to 1 (high

correlation) with scores above .70, .80 being

considered good indicator of high test-retest

reliability

Internal Consistency

reliability

Used for composite measures that contain multiple items

measuring the same concept (e.g., self-esteem scale, a

depression scale)

• If items in the scale represent the same underlying concept,

then people should respond to them in a similar or consistent

way

Cronbach’s alpha

To check how well the items "agree" or hang together, Range 0-1 (higher scores = greater correlation

Reverse coding

When a measurement tool includes items that are worded in the opposite

direction from the others.

Inter-rater reliability

Different data collectors or raters using the same measure get the

same results (for observational or archival measures)

Validity

Goodness of fit between operational definition and

concept it is purported to measure

Construct Validity

Validation of our measure based on an

accumulation of research evidence indicating

that the measure is related to other variables as

theoretically expected

Convergent Validity

Does our operational definition correspond with

another existing alternative measure of the same

concept?

Measurement error

difference between a

measured quantity and its true value

Gross Error

large, unpredictable mistakes caused by human oversight, carelessness, or equipment failure during data reading, recording, or calculation

Social desirability error

the tendency of survey or interview respondents to answer questions in a way that makes them look good or socially acceptable, rather than sharing their true beliefs, feelings, or behaviors

Systematic error

a consistent, repeatable deviation between an observed measurement and the true value

Random error

an unpredictable, chance-based variation between an observed measurement and the true value of a phenomenon

Face Validity:

Does this stress measurement tool look like it measures

stress? (ask experts….)

Paper-Pencil Survey

Hard copy mailed to participant

• Hard copy completed by participant in person

Computer Administered

Computer Assisted Self Administered Interviews (CASI)

Interviewer Administered

• Computer Assisted Personal Interviews (CAPI) →Interviewer asks

questions in person enters information on tablet/computer

• Computer Assisted Telephone Interviews (CATI) --> Interviewer asks

questions on the phone and enters information on tablet/computer

Administrative Data

Data routinely collected by organizations about individuals as part of

government or program operations, not collected explicitly for research

purposes.

Deterministic matching

uses an identifier to link records belonging to the same person

probabilistic matching

uses other variables (eg birth date) to link records likely belonging to the same person

survey data

Data collected specifically for research purposes

De-identified dataset

no direct (unique to individual: name, SSN, street address) and no indirect

identifiers (not unique to individual: Dates (except years) of services, Birthdate, death date, Ages over 89 [unless have category 90+])

Limited dataset

contains some of the indirect identifiers, such as

admission, discharge, date of service, DOB, DOD; State, city, and five digit

or more zip code; ages in years, months, days or hours

program evaluation

Systematic investigation of the structure,

activities, or outcomes of programs

external evaluator

from outside/independent of an organization

internal evaluator

from inside an organization

Formative evaluation

done during the

planning or implementation of a

program

Summative evaluation

An evaluation done at the

end or completion of a

program

resources/inputs

what you require for your program

activities

what your program will do

outputs

immediate products of your program (# of products/services etc)

outcomes

results for your participants (short-term = 1-3 years, long-term 4-6)

impacts

larger changes resulting much later after your program