Research Methods Surveys and Observation

Observational Research

when a researcher watches or monitors participants (people or animals) and systematically records their behavior

Naturalistic

observation “in the wild”

Controlled

observation in controlled setting such as a lab

Event sampling

count frequency of “events” of interest during a designated period of measured time

Time-Based Sampling

record pre-specified short periods of time and record whether the behavior is present or not or count frequencies within clips (e.g. randomly choose 10 30-s segments)

Observer Bias

when observers’ expectations influence their interpretations of the participants’ behaviors or outcome in the study

Observer Effects

when observers’ expectations influence how they behave towards participants

Inter-rater reliability

computed from pairs of ratings typically based on a subset (15-20%) of the data

Guidelines for assessing reliability

• 0-0.2 = poor agreement

• 0.3-0.4 = fair agreement

• 0.5-0.6 = moderate agreement

• 0.7-0.8 = strong agreement

• >0.8-1 = almost perfect agreement

  • >.7 or above is preferable for any publishable study

Reactivity

tendency of participants to act differently if they know they are being observed

Survey

best if perspective of the self is of interest, it’s possible to accurately self-report a behavior, using open-ended responses to get to know phenomenon

Observation

allows measurement of behavior that otherwise would be hard to access from self-perspective

Single-variable frequency claims

based on data from one self-reported variable

Leading questions

wording leads people to a particular response

Double-Barreled Questions

asks two questions in one which can confuse the respondent.

Negatively worded questions

questions with negative phrasing which can lead to confusion or biased responses.

open-ended questions

allow respondents to answer any way they like

forced-choice questions

people give their opinion by picking the best of two or more options

Likert-scale

each response value is labeled as strongly agree, agree, neither agree or disagree, disagree, and strongly disagree used to measure attitudes or opinions on a given statement.

Semantic Differential Format

respondents are asked to rate a target object using a more open-ended rating on numeric scale anchored with bipolar adjectives

Response Sets

a type of shortcut people can take when answering survey questions (also known as non-differentiation) - weaken construct validity

Acquiescence

yea-saying - Occurs when people say “yes” or “strongly agree” to every item instead of thinking carefully of their answer

Reverse-worded items

change the wording of some items to mean the opposite to reduce response bias and encourage thoughtful answers.

Fence sitting

refers to when participants are hesitant to choose extreme answers and always opt for neutral responses

Socially Desirable Responding/faking good

the phenomenon of survey responses that give answer to make them look better but they decrease the survey’s construct validity by portraying an overly favorable image of oneself.

Flashbulb Memories

Remain vivid over time even if they decline with accuracy and are typically associated with significant emotional events.

Frequency Claims

online ratings based on how often something occurs in a given dataset.

Codebooks

clear rating instructions for recording data in research studies, detailing variable definitions and coding schemes.

Masked Design (blind design)

 a common way to prevent observer bias and observer effects - Observers are unaware of the purpose of the study and the conditions to which participants are assigned

Unobtrusive observations

the observer makes themselves less noticeable

Unobtrusive Data

instead of observing behavior directly, researchers might measure the traces a particular behavior leaves behind

construct validity

depends on a good match of the type of information needed with the feasibility of self-report in your target population