research methods

Goals of surveys

Look at thoughts, feelings, or opinions of people

Mail (conducting surveys)

Advantages—> low cost, anonymity, no time pressure

Disadvantages—> response rate (low), are the people who respond representative

Phone (conducting surveys)

Advantages—> possibility for wide range of responses, slightly harder to say no

Disadvantages—> access to #’s, lack of anonymity

Internet (conducting surveys)

Advantages—> possibility for wide range of responses (and # of people), low cost

Disadvantages—> representativeness

Personal interviews (conducting surveys)

Advantages—> high response rate

Disadvantages—> costly (time), lack of anonymity

Population

Set of all cases of interest, group you want to know about

Sampling frame

Operational definition of population

Sample

Actual subset of population drawn from sampling frame

Representativeness

A sample is representative to the extent that is exhibits that same distribution of characteristics as the population from which it was selected

Bias

When differences between sample and population are systematic as opposed to random

Selection bias

When procedures used to select the sample result in the over- or underrepresentation of some segment of the population

Response bias

When certain segments of the population are more or less likely than others to complete the survey

Probability sampling

Every member of the population has a chance of being included in the sample

Non-probability sampling

Members of population are not selected randomly/do not have an equal chance of being selected

Convenience sampling

Participants are selected based on easy accessibility and proximity

Snowball sampling

Rely on group members to get more people for survey

Typical case sampling

Sample in a way that allows you to survey the “average” part of the population

Quota sampling

Population divided into subgroups of interest and then sample from the subgroups based on some variable of interest

Why use non-probability sampling

• Often easier

• Sometimes a non-random sample isn’t a problem

• Sometimes only sampling design that works

Simple random sampling

Sample is chosen entirely randomly from population

Systematic sampling

Each member of sampling frame is listed, divide by sample size, pick a random number

ex. select every 5th member of population

Stratified sampling

Each member of sampling frame is assigned to a group and randomly select some people from all groups

Cluster sampling

Members of population assigned to cluster, then randomly pick clusters

Open-ended question

Participants allowed to respond to question in any way they want

Closed-ended question

Set of responses given to participant, limited in how they can respond

Likert scale

Overall scale used to survey people’s levels of agreement, frequency, or importance

Likert-type scale

Individual questions on the survey that assess people’s levels of agreement, frequency, or importance

Advantages of open-ended questions

• Easier to write

• Provide flexibility for respondent

• Useful if no idea how respondent will respond

Disadvantages of open-ended questions

• Can be difficult to code or summarize

• Responses can be vague, or not complete

Advantages of closed-ended questions

• Answered quickly and easily

• Usually easy to score or code

Disadvantages of closed-ended questions

• Can be difficult to write

• Might leave out options

BRUSO model

B- Brief

R- Relevant

U- Unambiguous

S- Specific

O- Objective

Double-barreled questions

Two conceptually different things asked in the same question

Loaded questions

Contain emotion-laded words

Leading questions

Phrased in such a way to lead people to a particular response

Context effect

The way questions are asked can influence how participants respond, surveys are also sources of influence themselves

Order effects

• Sensitive questions should come after more generic questions

• Demographic questions almost always come at the end

N=1 (single-subject design)

Small number of subjects individually analyzed. No comparison group or balancing (quasi-experimental)

Reasons to conduct N=1 design

• When N=1 is your whole pop

• When N=1 is sufficient because of perfect generalizability (all people have blink reflex)

• When a single instance is all that is necessary to refute a theory

• Limited opportunities to observe particular subjects/behaviors

N=1 procedures

• Repeated measures (behaviors across time)

• Baseline measurement (no pretest)

• Treatment (look at multiple instances after treatment)

Basic A-B design

A is baseline (no treatment, multiple times)

B is after treatment (multiple times)

Reversal designs

A-B-A: Baseline, treatment, take away treatment

A-B-A-B: Baseline, treatment, take away treatment, treatment

Data analysis of N=1 designs

• Visual inspection

• Statistical analysis

Weaknesses of N=1 design

• External validity

• Data analysis issues

• Inability to assess interactions

Observations without interventions

Sitting and observing

Observations with intervention

Person doing observing interacts with those being observed in some way

Structured observation

Researcher intervenes in a very specific way

Field experiments

Researchers manipulate one or more variable in a field

Recording behavior (observational)

Narrative recording (video/audio tape), recording units of behaviors (checklists)

Behavioral sampling

Choosing particular behaviors to observe and picking various times of day to observe

Situational sampling

Observing in a specific situation/setting (child in classroom, places people eat)

Analysis of observational data

• Data reduction (how do you reduce data down and categorize it)

• Statistics (descriptive stats)

Strengths of observational methodologies

• Natural settings

• Theory construction (base theories off of observation)

Weaknesses of observational methodologies

• Reactivity (will your presence/interjection impact behavior)

• Bias (may have bias observing that will impact how you interpret what you observe)

• Basically no control