Social Research Methods Exam #2

Sampling

the process of deciding what or whom to observe when you cannot observe and analyze everything or everyone

Probability Sampling

the process of selecting a sample based on probability, typically involving some random-selection mechanism

Representativeness

When a sample has the same distribution of characteristics as the population from which it was selected

Generalizability

the degree to which you can apply the results of your study to a broader context

Cost-efficiency

the ability to maximize research outputs—such as data quality, sample size, or actionable insights—while minimizing the expenditure of resources, including time, money, and labor

Bias

any systematic error or deviation from the truth in data collection, analysis, interpretation, or publication, leading to skewed results or false conclusion

Margin of error

the amount of uncertainty in an estimate

population

the group that we're interested in generalizing about

sampling frame

the list of units composing your population

elements

the individual units comprising your sample

simple random sampling

each element has the sample probability of being selected

systematic sampling

elements are selected using a fixed interval of every kth element on a list

cluster sampling

a two-stage random sampling process whereby you first sample groups (clusters), then randomly select elements within each cluster

stratified sampling

divide the population into groups, or strata, then sample members in strategic proportions from each group

oversampling

deliberately sampling a group at a higher rate than its frequency in the population

Types of probability sampling

simple random, systematic, cluster, stratified

benefits and drawbacks of probability sampling

benefits: representativeness, generalizable, cost-efficient, avoids bias, provides a calculable margin of error

drawbacks: high costs, logistical complexity, and practical limitations

How does size impact the margin of error

Larger sample → Smaller margin of error → more closely approximates the true population

Steps to conduct probability sampling

1. Identify population

2. Establish sampling frame

3. List of every enrolled student

4. Use random sampling to sample your elements

Nonprobability sampling

A sample that is not drawn using a method of random selection.

Convenience Sampling

- selected due to convenience

Pros: fast, inexpensive, ideal for exploratory research

Cons: selection bias, not generalizable, low representativeness

Purposive Sampling

- selection based on which will be the most useful or insightful

Pros: qualitative studies, provides deep insights

Cons: research bias, limited generalizability

Sampling for range

maximizing respondents' range of experiences with the phenomena under study

Quota Sampling

- units are selected on the basis of pre-specified characteristics, so that the sample will have the same distribution of characteristics assumed to exist in the population being studied

Pros: cost-effective

Cons: selection bias, not generalizable

Snowball Sampling

- interviewees suggest additional people for interviewing

Pros: cost-effective

Cons: selection bias, not generalizable

Key informants

the first point of contact a researcher has with his or her study population

Saturation

when additional data fail to yield new insights and simply reinforce what the researcher already knows

Causal mechanisms

A sequence of events or conditions leading from the cause to the outcome

Probability vs. nonprobability

Probability - random selection

nonprobability - no random selection

Survey

A method in which researchers ask a sample of individuals to answer a series of questions

Closed-ended questions

A focused interview question to which subjects can respond only in preset ways

Dichotomous

Only two options (e.g., "yes" or "no")

Likert scale

ranking

Survey blocks

grouped, organized section of questions within a larger survey

bipolar scale

places two opposing statements on a spectrum

nominal (type of closed-ended question)

set of choices

ranking (type of closed-ended question)

respondents rank-order priorities or preferences

mutual exclusivity

categories that do not overlap with one another

exhaustivity

all potential responses are available

forced-choice

Survey questions that force the respondent to indicate their leanings toward agreement or disagreement

Acquiescence Bias

the tendency to agree no matter what

Open-ended questions

A broad interview question where subjects are allowed to respond in their own words rather than in preset ways

advantages and disadvantages of surveys as a research method

Advantages of Surveys: good for describing large populations, Large samples are feasible, Breadth of topics, Reliable

Disadvantages of Surveys: assesses causality, measurement validity, lack of context error, nonresponse

Understand why closed-ended questions or open-ended questions are used for different purposes

Closed-Ended Questions: The Path to Quantitative Data

Open-Ended Questions: The Path to Qualitative Depth

Know the 3 criteria of closed-ended question design

1. mutually exclusive

2. exhaustive

3. simple and easy to understand

face-to-face interviews

interviewer meets in person with the respondent and asks a series of questions aloud

social desirability bias

study participants report positively valued behaviors and attitudes rather than giving truthful responses

Telephone Survey

interview administered over telephone

self-administered questionnaire

individual independently completes questionnaire themselves

online surveys

respondent answers survey online

ordering

When the order in which questions appear biases the responses

priming effects

when exposure to a particular image, word, or feeling shapes how respondents think and feel in the immediate aftermath

formatting

Consistent

Uncluttered

Intuitive

Clear

index

A composite measure that sums responses to survey items capturing key elements of a particular concept being measured.

scale

A composite measure that averages responses to a series of related items that capture a single concept or trait, such as depressive symptoms or self-esteem.

filter questions

A question or series of questions associated with a conditional response to a prior question

split-ballot design

one half of the sample receives one module, and one half receives another, determined by random assignment

advantages and disadvantages of the different survey administrations (face-to-face, telephone survey, self-administered, online)

face-to-face advantages: high completion rates

face-to-face disadvantages: interviewer effects, social desirability bias

telephone survey advantages: high completion rates and data quality, more cost-efficient, fewer interviewer effects

telephone survey disadvantages: response rate, less rapport, response fatigue, sampling bias

self-administered advantages: low cost, low bias, convenient

self-administered disadvantages: low response and completion rates

online advantages: cheap, easy access and administration, fast

online disadvantages: sampling bias, response rate

background of MTurk

launched by Amazon, hires people to take surveys, low compensation, not moral

what to avoid when designing a survey

avoid monotony and jargon

KISS Acronym

Keep it simple, stupid

Double-barrel question

asks about two or more ideas or concepts in a single question

Leading question

influences or guides respondents toward a particular answer, often implying a preferred response

negation

Contains the words "not" or "no" or "don't" or "without", or other negative words

sensitive topics

Discomfort, shame

--> Social desirability bias or nonresponse bias!

recall bias

when respondents do not accurately remember a past event or experience or leave out details when reporting about them

why is KISS important

Short

Consistent

Easy

What are the seven things to look for when analyzing survey design

double-barreled questions

requires expertise

involves a negation

unclear wording

leading question

requires excess time

sensitive topics

independent variable

in an experiment, the "cause"

experimental group

the group that receives the treatment condition

control group

the group that does not receive the treatment condition

random assignment

subjects are assigned to E or C randomly

pretest/posttest

measuring values on DV before AND after experiment is conducted

double blind

neither the subjects nor the experimenters know which is the experimental group and which is the control

random selection

determines entry into study; everyone has equal chance of being selected

lab experiments

take place in a lab

pros: high internal validity

cons: low external validity

field experiments

takes place in a natural or "real-world" setting

pros: higher external validity, applied questions

cons: low internal validity, not generalizable, low external validity

population survey experiments

conduct an experiment within a (representative) survey

pros: higher external validity

cons: lower internal validity

Natural Experiments

the independent variable is manipulated by "nature," not by the experimenter

pros: occur in realistic settings, effective for evaluating social interventions

cons: less control, not true random assignment

factorial designs

uses two or more independent variables

between subjects

a study design in which participants are randomly assigned to different levels of the independent variable

within subjects

a study in which participants receive all levels of the independent variable

3 requirements for causality

time ordering, nonspuriousness, correlation

elements of a classic experiment

experimental subjects

independent variable

experimental and control groups

random assignment

pretest/posttest

double blind

AA experiment

Interview 100 alcoholics who never attended AA. Measure their alcoholism. Ask a year later if they attended AA in the meantime. Measure alcoholism again.

Finding: Alcoholism decreased among AA attendants and did not decrease among non-attendants

random selection vs. random assignment

Selection: determines entry into study

Assignment: treatment condition once already selected

how to apply a factorial design if given (2x3 and 2x2)

#x# = number of categories for IV1 x number of categories for IV2

Number of conditions = the product of those numbers

BETWEEN vs. WITHIN subjects

Between-subject: participants are randomly assigned to EITHER treatment OR control

Within-subjects: participants receive BOTH treatment AND control

key advantages and disadvantages of experiments

Three key advantages of experiments: Causality, High internal validity, They uncover mechanisms, explaining both if and why

Disadvantages of Experiments: External validity, Ethical issues

What was the main goal of the Pager Study?

To examine how criminal records and race interact to affect employment opportunities.

Why did Pager use an audit methodology instead of self-reports?

Self-reports do not allow for causal conclusions; audit studies allow for direct testing of the impact of a criminal record on hiring.

In Pager's study, how were race and criminal record treated in the design?

Race was a between-subjects variable (randomly assigned tester pairs), and criminal record was a within-subjects variable.

What were the key findings of Pager's study regarding race and criminal records?

A criminal record reduces job prospects significantly, and White applicants with a record received more callbacks than Black applicants without one.

What is a limitation of using MTurk for research?

The sample is limited to tech-savvy, often socially liberal users, which limits generalizability.