Exam 2 - Research Methods for the Social Sciences

Population

The entire group of individuals or instances about whom researchers want to learn, often specified by certain characteristics relevant to a study.

Sample

Comprises a subset of population

Sampling

Process of selecting cases from the population for inclusion in the sample.

Unit of analysis

The level or unit of social life on which a research question is focused, such as people, households, groups, events, towns, or countries. and the distinct entity being studied.

Generalizability

The extent to which findings from a particular study hold for the larger population. The extent to which we care about generalizability depends on the research question.W

When do we need to generalize?

Frequency of some characteristic, behavior, or phenomena; if there is a relationship between some characteristics in a population.

When do we NOT need to generalize

Inductively understand the meaning, significance, context, or experiences of a place, people, organization, or phenomena; Build an understanding of how some characteristics may be related or test casual mechanisms.

Problems for census

Expensive, time-consuming, infeasible, and unnecessary

Sampling types

Probability and non-probability

Probability sampling

The probability of selection into the sample is known for all members of the population. CAN BE USED TO GENERALIZE.

Non-probability sampling

The probability of selection into the sample is not known for members of the population. NOT GENERALIZABLE

Probability vs Random

Probability = all population members’ probability of selection into sample is known. Random = all population members’ probability of selection into sample is known and equal.

Random sample

Sample in which every member of population has the same probability of being selected. Desired sample size/population size.

Sampling frame

A list of all possible members of the target population

Steps in probability sampling

Define target population, construct sampling frame, devise sampling design, determine sample size, draw sample.

Types of probability samples

Simple random sample, stratified sample, systematic random sample, cluster sample.

SRS

Create a sampling frame, select desired number of cases at random from sampling frame, probabilty of selection is the same for everyone in the population (= sample size/population size)

Stratified random samples

Divide membesr of population into strata (homogenous groups) based on some relevant characteristic (e.g. gender). Randomly select cases from each of the sampling frames in proportion to the strata’s share of the population.

Systematic random samples

Similar to simple random sample, but only use one random number. Steps: create a sampling frame, decide on a sample size, count how many cases there are in the population, calculate the sample interval (N/n - gap beteween cases that are selected), select a case at random from the first of N/n cases, select every N/nth case that follows.

Cluster samples

Identify “clusters” of cases, such as geographic units; a cluster is a naturally occuring, diverse group of cases in the population. Steps: make a sampling frame of all clusters, randomly select some of the clusters, take all cases within the selected clusters or randomly select some cases from within the selected clusters.

Determining sample size

Desired precision, or margin of error - Absolute size of the sample determines precision. Need large, but not enormous samples for precise estimates; Avaliable resources - Money and time must be considered.

Sampling Error

The difference between actual (true) value for the population and the value estimated using data from a sample. Population value - sample estimate = error.

What’s needed for accurate inferences?

No systematic error (known as selection bias), minimal random error.

Sources of selection bias in probability samples

Coverage error - due to mismatch between target population and sampling frame; Non-response error - due to incomplete data collection.

What happens if non-response is entirely random?

Random error, however, non-response often occurs in a patterned way.

Random error

The amount of random error is directly related to sample size; the larger the sample, the less random error.

Error summary

Large unbiased sample gives good information about the population; Large sample size reduces random error; probability sample eliminates (or substantially reduces) systematic error due to selection bias.

Non-probability sampling

Convenience sample; purposive (or quota) sample; Snowball sample

Convenience sample

Include in your sample whomever is convenient to include; common in qualitative research, experiments, and exploratory research

Purposive (or quota) sample

Include in your sample a pre-set number of cases of different types so that the sample represents those types of interest (e.g. based on gender, occupation, or other relevant characteristics); Picking each case for a reason—usually because of their characteristics (try to get range of views).

Snowball sample

Ask each respondent if they know someone else who might be willing to talk with you. Good for finding people who don’t want to be found, with very specific characteristics, with very rare characteristics.

Probability samples vs non-probability

Probability: hard to create, little to no selection bias, can make generalizations to population. Non-probability: easy to create, generally have selection bias, cannot generalize to population, well-suited to inductive research. Probability samples use random selection to ensure every member of the population has a chance to be included, allowing for generalization to the larger population, while non-probability samples do not use random selection and often include bias, limiting their generalizability.

Qualitative data analysis

Fieldwork research, historical-comparative research, unobstrusive research

Qualitative research

Involves a continuing interplay between data collection and theory building; Central to the qualitative researcher’s is uncovering of social patterns.

Field research

Type of data collection method; emphasis on studying people, social life, and human behavior in its natural settings; Researchers can develop a deeper and fuller understanding of social phenomena within natural setting.

Why field research?

Natural setting, observing social processes, rather than reconstruction of events, Reveal things that otherwise would not be apparent.

Emic and Etic perspective

Emic perspective - you have to believe what your subjects believe in order to fully appreciate their worldview and experience; etic perspective - there is a danger in adopting the points of view of people you are studying (loss of objectivity)

Hawthorne effect

Participants behave differently when they know they are being watched

Selective competence

Treading line between emic and etic perspectives.

Naturalism

Research paradigm - Assumption that objective social reality exists and can be observed and reported accurately.

Ethnography

Entails immersing yourself in the social life of a group, observing and writing about what you see. It focuses on understanding cultures through direct engagement and detailed descriptions.

Ethnomethodology

Research paradigm - Looks at how social order is produced and shared. Focuses on procedures by which people describe the social settings of which they are participants.

Grounded Theory

Research paradigm - Analysis of patterns, themes, and categories emerging from observational data. Inductive approach. Does not require prior literature review.

Conducting qualitative field research

Preparing for field - be familiar with relevant research, gain entry, establish rapport, research or participant, suitable venue. Methodologcal and ethical considerations - guidlines, general direction, be interested, be a good listener (not passive).

Collecting data: qualitative interview

Qualitative interview is based on a set of topics to be discussed in-depth rather than based on standardized questionnaires.

Qualitative in-depth interviews

Start with interview guide, go with flow of interview, in-depth (probe when necessary)explore participant experiences and perspectives.

Collecting data: focus group

A group of subjects interviewed together, prompting a discussion. Advantages: real-life data, flexible, high degree of face validity, fast, inexpensive. Disadvantages: not representative, little interviewer control, difficult analysis, interviewer/moderator skills, difficult logistically.

Strengths of Qualitative field research

Depth of understanding, flexibility, relatively inexpensive, qualitative, greater validity.

Weaknesses of Qualitative field research

Weaker in regard to reliability; suggest an “audit trial”.

Unobtrusive research

A research method that examines social behavior without affecting it, often using existing data or artifacts.

Elements of Social Life Appropriate to Field Research

Practices, Episodes, Encounters, Roles and Social Types, Groups and Cliques, Organizations, Settlements and Habitats, Social Worlds, Subcultures and lifestyles.

Reactivity

Problem that subjects of social research may react to the fact of being studied, thus altering their behavior from what it would have been normally.

Survey

List of questions aimed for extracting specific data from a particular group of people. Researchers select a sample of respondents and administers a standardized questionnaire to them.

Survey Questions

Questionnaire with exact wording of all questions and interactions is prepared ahead of time (predetermined). Answers are usually provided; respondents choose from among available options (closed-ended)

Closed-ended question

A type of survey question that provides respondents with specific options to select from, limiting their responses to those choices.

Open-ended question

A type of survey question that allows respondents to answer in their own words, providing detailed responses without pre-set options.

Steps for Primary Surveys

Choose mode of data collection, construct and pretest questionnaire, recruit sample and collect data, code and edit data.

Survey Questionnaire

Highly structured list of questions and response categories that will be administered exactly to all respondents.

Surveys and Measurement

The accuracy of survey data depends in large part on issues related to measurement validity

Closed-ended Requirements

Comprehensive - all individuals have a category, Mutually exclusive - Only one category per individual, Homogenous - Similar individuals are in a category.

Measurement validity

Are you actually measuring what you think you’re measuring?

Guidlines for survey questions

Ask concise, focused questions, be simple and clear (avoid confusing phrasing), be explicit, be neutral (don’t use politically or emotionally charged language).

Concise, focused questions

Only ask one question at a time (AVOID DOUBLE-BARRELED QUESTIONS).

Avoid confusing/value terminology

Don’t use unknown definition and confusing vocabulary, use explicit frequencies instead of vague quantifiers.

Be Neutral: Minimize risk of bias

Using politically or emotionally charged words can bias responses. So can the use of “leading questions.”

Social desirability

People want to portray themselves in a positive light. As a result, people will report that they did/believe the “right thing” AND people will avoid saying they did/believe the “wrong thing”

Potential problems with survey measurement

Social desirability bias, effect of question order and examples, interviewer effects, item non-response.

Effect of question order

(refer to abortion question on slides Chapter 9 )

Effect of examples

(refer to examples slide Chapter 9 )

Interviewer effects

Interviewer effects occur when the answers respondents provide are affected by the characteristics of the person who is interviewing them.

Survey Sampling

Generally, survey samples are probability samples representative of populations. Method is tailored to extracting data from large number of people. When collecting primary survey data, a major and crucial task is sampling respondents and then tracking the quality of the sample.

What type of sampling do surveys usually use?

Probability sampling

Sampling and error

A large unbiased sample gives good information about the population. A large sample size reduces random error. An unbiased or probability sample eliminates systematic error

Selection bias

Systematic error can happen because of selection bias. Some people (or elements) are systematically not included in the sample and/or have a very low probability of selection.

What might go wrong with getting a generalizable sample?

Not actually a complete list of the population.

Coverage error

Some people are in the population and not in the lsit

Non-response error

Not everyone in the sample answers the questions

Survey response rate

The proportion (or percentage of individuals selected into the sample who complete the surveyR

Response rate

Response rate = # of individuals who completed the survey/# of individuals selected for sample

How can we check to make sure our sample is representative?

Other surveys known to be high quality and are representative of the same population provide a standard for comparison. Compare characteristics of sample from your survey to a standard.

Limitations even with perfect samples

Limits of generalization, geography, change over time

Limits of generalization

The population you can generalize to is the population from the sampling frame

Geography

Often we use samples of smaller, subnational populations; findings from those studies do not hold for national or regional (e.g. survey of Maryland residents does not hold for all of US or North America)

Change over time

Populations also change over time. Considerable lags in data processing and release may take data out-dated (e.g. Results from the latest available survey collected in 2015 may no longer hold for 2021).

Item non-response

People choose not to answer that one question. Researchers have to decide whether to guess at their answer (“impute values”) OR not use that person’s whole survey?

if item non-response is orthogonal to (unrelated to) the variables in your analyses…

the consequences are not important

If item non-response is a function of the variables in your analyses…

the consequences can be large.

Ethical issues in surveys?

Researchers need to disclose who they are and what they’re doing so participant can consent and confidentiality of results is crucial.

Cross sectional survey

Data are collected at one point in time (Retrospective)

Longitudinal

Data are collected are more than point in time

Trend study

Repeated cross-sectional surveys with different samples; show change over time in a population

Panel study

Repeated surveys of same sample; shows change over time in individuals

Primary data

Data you collect

Secondary data

Data collected by others

Secondary survey analysis pros and cons

Pros: efficient, large samples, done by professionals. Cons: May not be appropriate for your question, no control over questions, confidentiality.

Weight

Disproportionate samples (in which some members of the population are more likely to be selected than others) provide biased population estimates. So we use sampling weights in our calculations to get unbiased estimates. An individual’s sampling weight is inversely proportional to the probability of being selected into the sample. Sometimes population estimates are further adjusted for known discrepancies between the population and sample characteristics, these are known as post-stratification weights.

Bivariate table

A table that displays the distribution of one variable across the categories of another variable

Column variable

A variable whose categories are the columns of a bivariate table

Row variable

A variable whose categories are the rows of a bivariate table

Cell

Intersection of a row and a column in a bivariate tableM