Advanced Research Methods - Project 1 - 2

Research

Systematic investigation designed to develop or contribute to generalizable knowledge

Research must

Be conducted with the intention of drawing generalizable conclusions. Use a commonly accepted scientific method.

Overgeneralization

Concluding that what is observed in a few cases applies to all cases

Incorrect observation

Seeing or hearing something inaccurately and drawing conclusions from it

Selective observation

Choosing to look only at information that aligns with existing beliefs and ignoring disconfirming evidence

Illogical reasoning

Reaching conclusions on the basis of invalid assumptions

Resistance to change

Reluctance to update beliefs even in light of new information

Descriptive research

Describes fundamental aspects of a phenomenon, like how much and how many

Exploratory research

Investigates what is going on without expectations, often qualitative

Explanatory research

Seeks to identify causes and effects and implications of changes

Evaluation research

Determines the effects of a social program or policy

Quantitative

Typically descriptive, explanatory, or evaluation research. Data are numbers and attributes. Uses statistical analyses like experiments and surveys.

Qualitative

Typically exploratory. Data is text-based and seeks in-depth understanding. Methods include intensive interviews and participant observation.

Mixed Methods

Intentionally combines both approaches for richer data and triangulation. Relatively newer tradition since the 1980s.

Research Question

Start with an area of interest, narrow to the most workable and important question. Evaluate by: Feasibility (time, resources, data access), Social importance, Scientific relevance

(grounded in prior literature). Research question helps determine methodological approach

Policy motivations for research

agencies need data to identify needs, formulate policy, allocate resources

Academic motivations for research

fundamental questions about how/why/where things occur

Personal motivations for research

 keen personal interests that drive scholars to study specific topics

General Social Survey

National, cross-sectional survey of persons 18+ living in U.S. households. Representative of non-institutionalized adults in the U.S. (multistage area probability sample. Started in 1972; was annual (with few exceptions); now every 2 years since 1994. Wide range of questions — some core items repeated every wave. Since 1994: split-sample design for special topics (half receive one set, half receive another. Data collected by NORC; available through ICPSR (UM is a member institution).

Inflict no harm:

Physical, psychological, emotional; threats to self-image and self-esteem. If harm may occur, protections must be in place for anticipated and unanticipated harms

Confidentiality:

Collect identifying info but do not disseminate it; destroy identifying info as soon as possible

Anonymity:

Do not collect identifying information at all

Voluntary participation: 

No coercion; remuneration must be reasonable, not coercive

Informed consent/assent:

Participants must be informed of nature of study, risks, and possible benefits. Consent from adults; assent from minors

Deception:

Only when necessary to conduct the study and when risk is minimal; must debrief participants as soon as possible

Institutional Review Board

Oversee the ethical conduct of research involving human (and animal) subjects. Research MUST be approved PRIOR to initiation. Monitor any negative outcomes during the study.

Ethical Obligations to the Scientific Community

Acknowledge the limitations of your study — no perfect research exists. Honestly report all methods and techniques used. Do not overstate confidence in results given methodological limitations. Disseminate negative and null findings, not just positive ones. Do not plagiarize; do not fabricate data or results. Be honest about academic record and qualifications; only engage in work consistent with your training.

Ontological Quantitative Assumption

Reality exists outside and independent of the researcher; is measurable (consistent with Durkheim's social facts)

Ontological Assumption

 ("What is real?")

Ontological Qualitative Assumption

Reality is constructed by people; subjective; multiple realities exist

Epistemological Assumption

("Researcher's relationship to the researched?")

Quantitative Epistemological Assumption

Researcher is independent; bias controlled through sampling/randomization; objectivity is the goal

Qualitative Epistemological Assumption

Researcher interacts with participants; seeks to minimize distance

Axiological Assumption 

("Role of values?")

Quantitative Axiological Assumption

Research is value-free and unbiased; report facts from data

Qualitative Axiological Assumption

Research is value-laden; researcher reveals values and biases

Methodological Assumption

("Research process?")

Quantitative Methodological Assumption

Deductive; cause-and-effect; static design; context-free results; seeks generalization; reliability and validity assessed

Qualitative: 

 Inductive; emergent and changing design; context-specific results; triangulation and verification for accuracy

Theory

a systematic set of logical, interrelated statements intended to explain some aspect of social life. Theories are conceptual — they cannot be tested directly. Researchers use observable markers (variables) of concepts.

Wallace’s Research Wheel

Begin with theory. Derive hypotheses: statements of expected relationships. Make observations: collect and analyze data. Form empirical generalizations: link results back to population and theory. Modify theory and repeat the cycle

Serendipity pattern

Unanticipated findings become the occasion for modification of or

development of new theory

Recasting of theory

New data exert pressure for elaborating or refining existing theory

Refocusing of theoretic interest

New research methods create pressure for new foci of theoretical interest

Clarification of concepts

Clear, precise concepts are needed before they can be measured

and studied empirically

Unit of observation

Level at which data are collected

Attribute

An aspect/characteristic of a single unit of observation

Variable

An attribute that varies across multiple members of a population 

Constant

An attribute that does NOT vary across a population

Informants

Those who can provide information on behalf of a unit of observation

Nominal

Discrete classification only (e.g., race, religion). No rank order.

Ordinal:

Discrete classification + rank order (e.g., education level: less than HS, HS, some college, BA+). Intervals between ranks are unknown.

Interval:

 Discrete classification + rank order + known, equal metric between values (e.g., temperature in Celsius). No true zero.

Ratio:

All of the above + a true, meaningful zero point indicating complete absence of the attribute (e.g., income, age, number of arrests).

Dichotomous variable

Two categories 

Polytomous variable

Three or more categories

Dependent Variable (DV):

The outcome variable — what we are trying to explain

Independent Variable (IV):

The predictor or explanatory variable — what we think causes the

DV

Positive relationship

X and Y move in the same direction (both increase or both decrease)

Negative relationship

X and Y move in opposite directions (as X increases, Y decreases)

Bivariate

Relationship between exactly 2 variables

Multivariate

Relationship between 3 or more variables — essential for understanding indirect effects and controlling for spuriousness

Direct effect

X impacts Y directly with no intermediary variable

Indirect effect

X impacts Y through its influence on a third variable (must be multivariate)

Deterministic Relationship

X always/never causes Y — extremely rare in social science

Probabilistic Relationship

X sometimes causes Y — the standard in social science research. We examine patterns/associations, not absolute causation.

Three requirements for establishing caulsality

 Correlation: There must be an empirical relationship between X and Y (a general pattern, not necessarily in every case). Temporal order: X must precede Y in time (after cannot cause before). Non-spuriousness: The relationship cannot be explained away by another, prior variable.

Recursive Relationship

One-directional relationship (X → Y) — most common and preferred

Non-recursive relationship

Feedback loop (X → Y and Y → X) — problematic for causal analysis

Exogenous variables

Variables with no prior causes in the model (e.g., X1)

Endogenous variables

Variables influenced by other variables in the model (can be IV or DV)

Total Effect

Direct + Indirect + Spurious effects

Spuriousness

an apparent relationship between two variables is explained wholly or largely by another variable that is prior to both. Spuriousness is a matter of degree.

Type 1 Specification Error

Failure to include all relevant variables — leads to incorrect results due to omitted variable bias

Type 2 Specification Error

Inclusion of theoretically irrelevant variables — violates parsimony (include what is needed, exclude what is not)

Type 3 Specification Error

Incorrect functional form — assuming relationships are linear/additive when they are not (e.g., non-linear relationships, interaction effects)

Conceptualization

Defining what you mean by a concept

Operationalization

Specifying operational definitions — precisely how the concept will be measured

Propositions

"Theoretical hypotheses" — statements of expected (causal) relationships between CONCEPTS. Use concept terminology.

Hypotheses

 Statements of expected (causal) relationships involving VARIABLES.

Interchangeability of Indicators

If multiple variables operationalize the same concept, they should produce a consistent pattern of relationships with other variables

Dimensionality

Consider whether a concept involves one or more than one dimension

Reliability

the extent to which a measuring procedure yields the same results on repeated trials. Reliable measures produce consistent results. Can have reliable data that is NOT valid.

Consistency across administrations: 

Test-retest reliability; Alternative forms reliability

Internal Consistency

Cronbach's alpha (ranges 0–1; higher = more reliable); Split-halves reliability (correlate results from random half of items with other half)

Validity

the extent to which a measure actually measures the concept it is intended to measure. More of a theoretical issue. Measures cannot be valid without being reliable, but reliable measures can be invalid.

Face validity:

Does the measure appear to measure the concept on its face? 

External validity: 

 Do results generalize from the sample to the population? Across other

populations and contexts?

Criterion-related validity:

Empirical validity — correlation between a measure and a criterion

variable. Concurrent validity (measured at same time) and predictive validity (predicting future

standing)

Content validity: 

Does the measure reflect the full domain/content of the concept? Especially important for multidimensional concepts

Construct validity: 

 Is the measure related to other measures consistent with theoretically derived hypotheses? Requires: (1) specify theoretical relationship; (2) assess relationship between items and latent construct; (3) confirm latent variables correlate as hypothesized

Alpha

 (α): Typically set at .05. The probability of a Type I error (rejecting a true null).

Beta

The probability of a Type II error (accepting a false null).

Power 

(1−β): The long-run probability of correctly rejecting a false null hypothesis. Typically set

at .80 (β = .20).

The Four Determinants of Power

Power, effect size, sample size, and alpha level are interrelated — each is determined by the other three. Power also depends on the type of statistical analysis used.

Effect size: 

The magnitude of the relationship in the population. Bigger effect size = fewer cases needed. Smaller effect size = larger sample needed.

Sample size (N):

Larger N = greater power. Must be calculated prior to data collection.

Alpha level:

Stricter alpha (e.g., .01) reduces power; more lenient alpha increases power.

Type of analysis:

Different statistical tests have different power. Cohen (1992) details formulas for different analyses.

Must consider power

at the DESIGN phase of a study, before data collection begins