Introduction to Research Design and Credibility in Social Science
Research Design and Academic Modeling
Scientific models serve as frameworks to describe social phenomena and provide hypothetical scenarios to test theoretical expectations.
The scientific method in social science is a process of predictive modeling. It begins with a hunch or informal observation about the world, leading to a prediction.
Findings are framed through the principle of falsifiability: researchers either reject a theory or fail to reject it.
Scientific studies do not "prove" a theory to be true, as alternative explanations that the researcher has not considered may also predict the observed data.
The presence of empirical backing, such as systematic evidence from studies like Mamalo , strengthens academic arguments by providing a large-scale counterpart to anecdotal or hypothetical scenarios.
Variables in Causal Analysis
Causal questions are typically framed as "Do" or "Does" questions (e.g., "Does the presence of state judicial elections affect a judge's behavior?"). These differ from "Why" or "What" questions, which often require qualitative or long-form narrative answers rather than statistical correlations.
X Variable (Independent Variable): This represents the cause, treatment, exposure variable, explanatory variable, predictor, or feature. In criminal justice research, the term "treatment" is frequently used, a practice inherited from epidemiology (e.g., assigning a placebo vs. a medicinal aspirin).
Y Variable (Dependent Variable): This represents the outcome, effect, response, or target. This is the phenomenon the researcher seeks to explain or change, such as the rate of police traffic stops or judicial outcomes.
Variation is essential for both variables. An independent variable must have multiple levels (e.g., different types of license plates: regular, university-affiliated, or none) to allow for comparisons.
Components of a Comprehensive Research Design
Research Question: This consists of a broad phenomenon of interest (e.g., "Does police reform work?") and a specific, answerable question (e.g., "Did the Los Angeles Police Department's pretextual stop policy decrease the actual number of pretextual stops?").
Motivation: Research may be motivated normatively (based on subjective beliefs about what should happen, such as a desire to identify racial disparities) or through a puzzle (a contradiction in observed reality, such as why different socioeconomic groups view police differently despite reporting similar treatment).
Past Work (Prior Literature): New research must be positioned within existing scholarship. This involves identifying whether the new study extends, qualifies, or contradicts previous findings. Resources like "Annual Review" provide high-level summaries of existing literatures.
Argument: A formal thesis statement should begin with "I argue that…" and represent a logical, falsifiable chain of reasoning. Flowcharts are often used to illustrate how one factor leads to another ().
Observable Implications: If a model is an accurate representation of reality, specific things should be observable in the world. For example, to test the legal model of judicial behavior, one would watch for instances where a judge follows a precedent they are known to personally dislike.
Scope, Design, and Methods:
Scope: The specific population or setting where the claim applies (e.g., a study of students cannot automatically be generalized to or students).
Design: The strategy used to collect evidence, such as an experiment or an observational analysis of downloaded data.
Methods: The specific procedures for drawing inferences from evidence.
Credibility in Social Science and Causal Empiricism
Causal empiricism involves rigorous attention to the validity of claims, requiring researchers to account for why an association between variables exists.
A raw correlation (e.g., a difference in police support between those with and without positive contact) can be misleading if it is influenced by lurking variables.
Stratification (breaking data into strata): Researchers can control for confounding variables by comparing similar groups. For instance, breaking down data by education levels (High School, Some College, or more) may show that the effect of police contact on funding support is smaller than it first appeared once education is accounted for.
A central assumption in many observational studies is that people within the same stratum are the same in all other relevant ways, which is often a significant leap that requires further testing against other variables like political ideology or income.
Common Critiques of Social Science Research
Selection Critique:
Sample Selection: Issues regarding who ends up in the study (e.g., only those who choose to respond to an email survey).
Treatment Selection: The social phenomenon that determines which individuals receive the treatment (e.g., certain types of people may be more likely to have positive interactions with police due to their own behavior or background).
Measurement Critique: This questions whether the survey or tool is appropriate. It includes concerns about phrasing (e.g., "positive contact" is subjective) and how the order of questions might influence responses.
Omitted Variables: These are other causes of the outcome that also determine selection into the treatment, forcing a false association between and .
Reverse Causality (Endogeneity): The possibility that actually causes (e.g., people who already support police funding are more likely to seek out or perceive interactions as positive).
Questions & Discussion
Question: Should citations be included when describing a model?
Response: Yes, papers and written answers are improved by citing specific systematic evidence, such as Mamalo regarding the militarization of police, alongside the explanation of the model.
Question: Do we need to know verbatim definitions for the midterm?
Response: Verbatim definitions are not required. The goal is to be able to explain concepts in professional academic English.
Question: Can notes be brought to the midterm exam?
Response: One single-sided, standard-size index card or note card is permitted. It must be handwritten to aid memory retention. No printed cards or sharing of cards is allowed.
Question: What is the required length for the midterm essay?
Response: There is no strict length requirement. The focus should be on answering the prompt professionally, ideally one paragraph per idea. An essay is expected to be roughly to in a blue book, but the quality of the argument and the avoidance of repetition are more important than page count.
Question: If a police officer pulls over people from specifically because they dislike that school, does that bias the study on license plates?
Response: Yes, that would bias the inference. It demonstrates why researchers use the language "fail to reject" rather than "prove." A result might be consistent with a theory, but it could also be explained by a localized bias or a different mechanism. This is why future research often seeks to replicate findings in different contexts to see if they hold.
Question: What if a study finds that police give tickets to brand new cars but not old cars because they think owners of new cars can pay and poor people need a break?
Response: This highlights a different theoretical mechanism (economic vs. psychological bias). A literature review could articulate these competing expectations from different academic traditions (e.g., economics vs. psychology). If the study shows the opposite of what was expected, that becomes the basis for discussing new mechanisms in the conclusion.
Administrative Deadlines and Logistics
Midterm Examination: Covers the first of the course, including models of judicial and police decision-making and agency problems. Students must bring a blue book, which can be purchased for approximately at the library.
Research Question Approval: All final project research questions must be approved by the instructors by Friday at .
Final Project Structure: Consists of a research design memo and a take-home short answer. Both are due Sunday of week .
One-on-Two Meetings: Students will sign up for meetings during week (Wednesday or Thursday) to receive feedback on their design memos. Grades will be based in part on how well students address this feedback.
Grading Constraints: Because this is a summer session, all final grades must be posted by the Tuesday following the end of week .