The Comparative Method — Comprehensive Notes

Scope and Purpose of the Comparative Method

• Comparison as a fundamental tool of analysis in political science

  • Sharpens description and aids concept-formation by highlighting similarities and contrasts across cases

  • Used to test hypotheses, discover new hypotheses inductively, and to build theory

• Forms of comparison vary widely (statistical analysis, experimental research, historical studies)

• Within the social sciences, the term “comparative method” often means the systematic analysis of a small number of cases, i.e., a small-N analysis

• This chapter surveys alternative perspectives on small-N analysis that emerged roughly over the past two decades

• Main focus is comparative politics and international studies, but applicability is broader

• Decision to study few cases is driven by: the nature of phenomena and how they are conceptualized

  • Productive topics for small-N study include revolutions, post-communist regimes, certain forms of urban politics

  • Small-N focus is common because there are relatively few instances that exhibit the attributes of interest

• Competing view: some analysts believe political phenomena are best understood through careful study of a small number of cases

• Rise of comparative historical analysis strengthened legitimacy of small-N in recent years

• A central theme: refinements in small-N methods broaden the toolkit; the most fruitful approach is eclectic, drawing on diverse techniques

• The central tension: how to adjudicate rival explanations with few cases while managing many variables

• Key equation-y idea (conceptual): small-N involves $N ext{ (number of cases)} ext{ is small}$, yet aims to adjudicate rival explanations with limited data

Synopsis of Lijphart's View on the Comparative Method

• Lijphart defines the comparative method as the analysis of a small number of cases, entailing at least two observations, yet too few to permit conventional statistical analysis

• He compares four methods for theory testing: experimental, statistical, case-study, and comparative method

• Evaluation criteria (two aims):
  1) How well each method tests theory by adjudicating among rival explanations
  2) How difficult it is to obtain the data needed for each method

• Experimental method: strong control to eliminate rival explanations, but often infeasible for political topics

• Statistical method: uses statistical control but requires large, reliable datasets which are often unavailable

• Case-study method: accessible with modest resources; enables data generation for a case but offers limited opportunities to test hypotheses systematically

• Lijphart’s typology of case studies (based on their contribution to theory):
  1) atheoretical case studies
  2) interpretative case studies (theory illuminated a case)
  3) hypothesis-generating case studies
  4) theory-confirming case studies
  5) theory-infirming case studies (raise doubts about a theory)
  6) deviant case analyses (refine theory via a case that departs from predictions)

• Lijphart argues that certain types of case studies can be implicit parts of the comparative method, especially when hypotheses are tested within a comparative framework

• Overall assessment: the comparative method has an intermediate status

  • Weaker than experimental or statistical methods for testing hypotheses due to less control and the problem of many variables, small-N

  • Stronger than standalone case studies for evaluating hypotheses

• Data requirements: comparative method requires more data than case studies but less than experimental or large-N statistical work

• Practical takeaway: the comparative method is most appropriate when resources are modest; it can be the first stage of research, with statistics as the second stage when possible

• Guidance: when possible, prefer statistical (or experimental) methods over the comparative method; but in resource-constrained settings, intensive small-N analysis can be more productive

• Lijphart's proposed solutions to the many-variables, small-N problem (1971, 685):
  1) Increase the number of cases when feasible
  2) Use comparable cases to control for extraneous variables
  3) Reduce the number of variables via data reduction or stronger theoretical parsimony

• Conceptual takeaway: Lijphart provides a compact framework for thinking about how the comparative method relates to other methods and how to address its core dilemmas

The Three Methods in Brief (Compared to the Comparative Method)

• Experimental Method

  • Merit: strong ability to eliminate rival explanations through controlled experiments

  • Inherent problem: many political topics cannot be studied experimentally; control is often impractical or impossible

• Statistical Method

  • Merit: tests rival explanations via statistical control across cases

  • Inherent problem: data collection can be prohibitive; large-N data sets may be unavailable or of dubious quality

• Case-Study Method

  • Merit: feasible with limited resources; rich data on a single case

  • Inherent problem: systematic hypothesis testing across many cases is limited

• Comparative Method (as per Lijphart)

  • Merit: systematic cross-case comparison with a small number of cases can yield useful theory testing and development

  • Inherent problem: weaker adjudication among rival hypotheses; still valuable when data and resources constrain other methods

Innovations in Small-N Analysis (Two Decades After Lijphart)

• The modern landscape combines multiple approaches, often blending insights from experimental, statistical, and case-study traditions

• The aim is to broaden the toolkit for small-N researchers and improve causal inference in limited-N settings

• Figure reference: innovations span goals of comparison, justification for small-N focus, and the problem of many variables, small-N

Innovations in the Goals of Comparison

• Theda Skocpol and Margaret Somers (1980) argue for three distinct but connected goals of comparative analysis:
  1) Systematic examination of covariation among cases for causal analysis (causal inference)
  2) Demonstrating that a model or set of concepts illuminates many cases (theory development/validation, not necessarily a formal test)
  3) Highlighting how different cases produce different outcomes (contrast of contexts; interpretive understanding)

• These three goals are interconnected in a broader research cycle: parallel demonstration, hypothesis testing, and contrast of contexts

• The cycle supports a more flexible, iterative approach to comparative work, beyond a single exclusive focus on hypothesis testing

• Implication: comparative work should be understood within a research community that values multiple goals and their interaction

Justifications for a Small-N Focus (Rethinking Lijphart)

• Lijphart’s rationale (resources) appears modest in light of newer debates

• An alternative defense draws on configurative approaches:

  • Verba (1967) argued for a disciplined configurative approach to theory building and hypothesis testing in case studies

  • Almond and Genco (1977) reinforced the disciplined configurative stance, integrating interpretive understanding with systematic hypothesis testing

• Sartori (1970; 1984; 1991; 1993) warned about concept misformation and conceptual stretch when applying broad concepts across diverse cases; careful fine slicing (finer slicing) of concepts is crucial

• Cognitive science insights (Lakoff, 1987) challenge classical categorization: meanings arise from implicit cognitive models and exemplar cases, not just strict boundaries

• The call for more work on concept formation remains important to avoid overstretched or ill-defined concepts in comparative work

• The rise of comparative historical analysis (Bendix, Moore, Rokkan, Tilly, Skocpol, Ragin, etc.) has strengthened the legitimacy of small-N with historical-contextual depth

• The overall message: there is value in a disciplined, eclectic small-N approach that combines case-oriented insight with causal analysis and theory testing

Debates on Solutions to the Problem of Many Variables, Small-N

1) Increase the Number of Cases

• Early belief in a move toward large-N quantitative cross-national analysis faded in some subfields

• Reasons for the persistence of small-N:

  • Valuing close contextualized analysis and interpretive depth

  • Difficulty of developing reliable, comparable cross-national data for many contexts, especially outside advanced industrial settings

  • Quality concerns: some cross-national quantitative work used weak concepts or weak causal tests

• Some scholars still defend and advance quantitative cross-national work (e.g., Jackman 1985; Lijphart 1990)

• New statistical techniques make meaningful small-N quantitative work more viable (roughly 10–15 cases can be informative)

• Practical stance: increasing N to around ten to fifteen cases can be productive when data and resources permit

2) Focus on Comparable (Matched) Cases vs. Most Different Systems Design

• Most Similar Systems Design (Lijphart 1975; matches on many non-central variables to hold contexts constant)

• Most Different Systems Design (Przeworski and Teune 1970; analyzes highly diverse cases to distill common elements by tracing similar processes of change)

• Proponents of matched cases emphasize controlled comparison to isolate effects; critics warn of overdetermination (too many rival explanations remain)

• The debate connects to broader methodological choices about how to balance similarity and difference in case selection

• A synthesis example: Collier and Collier (1991) start with a roughly matched set of eight Latin American countries and then pairwise compare within the set to reveal parallels and differences

• Within-case analysis (pattern matching, process tracing) helps to increase the analytic N by adding internal comparisons and strengthening causal inference

• Selection bias risk if cases are chosen on the dependent variable; adding within-case variation can mitigate this risk

• Area studies remain valuable; cross-area studies benefit from deep area knowledge, while cross-area generalizations gain from strong case knowledge as foundation

3) Reduce the Number of Variables (Parsimonious Explanations)

• Focusing on a smaller set of explanatory factors improves tractability in small-N studies

• Rational choice approaches offer a parsimonious framework (e.g., Geddes 1991) to model incentives and reforms with fewer variables

• Example: Geddes 1991 analyzes how electoral and party-system configurations influence legislators’ reform incentives

• Ongoing need for conceptual clarity and parsimony to avoid conceptual stretch when reducing variables

Innovations from Work in Other Methods (Bringing New Tools to Small-N)

Experimental Method and Quasi-Experiments

• Campbell and Stanley (1963) introduced quasi-experimental designs for observational settings

• Interrupted time-series design: examine long series of observations before and after an intervention to avoid misattributing abrupt changes to a single observation

• Connecticut speeding crackdown (Campbell & Ross, 1968) as a classic quasi-experimental exemplar

• The broader literature on evaluation research spread these ideas to political development analyses

• Achen (1986) notes the difficulties of quasi-experiments when randomization is absent; issues include selection bias

• Two-stage statistical approaches may be required to control for selection effects, which can be challenging with small-N data

• Practical takeaway: quasi-experimental logic offers valuable warnings about causal claims when data do not meet randomized criteria

Innovations in Statistics for Small-N

• David Freedman (1987, 1991) criticizes some standard statistical practices in social science research, urging more careful research design and validity checks

• New techniques enable meaningful analysis with small-N datasets:

  • Resampling methods: bootstrap and jackknife to generate replication-based inference when distributions are unclear

  • Robust and resistant statistics (Hampel et al. 1987; Hartwig 1979; Mosteller & Tukey 1977): less sensitive to outliers

  • Regression diagnostics (Bollen & Jackman 1985; Jackman 1987): identify influential cases and assess their impact on results

• Case study of the corporatism and growth debate (Lange-Garrett-Jackman-Hicks-Patterson): regression interaction terms, model extensions, and data scrutiny via diagnostics

  • Example: Lange & Garrett (1985) propose an interaction term between union strength and left-party strength affecting growth; Jackman (1987) uses regression diagnostics to identify influential cases and reanalyze

  • The debate then includes expanding the model and collecting more data

• Key lesson: even with small-N, careful statistical thinking and diagnostics can yield insightful results when used responsibly

• Jackson (1992) demonstrates methods for handling coefficients that vary across cases and addressing heterogeneous effects

Innovations in the Case-Study Method

• Campbell (1975) shifts the view on case studies: they are not only hypothesis-generating but can be central to falsifying hypotheses through pattern matching across multiple facets of a case

• Pattern matching (Campbell): test whether case implications derived from a hypothesis are realized in a given case by examining multiple aspects of the case

• The ex post facto problem (patterns tested after data collection) can be mitigated by pattern matching across multiple implications and contexts

• Eckstein (1975): argues for the testing value of case studies, including the use of a critical case where strong expectations will be met if the hypothesis is correct

• George & McKeown (1985) integrate two tools for testing hypotheses in case studies:

  • Congruence procedure: assess whether case values for independent and dependent variables align with the predicted pattern

  • Process tracing: analyze unfolding temporal sequences within a case to assess whether causal mechanisms plausibly connect the case observations to the hypothesized process

• The combination of congruence and process tracing provides a structured approach to within-case testing

• Yin (1984) Case Study Research provides a formalized methodological framework for designing and conducting case studies

• The World Politics 1989 special issue on deterrence and rational choice highlights tensions between generic, parsimonious explanations and the detailed, theory-specific needs of particular theories

  • Achen & Snidal (1989) argue that many deterrence-case studies fail to address core theoretical ideas and note selection bias (over-representation of deterrence failures)

• Ongoing debate: how to balance generic, parsimonious analysis with the complexities of individual cases; how to link methodological concerns with theory-specific needs

Within-Case and Across-Case Testing: Pattern Matching and Process Tracing

• Within-case comparisons are critical for increasing the effective N beyond the raw N across cases (Stanley Lieberson’s point about the role of internal comparisons)

• Pattern matching and process tracing complement cross-case comparison by testing hypotheses against rich case-level data

• Rule of thumb: single-case tests should not be overemphasized; internal comparisons help avoid selection bias and strengthen causal inference

• Area studies have a dual role: deep, cumulative knowledge of a region strengthens cross-area analysis while contributing to generalizable insights

Conclusion: Three Major Analytic Alternatives Persist

• Three viable paths persist for addressing questions with small-N data:
  1) Case studies with enhanced within-case testing and interpretive depth; increasingly legitimized by interpretive social science and comparative-historical work
  2) Quantitative techniques with a small number of cases; supported by newer statistics and diagnostics to add causal insight
  3) Systematic comparison of a small number of cases intended for causal analysis, as originally advocated by Lijphart; strengthened by comparative historical analysis and robust cross-case synthesis

• The field supports cross-fertilization: cross-fertilization between qualitative and quantitative methods strengthens overall causal inference and theory validation

• A productive research agenda combines: rigorous data collection, thoughtful case selection, careful theoretical parsimony, and strong methodological awareness

Implications for Graduate Training and Practice

• Training should cover both quantitative and qualitative methods, enabling students to evaluate and employ a range of approaches as appropriate

• Students should gain exposure to the philosophy of science and the logic of inquiry to make informed methodological choices

• The goal is eclectic practice: learners should be equipped to blend case studies, small-N quantitative work, and comparative historical analysis to advance theory

• The broader objective is to facilitate communication between area specialists and generalists, leveraging cross-fertilization to strengthen both qualitative and quantitative analyses

Key Points, Concepts, and Terms (Conceptual Frame)

• Small-N analysis: analysis of a small number of cases (denoted $N$) with at least two observations, insufficient for conventional statistics

• Lijphart’s three-method comparison framework: experimental, statistical, case-study; with comparative method occupying an intermediate space

• Three core problems in small-N: few cases, many variables; data limitations; and the challenge of drawing causal inferences

• Typology of case studies (Lijphart): atheoretical, interpretive, hypothesis-generating, theory-confirming, theory-infirming, deviant cases

• Skocpol & Somers’ three goals of comparison: covariation for causal analysis; model illumination; contrast of contexts

• Research cycle in comparison: parallel demonstration, hypothesis testing, and contrast of contexts

• Concept misformation (Sartori): danger of stretching concepts across diverse cases; importance of refining concepts to the cases

• Finer slicing vs. classical categorization (Sartori vs. Lakoff): better resolution of categories and boundaries to prevent misapplication

• Pattern matching (Campbell): testing multiple implications of a hypothesis across a case

• Process tracing (George & McKeown): detailed, time-sequenced investigation within a case to establish causal mechanisms

• Critical case, most-similar vs. most-different designs (Campbell vs. Przeworski & Teune): strategic case selection to maximize informational yield

• Area studies: deep regional knowledge as a foundation for broader cross-area analysis

• Quasi-experiments and interrupted time series: adapting experimental logic to observational data; caution about internal and external validity

• Regression diagnostics and robust statistics: strategies to guard against influential cases and outliers in small-N contexts

• Interaction terms and heterogeneous effects: modern approaches to capturing context-dependent causal effects in small-N data

Illustrative Examples and Implications

• Connecticut speeding crackdown (1950s): quasi-experimental design illustrating internal and external validity concerns and the value of long time-series data for policy impact evaluation

• Democratic transitions literature (O'Donnell, Schmitter, Whitehead; 1986): illustrates how a broad, cross-context set of cases can generate generalizable insights while acknowledging context-specific pathways

• Rational choice models in Latin American reform (Geddes, 1991): demonstrates data reduction and modeling to address small-N challenges while maintaining explanatory clarity

• Classic works in comparative-historical analysis (Skocpol; Moore; Rokkan; Tilly): show how long-span historical data and cross-national comparison can yield robust causal inferences with small Ns

Connections to Foundational Principles and Real-World Relevance

• Links to foundational methodological debates: control, parsimony, and the balance between depth and breadth

• Relevance to real-world research: policymakers and researchers often face data limitations; eclectic, methodologically aware approaches enable robust analysis under constraints

• Ethical and philosophical angles: recognizing selection bias, avoiding overstated generalizations, and maintaining transparent between-case reasoning and data handling

• Practical exam relevance: understand the trade-offs among experimental, statistical, and case-study methods; articulate when small-N is advantageous; explain the role of pattern matching and process tracing; discuss the benefits and limits of comparative-historical approaches

Summary Takeaways for Exam Preparation

• The comparative method is a flexible, eclectic toolkit for studying few cases, aiming to test hypotheses, develop theory, and interpret variations across contexts

• Lijphart provides a practical framework for small-N work, emphasizing the balance between data requirements and analytical progress, with three solutions to the many-variables problem: increase cases, match comparable cases, and reduce variables

• Since Lijphart, innovations from statistics, quasi-experiments, and case studies have expanded the small-N toolkit, enabling more robust causal inferences with limited data

• Three major analytic pathways persist: (1) case-study with within-case testing, (2) small-N quantitative studies with careful diagnostics, and (3) systematic cross-case comparison for causal analysis

• Graduate training should cultivate fluency across methods and an understanding of conceptual clarity, measurement validity, and causal inference in both qualitative and quantitative contexts

Key Citations (selected)

• Lijphart, Arend. 1971. Comparative Politics and Comparative Method. American Political Science Review, 65:682-693.

• Skocpol, Theda, and Margaret Somers. 1980. The Uses of Comparative History in Macrosocial Inquiry. Comparative Studies in Society and History 22:174-97.

• Przeworski, Adam, and Henry Teune. 1970. The Logic of Comparative Social Inquiry. New York: John Wiley.

• Campbell, Donald T., and H. Laurence Ross. 1968. The Connecticut Crackdown on Speeding: Time Series Data in Quasi-Experimental Analysis. Law and Society Review 3:33-53.

• Campbell, Donald T., and Julian C. Stanley. 1963. Experimental and Quasi-Experimental Designs for Research. Chicago: Rand McNally.

• Achen, Christopher H. 1986. The Statistical Analysis of Quasi-Experiments. Berkeley: University of California Press.

• Freemen, David A. 1987, 1991. Works on statistics and model-building; bootstrap/jackknife; robust statistics; regression diagnostics.

• Sartori, Giovanni. 1970. Concept Misformation in Comparative Politics. American Political Science Review 64:1033-53.

• Geertz, Clifford. 1973. Thick Description: Toward an Interpretive Theory of Culture. In The Interpretation of Cultures.

• Yin, Robert K. 1984. Case Study Research: Design and Methods. Beverly Hills: Sage.

• Colloquial references to more recent debates: Lange & Garrett; Jackman; Ragin; Geddes; Collier & Collier; Collier & Norden; Lakoff (cognitive models of categorization)

Aim/Objectives

• To survey alternative perspectives and innovations in small-N analysis that have emerged over the past two decades.

• To broaden the toolkit for small-N researchers and improve causal inference in settings with limited data.

• To demonstrate the value of a disciplined, eclectic small-N approach that integrates case-oriented insight with causal analysis and theory testing.

• To primarily focus on comparative politics and international studies, with broader applicability noted.

Main Argument

• The most fruitful approach to small-N analysis is eclectic, drawing on diverse techniques and acknowledging multiple goals beyond just hypothesis testing.

• Refinements in small-N methods have broadened the toolkit, making robust causal inference possible even with limited cases and many variables.

• Cross-fertilization between qualitative and quantitative methods is crucial for strengthening overall causal inference and theory validation.

• A productive research agenda requires rigorous data collection, thoughtful case selection, careful theoretical parsimony, and strong methodological awareness.

Method

• Review and Synthesis: The paper surveys and synthesizes various approaches to small-N analysis, including:

  • Lijphart's Comparative Method: Analysis of a small number of cases (at least two, too few for conventional statistics).

  • Comparison of Methods: Evaluation of experimental, statistical, case-study, and comparative methods for theory testing.

  • Innovations in Small-N Analysis: Discusses strategies like increasing the number of cases, matched case designs (Most Similar/Different Systems), and reducing variables.

  • Integration of Tools: Incorporates concepts from experimental (quasi-experiments, interrupted time-series), statistical (resampling methods, robust statistics, regression diagnostics), and case-study methods (pattern matching, process tracing) to enhance small-N research.

Conclusions

• Three major analytic pathways remain viable for addressing questions with small-N data:

  1. Case studies with enhanced within-case testing and interpretive depth.

  2. Quantitative techniques applied to a small number of cases, supported by newer statistics and diagnostics.

  3. Systematic comparison of a small number of cases for causal analysis, reinforced by comparative historical analysis.

• Cross-fertilization between qualitative and quantitative methods is essential for stronger causal inference and theory validation.

• Effective research necessitates rigorous data collection, thoughtful case selection, careful theoretical parsimony, and strong methodological awareness.

Main Contributions

• Provides a comprehensive survey of advancements in small-N analysis over the past two decades, updating Lijphart's foundational work.

• Expands the understanding of the goals of comparative analysis, moving beyond a sole focus on hypothesis testing to include model illumination and contrast of contexts.

• Offers new justifications for a small-N focus, emphasizing configurative approaches, concept formation, and the legitimacy of comparative historical analysis.

• Integrates innovative techniques from statistics and quasi-experimental designs into the small-N toolkit.

• Highlights advanced case-study methods like pattern matching and process tracing as crucial for within-case testing and strengthening causal inference.

• Advocates for an eclectic methodological approach that blends diverse techniques for robust analysis under data constraints.

Relevance

• Methodological Debates: Connects to foundational debates concerning control, parsimony, and the balance between depth and breadth in social science research.

• Real-World Research: Provides a practical guide for researchers and policymakers who frequently encounter data limitations, enabling robust analysis under these constraints.

• Ethical and Philosophical Angles: Underscores the importance of recognizing selection bias, avoiding overstated generalizations, and maintaining transparent reasoning and data handling.

• Graduate Training: Informs graduate curricula by advocating for comprehensive training in both quantitative and qualitative methods, fostering methodological fluency and the ability to make informed choices.