Empirical Research on International Environmental Policy: Designing Qualitative Case Studies

Introduction

• Empirical research on International Environmental Politics and Policy (IEP) relies heavily on qualitative case studies.
• Qualitative case studies are less developed methodologically compared to quantitative procedures.
• Analysts can improve research by:
  • Selecting cases carefully.
  • Drawing appropriate causal inferences.
  • Addressing the tension between specificity and generalizability.
• Developing theoretically meaningful propositions before case selection is crucial.
• Politically "hot" cases might not be suitable for answering theoretical questions.
• Drawing internally valid causal inferences requires:
  • Clearly defining and measuring dependent, independent, and control variables.
  • Selecting cases to control for exogenous variables.
• Focusing on "hard cases" and analyzing rival hypotheses strengthens causal inferences.
• Internal validity should be prioritized over external validity.

Research Strategies in IEP

• Social scientists use various research strategies to understand IEP:
  • Game theory.
  • Regression analysis.
  • Simulations and experiments.
  • Surveys.
  • Historical analysis.
  • Case studies.
• Researchers can use qualitative, quantitative, or mixed methods to study causal relationships in IEP.
• Distinctions between quantitative and qualitative methods:
  • Type of information used.
  • Procedures for processing information.
  • Number of cases analyzed.

Quantitative vs. Qualitative Methodologies

• Quantitative methodologists:
  • Seek to understand causal relationships using numerical data representing defined variables.
  • Analyze data via statistical procedures to compare cross-sectional or longitudinal observations.
  • Aim to identify strong, nonrandom correlations between independent and dependent variables.
  • Use statistical algorithms to isolate the correlation between independent and dependent variables while holding other variables constant.
• Qualitative methodologists:
  • Also aim to evaluate and generalize causal inferences.
  • Rely on explicitly defined variables but capture values in words.
  • Analyze data through nonstatistical techniques.
  • Common when few cases are available.
  • Evaluate causal relationships by holding other variables constant through case selection.

Focus of the Article

• Focuses on qualitative case studies using:
  • Qualitative information.
  • Few cases.
  • Nonstatistical procedures.
• Aims to derive causal inferences and generalize them.
• Addresses accusations of qualitative analysis being "soft," imprecise, or subjective.
• Highlights that both quantitative and qualitative methods require care in design and execution.
• Seeks to translate general principles of case study methodology into guidance for investigating causes of variation in international environmental politics.

Views on Case Studies

• Scholars hold different views on how case studies should be conducted and what they can accomplish.
• Case studies can be descriptive, prescriptive, predictive, constructivist, reflectivist, interpretivist, and positivist.
• Debates exist about the possibility of deriving causal inferences from case studies or generalizing them.
• The article aims to provide guidance for positivist inquiries seeking to draw causal inferences about the sources of variation in IEP.
• It outlines six practical steps for addressing obstacles to systematic, rigorous, and informative findings from causal qualitative research.
• Motivated by the belief that studies often make methodological errors, leading to weak or inaccurate conclusions.
• Illustrates the article with positive and hypothetical examples to avoid errors.
• Seeks to adapt, refine, and consolidate general principles of case study methodology for effective application in IEP, with attention to case selection.

Advantages of Qualitative Analysis

• Unlike quantitative techniques, causal qualitative analysis of a small number of cases facilitates investigation of:
  • Difficult-to-quantify variables (e.g., power, interests, leadership).
  • Empirically rare or previously ignored cases.
  • Innovative international environmental policy strategies.
  • Causal, rather than merely correlational, relationships.
• Allows for a more nuanced understanding of causal pathways, strengthening arguments for causal relationships.

Performance Criteria and Research Steps

• The literature on quantitative empirical research methods is now being matched by literature on qualitative research design.
• Five criteria characterize innovative contributions to understanding the sources of variation in IEP (Table 1):
  • Construct validity.
  • Internal validity.
  • External validity.
  • Reliability.
  • Progressive research.
• Meeting these criteria requires attention to six tasks (Table 2):
  • Identifying an important theoretical question.
  • Developing hypotheses and identifying variables.
  • Selecting cases.
  • Linking data to propositions.
  • Examining correlations and causal pathways.
  • Generalizing to other cases.
• These tasks are laid out in logical order but recognize that research is often iterative.
• Qualitative case study research should address each task explicitly to produce compelling, convincing, and contributing findings.
• Critical features of the six steps:
  • Selection of cases.
  • Drawing of causal inferences.
  • Tension between specificity and generalizability.

Identifying an Important Theoretical Question

• High-quality research starts with an important research question.
• Innovative causal case study research should:
  • Address existing theoretical debates.
  • Aim at causal relationships.
  • Attend to current policy concerns.
• Research should target unresolved debates, untested theoretical claims, or previously uninvestigated relationships.
• Requires a sophisticated understanding of existing theory and empirical patterns.
• Familiarity with existing theoretical literature allows the researcher to frame the research to target a scholarly community.
• The researcher should delineate how major schools of international relations theory have answered the central research issue.
• If the question has not been explicitly addressed, the analyst should deduce likely answers from core principles of broader theoretical arguments.
• The analyst should resist the temptation to view an issue as undertheorized and start from the assumption that general international relations theory applies equally well to IEP.
• Empirical "puzzles" provide a useful way to frame research questions by highlighting contradictions between theoretical predictions and observed outcomes.
• Analyzing a puzzle ensures that the findings will refute one theory and lend support to another.
• Identifying and clarifying existing theory provides opportunities for framing a specific, causal, and generalizable research question.
• Such a question must be specific enough to yield a determinate answer, explain observed variation, and be generalizable to other cases.

Modes of Causal Questions

• Causal questions fit into one of three possible modes:
  • Explaining a particular outcome or change in a specified dependent variable (DV).
  • Investigating one or relatively few causal relationships by specifying both the independent variables (IVs) and DVs.
  • Analyzing the influences of a specified IV.
• Innovative researchers frame their concern in broadly theoretical terms that clarify how the research results apply to other cases.
• A tension exists between specificity and generalizability.
• Achieving internal and external validity requires scholars to continually reevaluate whether their work answers both a generalizable theoretical question and a matching and specific empirical question.

Developing Hypotheses and Identifying Variables

• Drawing causal inferences requires hypotheses formulated to be shown wrong easily.
• This requires identifying IVs, control variables (CVs), and DVs, their potential values, and their theorized causal relationships.
• To make a hypothesis falsifiable, the researcher must categorize variables into at least two categories or values that she predicts correspond to different likelihoods of the DV.
• To use case studies to draw valid inferences for multiple causal questions requires that the researcher include more cases than IVs in her analysis.
• Drawing valid conclusions about a single causal claim contributes more than attempting to evaluate many causal claims with an indeterminate research design.
• The remedy involves ensuring the number of IVs is less than the number of cases, or by refocusing the study on the effects of particular explanatory variables rather than on the causes of a particular set of effects.

Selecting Cases

• Careful case selection grounded in existing theory lies at the heart of qualitative research that seeks to identify valid causal relationships.
• A common threat to such design in IEP studies arises from the desire to make analyses applicable to current policy debates.
• Initiating research by selecting a case because of substantive interest and letting this case dictate the research questions often produces results with neither theoretical nor policy value.
• Theoretically "hot" questions cannot be answered using politically hot cases.
• Instead of analyzing hot issues, scholars concerned with causal inferences should select historical cases that will provide internally valid results but that also will, by intention, have characteristics that allow those results to be generalized accurately to the cases of policy concern.
• Cases are often defined in terms of environmental issues or entities, but a case is defined as "a phenomenon for which we report and interpret only a single measure on any pertinent variable."
• To isolate one variable's influence from that of others, case study research selects cases so that the primary IV of interest varies but other IVs that might also influence the DV do not.
• Drawing causal inferences requires comparing at least one case per value of the IV of interest.
• Convincing causal case studies must exhibit variation in the value of the IV of interest and exhibit lack of variation in the value of other potentially explanatory IVs (i.e., CVs).
• Careful selection to limit variation in certain variables across the cases to be compared eliminates those variables as potential explanations of variation in a DV that would otherwise have confounded the analysis.
• Careful counterfactual analysis can prompt the scholar to collect additional evidence that can strengthen conclusions drawn from the study.
• A researcher can increase a study's internal and external validity by seeking out "hard cases" in which the values of many of the CVs are "distinctly unfavorable" to the hypothesis being tested.
• Cases to be used in multiple case studies across issue areas should be selected so that each case leads either to identical or similar results (literal replication) or to different results for predicted reasons (theoretical replication).
• Selecting cases requires some initial knowledge of possible cases, which can be identified through discussion, literature, or a pilot study.
• Researchers may have to scrap cases, add cases, or reanalyze existing cases based on changing the unit of analysis.

Linking Data to Propositions

• With cases selected, the researcher can now evaluate the hypothesized causal relationship against the evidence.
• Having clearly defined all variables and their values, the analyst must identify appropriate observable proxies or operationalizations.
• Variables must be defined and operationalized so that data relate to the theoretical construct as accurately as possible (construct validity).
• Appropriate, reliable, and observable indicators of complex conceptual variables often prove difficult to find.
• The in-depth analysis permitted by case study techniques offers the potential to observe a variety of indicators for each conceptual variable, thereby more accurately and reliably approximating the values of the conceptual variables of interest.
• Developing multiple proxies to triangulate on a single value of the conceptual variable increases the researcher's confidence in her assessment.
• Examining different proxies for the same theoretical construct may not only strengthen construct validity and reliability but also lead to new theoretical insights.
• Another important advantage of qualitative causal case studies is their ability to bring to light variables that were not initially thought to play an important causal role.
• Armed with clear definitions and operationalizations of the variables, the researcher can proceed to gather data on the value of each IV, CV, and DV for each case.

Examining Correlations and Causal Pathways

• Unlike quantitative researchers, qualitative case study researchers cannot turn to well-established and well-accepted procedures for analyzing the data they collect.
• Good causal analysis can be promoted by simple and systematic comparisons of predicted and observed values of the DV, evaluation of causal narratives, and evaluation of rival hypotheses.
• Predicting the values of the DV for each case based on the theory being tested and the observed values of the IVs and CVs in each case provides a structure for evaluating whether the empirical evidence conforms with theoretical expectations.
• A simple table summarizing the values of the variables in each case imposes beneficial rigor on causal case analyses.
• If the IVs of interest and DV correlate, then, as in regression analysis, the analyst needs to determine whether the covariation is evidence of a causal relationship or simply spurious covariation caused by other variables.
• If the researcher finds covariation between the IV of interest and the DV (with CVs held constant), she should go on to provide a plausible causal narrative of why and how the IV caused variation in the DV.
• Case studies have a major advantage over quantitative methods in this regard because they allow disaggregated and in-depth analysis of such "causal mechanisms" or "causal pathways."
• Detailed causal narratives or "process tracing" are more than mere storytelling.
• Examining the causal pathways that link an IV and DV in a particular case allows the researcher to show how and why these two variables covary rather than merely showing that they covary.
• The internal validity of the study can be further enhanced if the researcher explicitly considers alternative explanations and finds them "to be less consistent with the data and /or less supportable by available generalizations."
• The researcher should also give the benefit of the doubt to these rival explanations, especially when strong a priori reasons exist to accept the power of these explanations.
• The strategy of explicitly evaluating rival hypotheses constitutes the most effective defense against charges that the outcome observed in the cases was caused by variables other than those claimed by the researcher.

Generalizing to Other Cases

• As a final step, researchers undertaking causal qualitative case studies should evaluate the extent to which the propositions supported by the case evidence are relevant to other cases.
• Assessing the study's external validity—that is, whether the causal relationships present in the cases examined also operate in a larger population of cases—links the findings back to broader theoretical debates.
• Convincing policy makers to adopt (or avoid) a particular policy (or scholars to accept a particular theory) requires demonstrating that the conditions in the case studied look sufficiently like the conditions in the targeted policy area to support an expectation that the same causal influences will operate there.
• It might be argued that case studies provide a weaker foundation for generalization than quantitative analyses.
• In contrast, the design of most causal case studies implies that the correlation of the IV with the DV is known only for a single value of the CV—that is, the value of the CV for which the cases were held constant.
• Selecting cases to hold the value of certain variables constant increases the internal validity of causal inferences derived from the study but simultaneously limits the range of cases to which one can validly generalize.
• Analysts can mitigate this problem in three ways:
  • By conducting additional case studies in which cases are selected so that the CV has a different value.
  • The analyst can increase generalizability by selecting hard cases in which, in all the cases analyzed, the CV has a value at which the findings from the case studies will be most generalizable.
  • The analyst can design the case studies to build on past case studies and contribute to a collaborative research program.
• The researcher should remain cautious about generalizing too broadly.
• Assessing external validity requires determining whether the values at which CVs were held constant were crucial to the observed influence of the IV of interest on the DV.
• The case study findings may still be generalizable so long as the values of the CVs are common to most cases.

Conclusion

• Causal case studies are a widely used method of research in IEP.
• Yet, causal case study methodology remains underdeveloped and rarely appears on political science teaching curricula.
• In carrying out qualitative case studies, researchers face myriad potential pitfalls.
• Much of our knowledge of IEP is likely to be drawn from qualitative case studies—the more so because qualitative case studies have the distinct advantage over quantitative studies of helping us to understand in more detail the nature of causal relationships.
• The article has outlined six key steps that constitute a solid foundation for rigorous, qualitative, and causal case-study research on IEP.
• In this vein, the authors believe that good causal research begins by identifying an important and compelling research question.
• This question is then transformed into testable hypotheses by clearly delineating variables and their potential values.
• Equipped with scientific resources, the article has argued that case selection provides a strong and crucial, but all too often ignored, foundation for validly identifying causal relationships and generalizing those findings to other cases.
• Case study research requires explicit attention to existing theory so as to use identification of IVs and CVs to predict the DV.
• By using a simple table to examine the degree to which observed variation in the DV corresponds to variation predicted by theory and by using process tracing of the pathways by which those factors caused that variation, good case studies can deepen our understanding of the complicated social and political processes at work in international environmental politics.
• A researcher can and should conclude her research by clearly identifying the cases to which the findings can (and cannot) be appropriately generalized.
• Even though specificity and generalizability may need to be traded off, ensuring that findings are internally valid must take precedence over claiming they are externally valid.
• Because of these problems, those undertaking qualitative case studies will be best served by following Mark Twain's (1894) dictum to "put all your eggs in the one basket and- WATCH THAT BASKET" (p. 15).
• The procedures outlined above are designed to reduce, if not eliminate, the risks that are inherent in qualitative empirical approaches.
• The authors believe that researchers attentive to these six steps can improve the analytic rigor of their work and thereby contribute to the growing scholarly effort to understand how international environmental problems arise; when, why, and how they can be resolved; and how we can better manage our social, political, and economic behaviors to have fewer detrimental impacts on the environment.