9.1 Introduction

In recent years, the “replication crisis,” or the finding that many attempts to replicate prominent published studies have failed to reproduce their original results, has roiled the medical, social science, public policy, and development research communities (Reference IoannidisIoannidis 2005).Footnote ¹ This has led to efforts to change both procedures and cultures in carrying out and publishing research, including a de-emphasis of p-values in statistical research, preregistration of studies using experimental designs or observational statistics, and, in some journals, preacceptance of studies based on their designs rather than their results.

Although many of the projects whose results could not be replicated were experimental studies, one response in the program evaluation community has been to increase the emphasis on experiments. Done well, these research designs, including field experiments and natural experiments as well as lab and survey experiments, remain powerful tools in program evaluation. Yet experiments impose demanding methodological requirements (Reference CookCook 2018; Reference Deaton and CartwrightDeaton and Cartwright 2018), they face challenges of external validity, and in some policy domains they are not practical for fiscal or ethical reasons. In addition, evaluators are often called upon to evaluate programs that were not set up as experiments, including programs instituted quickly to address pressing needs.

Thus, a second response in the evaluation community has been increased interest in “process tracing,” a method of causal inference that is applicable to single observational case studies (Reference Bamanyaki and HolvoetBamanyaki and Holvoet 2016; Reference Barnett and MunslowBarnett and Munslow 2014; Reference Befani and MayneBefani and Mayne 2014; Reference Befani and Stedman-BryceBefani and Stedman-Bryce 2017; Reference Busetti and DenteBusetti and Dente 2017; Reference Mendoza and WoolcockMendoza and Woolcock 2014; Reference Punton and WellePunton and Welle 2015; Reference Schmitt and BeachSchmitt and Beach 2015; Reference Stern, Stame and MayneStern et al. 2012; Reference Wauters and BeachWauters and Beach 2018). Process tracing has been common in political science for decades and has been the subject of recent methodological innovations, most notably the explicit use of Bayesian logic in making inferences about the alternative explanations for the outcomes of cases. Process tracing and program evaluation, or contribution analysis, have much in common, as they both involve causal inference on alternative explanations for the outcome of a single case (although process tracing can be combined with case comparisons as well). Evaluators are often interested in whether one particular explanation – the implicit or explicit theory of change behind a program – accounts for the outcome. Yet they still need to consider whether exogenous nonprogram factors (such as macroeconomic developments) account for the outcome, whether the program generated the outcome through some process other than the theory of change, and whether the program had additional or unintended consequences, either good or bad. Process tracing can address these questions, and it is also useful in assessing the validity of the assumptions behind natural, field, and lab experiments.

This chapter outlines the logic of process tracing and the ways in which it can be useful in program evaluation. It begins with a short discussion of the philosophy of science underlying process tracing and a definition of process tracing. It then turns to the role of process tracing in single case studies and in checking the underlying assumptions of experiments, field experiments, and natural experiments. Next, the chapter provides practical advice on process tracing for causal inference in individual cases and discusses the special considerations that arise in the use of process tracing in program evaluation. Finally, the chapter outlines an important recent development in process tracing methods: the explicit and transparent application of Bayesian logic to process tracing. It concludes that explicit Bayesian process tracing holds promise, but not yet proof, of improving the use of process tracing in causal inference and program evaluation.

9.2 The Philosophy of Science of Causal Mechanisms and Process Tracing

The increased interest in process tracing across the social and policy sciences is related to the turn in the philosophy of science over the last few decades toward a focus on causal mechanisms as the locus of causal explanation. Earlier, philosophers hoped that either “laws” or observed relations of statistical conditional dependence – analogous to what the philosopher David Hume called “constant conjunction” – would provide satisfactory accounts of causation and causal inference. The attempt to explain outcomes by reference to “laws” or “covering laws” foundered, however, when its advocates, including Carl Hempel, failed to come up with a justification or warrant for laws themselves (Reference Salmon and SalmonSalmon 1998, 69). In addition, Hempel’s approach, known as the “Deductive-Nomological (D-N) Model,” had difficulty distinguishing between causal and accidental regularities. In a common example, a barometer’s readings move up and down with changes in the weather, but they do not cause the weather. Rather, changes in air pressure, which are measured by a barometer, combine with changes in temperature and other factors (topography, humidity, ocean currents, etc.) to cause the weather. But the D-N model has trouble distinguishing between a barometer and a causal explanation of the weather, as the barometer readings exhibit strong law-like correlations with the weather.

In an effort to address these problems, philosopher of science Wesley Salmon attempted to work out a defensible schema of explanation based on conditional dependence, or, in Salmon’s terms, “statistical relevance.” After encountering several paradoxes and dead-ends in this effort, he ultimately concluded that “statistical relevance relations, in and of themselves, have no explanatory force. They have significance for scientific explanation only insofar as they provide evidence for causal relations … causal explanation, I argued, must appeal to such mechanisms as causal propagation and causal interactions, which are not explicated in statistical terms” (Reference SalmonSalmon 2006, 166).

Many philosophers and social and other scientists thus turned to exploring the role of causal mechanisms and causal processes in causal explanation and the roles of different research methods (experiments, observational statistics, case studies, etc.) in uncovering evidence about the ways in which causal mechanisms work and the contexts in which they do and do not operate. Within philosophy, the discussion of causal mechanisms has generally gone under the label of “scientific realism” (related but not necessarily identical approaches include “causal realism” and “critical realism”). This is the school of thought that Ray Pawson, Nick Tilley, and others in the evaluation community have drawn upon in their discussions of “realist evaluation” (Reference Astbury and LeeuwAstbury and Leeuw 2010; Reference Dalkin, Greenhalgh and JonesDalkin et al. 2015; Reference Pawson and TilleyPawson and Tilley 1997).

A detailed analysis of scientific realism and causal mechanisms, and of debates surrounding their definitions, is beyond the scope of the present chapter, but a brief summary will suffice. Realism argues that there is an ontological world independent of the mind of the observer or scientist, and causal mechanisms ultimately reside in that ontological world. Scientists have theories about how causal mechanisms work, and, to the extent that those theories are accurate, they can explain outcomes. In one widely cited formulation, causal mechanisms are “entities and activities, organized such that they are productive of regular changes from start or set-up to finish or termination conditions” (Reference Machamer, Darden and CraverMachamer, Darden, and Craver 2000, 3). In another definition that also emphasizes a kind of regularity, mechanisms are processes that cannot be “turned off” through an intervention (Reference Waldner and KincaidWaldner 2012). Fire happens, for example, whenever there is combustible material, oxygen, and a sufficient ignition temperature; we can intervene on the presence of oxygen or materials or the temperature, but we cannot intervene on the mechanism of fire happening when the requisite materials and conditions exist.

Mechanisms are in the world, and theories about mechanisms are cognitive or social constructs in our heads. Scientists make inferences about the accuracy and explanatory power of theories about mechanisms by outlining the observable implications of these theories and testing them against evidence. In frequentist studies, the observable implications of theorized mechanisms lie at the population level, such as the correlations one would expect to find if a theory were true. In contrast, process tracing gets closer to mechanisms where they actually operate: in individual cases. The operation and interaction of causal mechanisms is realized in specific cases and contexts, and scientists and evaluators are interested in building theoretical understandings of the conditions under which mechanisms are activated or deactivated and the ways in which they interact with other mechanisms.

In studying individual cases, process tracers focus not just on the values of the independent and dependent variables, but on diagnostic evidence of sequences and processes that lie in the temporal space between the independent variables and the observed outcome. Process tracing uses this evidence to make inferences about which theories most likely offer true explanations of a case’s outcome, sometimes called “inference to the best explanation.” Process tracers continually ask “What should be true about the sequence of events between the independent variables and the dependent variable if a theory is a true explanation of the outcome of a case?” In the social sciences, this often takes the form of asking “Who should have conveyed what information to whom, when, and with what effect at each stage in the process if this explanation is true?” Diagnostic evidence, ideally, is information that allows inferences about which processes are in operation, but that does not itself represent an additional variable that independently affects the operation or outcome of these processes. Diagnostic evidence, in other words, is not an “intervening variable” in a process, as the term “variable” implies an independent entity with its own potential causal effects.

9.3 Definition of Process Tracing

Process tracing is “the analysis of evidence on processes, sequences, and conjunctures of events within a case for the purposes of either developing or testing hypotheses about causal mechanisms that might causally explain the case” (Reference Bennett, Checkel, Bennett and CheckelBennett and Checkel 2015, 7).Footnote ² Process tracing is a within-case form of analysis: that is, it seeks to explain the outcomes of individual cases (sometimes called “historical explanation” or “token explanation”). At the same time, process tracing can be combined with cross-case comparisons or other methods. Researchers can use process tracing, for example, to assess whether differences between most-similar cases might account for these cases’ different outcomes. The theoretical explanations of case outcomes assessed through process tracing can be about individual mechanisms or processes, or combinations of mechanisms and processes. They can include structural mechanisms, agent-based mechanisms, or any combinations thereof.

A key difference between process tracing and frequentist statistical analysis is that statistical analysis faces the “ecological inference” problem: even if a statistical correlation correctly captures an average causal effect for a population, it does not necessarily explain the outcome for any particular case in that population. Process tracing, in contrast, focuses directly on the causal explanation of individual cases. It may or may not uncover strong evidence leading to a confident explanation of a case, but it does aspire to develop directly the strongest explanation of the case that the evidence allows. Rather than facing an ecological inference problem, process tracing explanations, even when strong, face challenges regarding the external validity or generalizability of findings from individual cases. As Chapter 4 argues, the challenges of generalizing the results of case studies, while real, are often misunderstood. The explanation of an individual case can indeed prove generalizable: a new understanding of how a causal mechanism works, derived from the study of an individual case, can give strong clues about the scope conditions in which we should expect that mechanism to operate.

While process tracing is most often focused on the explanation of case study outcomes, the logic of process tracing can also be used in interrogating the validity of the strong assumptions necessary for experiments, field experiments, and natural experiments. In lab experiments, in addition to carrying out various balance tests on the treatment and control groups, researchers can use process tracing to check the procedures through which individuals were assigned to one group or the other, to assess the ways in which and reasons for which individuals opted to drop out of one group or the other, and to check on the possible presence of unmeasured confounders. Similarly, in field and natural experiments, where there is less control over assignment to treatment and control groups, researchers can use process tracing to assess whether the actual assignment or election into treatment and control groups was “as if random,” and to evaluate evidence on whether the hypothesized process does indeed account for differences between the outcomes of the treatment and control groups (Reference Dunning, Bennett and CheckelDunning 2015).

Process tracing is much like detective work: the researcher is seeking an explanation of one case, and they can use both deductive and inductive inferences to find the best explanation. Deductively, the researcher starts with some “suspects” – the theories that have typically been applied to the outcome of interest. In program evaluation, this includes the theory of change explicitly or implicitly adopted by a program’s designers and managers, but it also includes alternative explanations that relate to variables exogenous to the theory of change, such as macroeconomic trends, demographic change, local and national political developments, wars, natural disasters, etc. The researcher then looks for evidence on the deductively derived observable implications of each potential theoretical explanation of the outcome of the case. Just as a detective can reason forward from suspects and backward from a crime to connect possible causes and consequences, researchers can trace processes in both directions. A researcher can trace sequences forward from the independent variables, asking whether each caused the next step in the hypothesized chain leading to the outcome, and the step after that, and so on to the outcome. She or he can also trace backward from the outcome, asking about the most proximate step in the process that caused the outcome, and the step prior to that, back to the independent variables.

Deductively derived implications of a theory are one type of “clue,” but researchers also gather other kinds of evidence or clues that they stumble upon inductively as they investigate or “soak and poke” in their cases. Inductively discovered evidence might point to an existing social science theory that the researcher had not identified as a possible explanation of the case, or it may lead to the development of an entirely new theory as a potential explanation of the case. It is possible that an inductively identified piece of evidence, even evidence for an entirely new theory or explanation, can be so strong – so uniquely consistent with one explanation and so inconsistent with all other explanations – that this theory could become the most likely explanation for the outcome even without further corroboration. This cuts against the common but erroneous intuition that a theory developed from a case can never be considered to have undergone a severe test from the evidence that led to the theory. Anyone who has done their own amateur home or car repairs knows the experience of finding physical evidence that not only suggests but makes highly likely a heretofore untheorized explanation for why a switch, appliance, or part is not working.

In addition, our confidence in a newly derived or newly added potential explanation of a case can be strengthened if the explanation entails additional observable implications within the same case that are then corroborated by additional evidence. This contravenes the frequent claim that one cannot develop a theory from a case and test it against the same case. We can develop a theory from a case and test it against different evidence from the same case that is independent of the evidence that gave rise to the theory. It would be illogical, for example, for a doctor to diagnose a rare illness in a patient based on an unexpected test result, and then insist on testing the diagnosis on a different patient, rather than on an additional diagnostic test in the first patient.

9.4 Practical Advice on Traditional Process Tracing

The general approach of process tracing is fairly intuitive as it follows a kind of inferential process that has been around as long as humankind. Yet despite its seeming simplicity and familiarity, researchers do not always do process tracing well, and, as the final section of this chapter argues, even trained researchers make common mistakes in employing the Bayesian logic that underlies process tracing. So how can we do process tracing well? Elsewhere I have elaborated with my co-author Jeffrey Checkel on ten best practices for being a good traditional process-tracing detective; here, I introduce these practices briefly and elaborate on the considerations of each that are most relevant to program evaluation (Reference BennettBennett and Checkel 2013, 20–31). In the final section of this chapter I address how to carry out the more formal Bayesian variant of process tracing.

9.5 Program Evaluation Process Tracing versus Social Science Process Tracing

There is one key difference between program evaluation process tracing and social science process tracing, and it generates both advantages and challenges for program evaluators. This is the fact that the experts who design policy interventions have the opportunity to outline in advance diagnostic indicators that will later provide evidence on whether a program is working as its theory of change suggests. Moreover, officials can require that program implementers begin gathering and reporting evidence on these indicators from the inception of the program or even the preprogram baseline. If the indicators are well designed, and if they also include data on alternative causal processes that might affect program outcomes, this greatly eases the task of program evaluation. Social scientists, in contrast, usually have to devise their own process tracing tests and gather the relevant evidence themselves after the events under study have already taken place.

Predesignation of program indicators can present challenges as well, however. First, indicators may be poorly designed and fail to provide strong evidence on the mechanisms through which the theory of change is expected to operate. Program outcomes can be difficult to conceptualize and measure, which can create a tendency to rely on measuring inputs or outputs instead of outcomes (Reference CastroCastro 2011; Reference Markiewicz and PatrickMarkiewicz and Patrick 2016; Reference Van der KnaapVan der Knaap 2016). Diagnostic process tracing evidence is not the same as measures of outputs or outcomes, as it focuses on hypothesized causal mechanisms and processes, but it can overlap with output measures. There can also be a temptation to focus on diagnostic measures that are easy to measure rather than those that provide strong evidence for causal inference.

Second, there is a risk that program managers and other stakeholders will “game” the measurement and reporting of indicators to slant them toward their desired evaluation results. It can be difficult to devise diagnostic measures that provide strong evidence on the causes of program outcomes and that are not also susceptible to gaming. Essentially, this requires devising diagnostic measures that program implementers cannot achieve unless they actually are faithfully carrying out the program in accordance with its theory of change. This can lead to another problem, however: if diagnostic measures are too demanding and detailed, or if program implementers think (rightly or wrongly) that the theory of change is imperfect and that their experience and skills (or changed circumstances) give them better ideas on how to achieve the program’s goals, these program managers will face unpleasant choices between following micromanaging guidelines that they think are inappropriate or departing from the prescribed practices and measures. This raises the classic dilemmas concerning how much authority and flexibility to delegate in principal–agent relations, how to monitor agents through management information systems, and whether and how to allow for changes in the middle of program implementation (Reference HonigHonig 2018). While there is no perfect solution to these dilemmas, consulting stakeholders and program managers on the design of appropriate diagnostic measures and putting in place procedures and decision-making processes for modification or adaptation of these measures can minimize the trade-offs between too much and too little delegation and oversight (Reference Gooding, Makwinja and NyirendaGooding et al. 2018).

Perhaps a more common challenge, however, arises when program designers had an under-specified theory of change or gave insufficient attention to developing and gathering evidence on indicators that would make later process tracing and program evaluation easy. Even when a theory of change is well specified, evaluators need to assess its coherence and consider alternative explanations that program managers may not have considered or on which they did not gather evidence. In this regard, program evaluators are often in a position similar to that of social scientists who design and gather evidence on alternative explanations only after the events of interest have taken place.

9.6 Bayesian Logic and Process Tracing

The best practices outlined earlier address the “traditional” process tracing that characterizes almost all published research and completed program evaluations to date. In the last few years, however, methodologists have begun to explore the possibility of applying more explicitly and formally the Bayesian logic that underlies process tracing. There are as yet few applications of this approach to empirical research, and there are strong pragmatic reasons why full formal Bayesian analysis of evidence from case studies is not appropriate in most research settings. Still, it is useful to understand the formal Bayesian logic that informs more informal process tracing practices, as this can lead to better implementation of these less formal practices. In addition, it may be useful to apply more formal Bayesian analysis to a few of the most important pieces of evidence in a study even if it is unduly cumbersome to do so for most of the evidence. While a full discussion of the Bayesian logic of process tracing is beyond the scope of this chapter, the brief outline that follows provides an introduction to the topic.Footnote ⁷

In Bayesian analysis, probability is conceived of as the degree of belief or confidence that we place in alternative explanations. This is quite different from the standard frequentist statistical conception of probability as representing the likelihood that a sample is or is not representative of a population. In Bayesian analysis of individual case studies, the analyst starts with a “prior,” or an initial guess regarding the likelihood that alternative explanations are true regarding the outcome of the case. The analyst uses the logic of the explanations, or of their underlying theories, to estimate how likely particular kinds of evidence are in the possible worlds represented by each explanation. The analyst then uses the laws of conditional probability to translate the likelihood of evidence given alternative explanations into the likelihood of alternative explanations given the evidence. This new, updated estimate of the likelihood that alternative explanations are true is called the “posterior” probability, or simply the posterior.Footnote ⁸

Bayesianism provides a formal language for discussing the relative strength or probative value of different pieces of evidence. We already have an informal language for this: “smoking gun” evidence strongly supports one explanation, but the absence of such evidence does not necessarily reduce confidence in that explanation. Passing a “hoop test” is asymmetrical in the other direction: an explanation is strongly undermined if it fails a hoop test, but we do not necessarily greatly increase our confidence in an explanation that passes a hoop test. These informal examples are points on a continuum: the “likelihood” of evidence taking on a certain value if a theory or an explanation is true can range from 0 to 1, and when we compare the likelihood of evidence under one explanation to its likelihood under an alternative – that is, when we divide the likelihoods – this ranges from 0 to infinity. The more likely evidence is under one explanation, and the less likely it is under the alternatives, the more strongly the discovery of that evidence affirms the one explanation it fits. It is the relative likelihood of the evidence under the alternative explanations, or the “likelihood ratio,” that matters, not the absolute likelihood that the evidence or data will take on a certain value if one explanation is true.Footnote ⁹

Bayesian inference, however, is only as good as the information that informs the analysis, which raises the obvious question: How do we estimate the priors and likelihoods? The prior, or our initial guess on the likelihood that a particular theory correctly explains the outcome of a case, in principle represents all of our “background knowledge,” or all of our conclusions and intuitions from previous research and experience. In some situations, such as when we have mountains of data, we can use well-informed priors, just as life insurance companies do when they use the ample data at their disposal to estimate life expectancies given a person’s age, health habits, and health indicators. Most of the time in social science settings, however, we lack a strong evidentiary basis for estimating priors. One option here is to use uninformed priors – that is, to give each alternative explanation an equal prior (such as a prior of 1/3 if there are three candidate explanations). Another option is to try the analysis with different priors to see how sensitive the conclusions are to the choice of the prior; if the evidence is strong, the estimate of the prior might not matter much to the estimate of the posterior (Bayesians call this the “washing out of priors”). A third approach that case study methodologists are beginning to assess is to “crowd source” estimates of priors, whether among subject matter experts or nonexperts.

Estimating likelihoods of evidence is challenging as well. This requires “inhabiting the world” of each hypothesis – that is, assuming that the hypothesis is true and then assessing the likelihood of a piece of evidence given the truth of the hypothesis. Estimating likelihood ratios requires performing this task for multiple hypotheses. On the other hand, it can be easier to assess the relative likelihood of evidence – to ask which of two hypotheses makes the evidence more likely, and even to estimate the ratio of these likelihoods – than to estimate the absolute likelihood of evidence given each hypothesis. As with estimating priors, researchers can try crowd-sourcing estimates of likelihoods.

A third challenge is arranging the alternative explanations, as Bayesian inference requires, in such ways that they are mutually exclusive and exhaustive. This includes explanations that combine several interacting theoretical variables or causal mechanisms, such as agents, institutions, norms, etc. In principle, this is possible for any group of hypotheses. To take a simple example, a criminal investigator might divide the explanations for a murder into four possibilities: the murder could have been committed by suspect A alone, by suspect B alone, by both A and B colluding together, or by neither A nor B. The next step is a bit more complex: the investigator has to think of the likelihood of different pieces of evidence under all these possible explanations, and, ideally, to find evidence that strongly discriminates among the explanations. This can be difficult for murder investigations: the detective has to ask what evidence would point to collusion that would not also be consistent with A or B acting alone. It is arguably even more challenging for social science researchers who are evaluating various combinations of structural, normative, macroeconomic, managerial, and other factors that can contribute to the success or failure of development programs.

A final difficulty with formal Bayesian analysis is that the calculations it requires become tedious and lengthy to write up and to read even for a small number of pieces of evidence and alternative explanations, and much more so for multiple pieces of evidence and explanations. For this reason, even the methodologists who have begun to explore formal Bayesian process tracing argue against trying to implement it fully for all the evidence (Reference Fairfield and CharmanFairfield and Charman 2017).

Still, it can be useful to do formal Bayesian analysis on one or a few of the pieces of evidence that a researcher judges to be most powerful in discriminating among alternative explanations, as this can make the analysis more transparent. Specifically, understanding the Bayesian logic of process tracing can contribute to better process tracing practices in at least four ways. First, Bayesian logic provides a clear philosophical warrant for much of the practical advice methodologists have given regarding traditional process tracing, including the ten best practices of traditional process tracing discussed earlier. One reason to initially consider a wide range of alternative explanations, for example, is that failing to consider a viable explanation can bias the estimates of the likelihoods, and thus the posterior estimates, of all the explanations the analyst does consider. Bayesianism also gives a clear explication of what constitutes strong evidence, of why diverse and independent evidence is important, of the trade-offs involved in stopping too soon or too late in gathering and analyzing evidence, and of why we should never be 100 percent confident in any explanation.

Second, Bayesianism leads to counterintuitive insights. Evidence that is consistent with an explanation, for example, can actually make that explanation less likely to be true if the same evidence is even more consistent with an alternative explanation. Also, numerous pieces of weak evidence (or what might be called “circumstantial evidence” in a court), if they all or mostly point in the same direction, can jointly constitute strong evidence that considerably changes our confidence in alternative explanations.

Third, formal Bayesian analysis, even if it is done only on a few key pieces of evidence, provides a transparent form of inference that allows researchers and their readers or critics to identify exactly why their inferences diverge when they disagree on how to update their confidence in explanations in light of the evidence. Researchers and their readers can disagree about their priors, the likelihood of evidence under alternative explanations, and the interpretation or measurement of the evidence itself. Leaving estimates and interpretation of each of these ambiguous obscures where authors and readers agree and disagree. Making judgments on each of these clear, in contrast, can prompt researchers and their critics to reveal the background information that underlies their judgments, which can narrow areas of disagreement.

The fourth, and perhaps strongest, rationale for learning Bayesian analysis is that it illuminates the logic that traditional process tracers have used informally all along in order to make causal inferences form individual cases, and it can help them to use it better. Research on the psychology of decision-making indicates that people often make mistakes when they try to be intuitive Bayesians or first attempt formal Bayesian analysis (Reference Casscells, Schoenberger and GrayboysCasscells, Schoenberger, and Grayboys 1978). Other research shows that deeper training in Bayesian analysis can help improve forecasting (Reference Tetlock and GardnerTetlock and Gardner 2015). Additional research indicates that a few simple practices consistent with Bayesian process tracing, such as actively considering alternative explanations, can help debias judgments (Reference Hirt and MarkmanHirt and Markman 1995).

9.7 Conclusion

Process tracing and program evaluation, especially forms of evaluation that emphasize contribution analysis, have much in common. Both involve inferences on alternative explanations of outcomes of cases. It is not accidental that the evaluation community has taken a growing interest in process tracing, or that process tracing methodologists have become interested in program evaluation. The best practices developed in traditional social science process tracing are applicable, with modest adaptations, to the task of program evaluation. The biggest difference is that in contrast to researchers doing process tracing in the social sciences, program evaluators may have the opportunity to designate in advance, and to require reporting upon, diagnostic indicators about alternative processes as well as measures of inputs, outputs, and outcomes. This can make later evaluation easier, but it can also introduce potential distortions and biases as program managers and stakeholders might “game the system” once they know what measures will be tracked. Program designers and evaluators need to be creative and flexible in designing indicators that are useful in subsequent program evaluations, that cannot be achieved without also achieving the desired results at which a program aims, and that do not become a straightjacket on program managers when modifications to a program can better achieve its goals.

Program evaluators can benefit as well from exploring the emerging literature on formal Bayesian process tracing. This literature clarifies the logic behind traditional process tracing methods, and it is beginning to explore and outline new practices, such as crowd-sourcing of estimates of priors and likelihood ratios, that might further strengthen process tracing. Although formally analyzing the weight of every piece of evidence is impractical, it can be useful to formally assess a few of the strongest pieces of evidence. This can contribute to more logically consistent and analytically transparent assessments of alternative explanations of program outcomes.

10.1 Introduction

Case studies can contribute valuably to the study of development generally and to the implementation of development policy in particular. Case studies are uniquely well situated to identify and disentangle causal complexity and to interrogate the role of contextual factors in shaping outcomes, among other strengths. In this chapter, I focus on the potential insights that can be derived from the study of a particular kind of case – the deviant case – and, more specifically, on anomalies that exceed expectations, or “positive deviant” cases. I argue that the study of positive deviance can offer two distinct types of benefits for development policy. The first is methodological: Building on the literature on case selection in the social sciences, I emphasize the value of deviant cases for hypothesis generation and for the analysis of causal heterogeneity. The second potential contribution is less technical. Deviant cases can play an important inspirational role, signaling to practitioners, policy-makers, and local development actors that improvement is possible, even in resource-constrained environments. At the same time, the celebration of positive deviant cases must proceed with caution, not only because idiosyncratic factors may deter the replication of their experiences to other units or contexts, but also because other actors operating in the same sectors or communities may feel undermined if they are implicitly judged vis-à-vis similar institutions or actors deemed to be more successful.

In Section 10.2, I elaborate on the concept of positive deviance and highlight the potential value of positive deviant cases for development policy. The discussion focuses on both the methodological strengths and limits of positive outliers and on the value and risks of using such cases as sources of inspiration for local actors and development practitioners. The section identifies ways of selecting positive deviant cases in systematic and less systematic ways. In the subsequent section of the chapter, I illustrate the value of positive deviant cases for several projects on the quality of social service delivery in the Middle East and North Africa to which I have contributed. Section 10.3 summarizes the main arguments and raises additional issues related to the pros and cons of using positive deviance as a way to formulate and propel beneficial reforms in development policy.

10.2 What Can Development Policy Learn From Positive Deviance?

The analysis of high-performing outliers is a promising approach for the formulation and implementation of development policy on both technical and nontechnical grounds. The literature on case studies in the social sciences highlights the methodological value-added of positive deviant cases in research designs. An additional, nontechnical merit of the approach emphasizes the role of such cases in encouraging greater performance from other actors and institutions operating in the same field. In this section, I define positive deviance and review these distinct justifications for the study of positive deviance.

10.3 Examples: What Explains High-Performance Service Delivery in Jordan and Morocco?

In this section, I aim to flesh out the potential contributions and limitations of case studies of apparent positive deviance with reference to two studies carried out in preparation for a World Bank report highlighting local success stories in service delivery in the Middle East and North Africa (Reference Belkâab and CammettBelkâab and Cammett, 2014; Reference Brixi, Lust and WoolcockBrixi et al., 2015; Reference Rabie, Lust, Clark, Cammett and LinnemannRabie et al., 2014). The first entailed a case study of high-performing public health centers in Jordan and evolved in two stages. The first part of the study was based on in-depth, qualitative analyses of the drivers of high performance at the facility level. The second stage of the research built on the findings of the case study to design a quantitative study based on a nationally representative sample. The Jordanian case therefore shows how a case study of positive deviance can fit into a larger mixed methods research design. The second study also focused on the drivers of high-quality health care through a case study of high-performing public health centers in provincial and semirural areas of Morocco. By describing the methods and findings of case studies of positive deviance in Jordan and Morocco, and detailing how they have contributed to larger-scale projects in the two countries, I hope to illustrate the value and limitations of this approach for development policy.

10.4 Conclusion: Can Positive Deviance Inform Development Policy?

In this chapter, I have argued that positive deviant cases have the potential to contribute in important ways to the formulation and execution of development policy in at least two overarching ways. First, building on insights from the methodological literature on case selection in the social sciences, I reiterate the value of deviant cases – whether positive or negative – for identifying previously overlooked factors that may help to explain a phenomenon and distinct causal pathways that account for causal heterogeneity. At the same time, it is critical to add the caution that potential causal factors identified through small-N research based on selection on the dependent variable must be assessed more broadly before informing development policy and practice. In particular, follow-up studies, whether qualitative or quantitative, which select on the hypothesized explanatory variable would be a valuable complementary component of a larger research strategy.

Second, deviant cases, and especially positive deviant cases, can serve an important inspirational function for multiple audiences, whether development practitioners, policy-makers, or organizational staff and citizens who experience and shape development policy on a quotidian basis. By signaling that beneficial reforms or outcomes are possible, even in a resource-constrained environment, positive deviant cases can provide hope, empowering others to pursue similar approaches or outcomes. At the same time, the depiction and dissemination of lessons from positive deviant cases must proceed with caution in order to avoid inducing resentment on the part of similar actors or organizations that were not singled out for commendation.

Ultimately, the potential value of the study of positive deviant cases for development policy is contingent on a number of factors, only some of which are in the control of the researchers. Given data availability and consensus over the indicators that enable the classification of high performance, researchers can and should follow systematic principles of case selection by identifying and choosing positive outliers in carefully specified models that build on the state of knowledge on the outcome of interest.

In practice, however, many real-world factors, which are largely out of the control of researchers, may impede adherence to best practices in case selection. First and foremost, officials or other gatekeepers must be willing to share data or facilitate the collection of data that enables the identification of positive deviance. Once the cases have been identified and the lessons extracted from their experiences, other conditions must be in place to ensure that these examples spur reforms or constructive outcomes. In particular, other actors or organizations that have not distinguished themselves must be ready to experiment with new approaches or systems. If they are resistant to adopting lessons generated from more successful peer institutions, then the dissemination of best practices based on positive deviance will be hindered. Furthermore, government officials and other stakeholders must exhibit the political will to promote the dissemination of such practices.

The very fact that the lessons of positive deviance are derived from outlier cases raises the question of whether their experiences are idiosyncratic or can be generalized. For example, case studies of high-performing health centers in Jordan and Morocco underscored the value of strong and effective leadership at the facility level for high-quality service provision. If leaders are born and not made, or at least not easily cultivated, then this finding is less useful for generating policy prescriptions. In this case, the best that development practitioners and government officials can hope for is to recruit and retain employees with demonstrated and appropriate leadership skills. In most cases, however, policy lessons will not be so specific to individual personality traits or other idiosyncratic factors. Furthermore, factors that seem difficult to replicate may be less idiosyncratic than they appear; as studies of leadership attest (Reference Nohria and KuranaNohria and Kurana, 2011), even this quality can be fostered.

To the extent that context matters, the same practices often do not have the same effects in different places, potentially limiting the generalizability of the lessons of positive deviance for development policy. Fortunately, the case study method itself may compensate for this potential drawback. Because case studies enable deep attention to context, it is possible to identify the factors that facilitated success and to adapt the lessons to other contexts or to identify similar contexts where the lessons may apply more readily. Attention to context also avoids a cookie-cutter or “best practices” approach to development policy by pointing to the ways in which local factors may moderate or completely alter the effects of a particular policy prescription and must be harnessed to design appropriate policy interventions.

11.1 Introduction

Analytic narratives (Reference Bates, Greif, Levi, Rosenthal and WeingastBates et al., 1998, Reference Bates, Greif, Levi, Rosenthal and Weingast2000; Reference Levi and MayntzLevi, 2002, Reference Levi, Shapiro, Smith and Masoud2004) involve selecting a problem or puzzle, then building a model to explicate the logic of an explanation for the puzzle or problem, often in the context of a unique case. The method involves several steps. First, the use of narrative to elucidate the principal players, their preferences, the key decision points and possible choices, and the rules of game, all in a textured and sequenced account. Second, building a model of the sequence of interaction, including predicted outcomes. And, third, the evaluation of the model through comparative statics and the testable implications the model generates. The analytic narrative approach is most useful to scholars who seek to evaluate the strength of parsimonious causal mechanisms in the context of a specific and often unique case. The requirement of explicit formal theorizing (or at least theory that could be formalized) compels scholars to make causal statements and to identify a small number of variables as central to understanding the case.

Case studies abound in the study of development. A weakness of case studies per se is that there typically exist multiple ways to interpret a given case. How are we to know which interpretation makes most sense? What gives us confidence in the particular interpretation offered? This problem is particularly difficult where the uniqueness of the situation precludes the collection of a data set that encompasses multiple cases. Many scholars augment their case study with a model. The model adds some discipline to the account. For example, observed choices must be consistent with the assumption about preferences in the model. A model is necessary, but not sufficient: that is, a model alone does not an analytic narrative make.

The analytic narrative approach provides a means to help get around these questions. The essence of many cases is unique, including the French Revolution, the American Civil War, or the surprising and quick development of Spain following the death of long-time dictator, Francisco Franco, in 1975. Analytic narratives deal with these cases by building a model that has multiple implications, and then testing an implication of the model that provides the possibility for both confidence in the claims and comparison across cases.

For example, Reference Weingast, Bates, Greif, Levi, Rosenthal and WeingastWeingast’s (1998) case study of the American Civil War builds on a unique feature of American institutions to explain long-term political stability – namely, the “balance rule”: the idea that both Northern free states and Southern slave states would be admitted in pairs, giving each set of states a veto over national policy. This institution fell apart in the 1850s. Weingast tests his account of this failure by using game theory to reveal a path not taken, given the interest calculations of those making choices about what path to take. This enabled him to estimate a counterfactual involving what would have happened had a contingency in the case study not occurred. Thus, the main thesis of the case – the balance rule – is unique and cannot be tested directly, but other implications of the approach can be tested so as to give confidence in the overall account.

A second example addresses the effect on public goods provision of a 2004 decentralization reform in postconflict Sierra Leone (Reference Clayton, Noveck and LeviClayton et al. 2015). The specificities of most laws are unique; even when the words are replicated, implementation varies across and within countries. In this instance, the narrative reveals the key stakeholders at the local level: elected councilors and paramount chiefs. The interests of the former should lead them to prefer successful implementation, ceteris paribus; but in some localities they conceded to the paramount chiefs, who preferred the status quo. Given other reforms that increased the power of the councilors and reduced that of the chiefs, the question becomes why the councilors deferred. What are the relevant comparative statics? This question produced a series of testable implications. The case, although unique, sheds light on the more general problem of the variation in the impact of decentralization on the delivery of health and education services.

This chapter proceeds as follows. In Section 11.2, we discuss criteria for case section. Section 11.3 discusses identifying processes and mechanisms, while Section 11.4 discusses the limitations of the approach. In Section 11.5, we briefly discuss the implications of the approach for the development context. Our take-aways follow.

11.2 Criteria for Case Selection

The analytic narrative approach combines a commitment to rational choice, a deep interest in a particular case, a method for devising a generalizable model of the case, and a means of providing empirical evidence, even in unique cases.

The combination also entails an aim most area specialists lack: to go beyond detailing the case to elaborate more general conditions for the problem or puzzle. This exercise requires criteria for selection of cases other than their intellectual appeal as puzzles demanding solutions. Standard approaches to case selection emphasize the bases for choice among a sample of cases which are informative about the causal chain of interest, because of the absence, presence, or extreme values of key variables. One traditional method advocates pairs of cases that are either “most similar,” hopefully allowing the analyst to identify similar mechanisms in the two cases, or “most different,” hopefully allowing the analyst the ability to isolate a mechanism that accounts for the differences. These traditional methods fail when more than one causal variable is relevant.

Bearing similarities to the analytic narratives approach is process tracing (Reference George and BennettGeorge and Bennett 2005; Reference CollierCollier 2011; Bennett, Chapter 4, this volume), which shares an emphasis on both sequencing and fine-grained description as means for making causal inferences. Process tracing also shares a concern with generating testable implications, but its emphasis is on key variables rather than the key actors, their interaction, and their strategies. This makes game-theoretic analysis largely irrelevant to process tracing.

Analytic narratives include features that make the cases amenable to modeling, which not all puzzles or problems are. Essential to the model building is the choice of cases in which the key actors interact strategically. That is, the choices of one actor depend on the choices of the other. In addition, analytic narratives consider situations that can be modeled as an extensive-form game, which generates a subgame perfect equilibrium.

Another necessary feature of an analytic narrative is the opportunity to get at an important process or mechanism not easily accessible through other means. For example, the extensive-form game allows the analysis to demonstrate the existence of a self-enforcing institution that often solves an important economic or political problem through creating a credible commitment. The advantage of the game is that it reveals the logic of why, in equilibrium, it is in the interest of the players to fulfill their threats or promises against those who leave the equilibrium path.

The formalization itself is not a requirement of a successful analytic narrative; indeed, in some cases, there are too many actors and no benefit from reducing the multiple players to the small number required for a game-theoretical model. Levi’s case on conscription (Reference Bates, Greif, Levi, Rosenthal and WeingastBates et al. 1998) illustrates how one can still use the logic of extended form games to assess the strategies and actions – and paths not taken – without formalization. Another example is Reference FerraraFerrara’s (2003) analytic narrative of the Burmese uprising in 1988 as a means to understand both a particular historical event and the more general question of the relationship between coercion and protest.

The final expectation of an analytic narrative is that the causal mechanisms and the structures or relationships must be generalizable to other cases under specifiable conditions. We deal with this issue below.

11.3 Identifying Sequence and Mechanisms

Analytics, in this approach, refer to the building of models derived from rational choice, particularly the theory of extensive-form games.Footnote ¹ The steps toward building the model include:

• First, extracting from the narratives the key actors, their goals, the sequence of options available to an actor at a given moment, and the effective rules that influence actors’ behaviors.

• Second, elaborating the strategic interactions that produce an equilibrium that constrains some actions and facilitates others. By making clear and explicit the assumptions about who the key actors and their preferences are, it is possible to challenge the assumptions to produce new insights and competitive interpretations of the data.

• Third, the equilibrium analysis leads to comparative static predictions that produce testable implications even if they’re not the main assertion of the case.

We emphasize this third criterion. An important advantage of relying on game theory is that this method often produces comparative statics – that is, predictions about how the equilibrium shifts in response to changes in the exogenous variables. This approach allows the analyst to identify the reasons for the shift from one equilibrium to another. It therefore produces expectations of behaviors in the form of testable implications if the key actors are staying on the equilibrium path and if they are not. A case study that includes a model may involve the first two criteria, but generally not the third. This is especially true for unique cases where it is hard to test the model directly. Both authors have written many case studies of this type. These cases may provide insights, but they are not analytical narratives (see, e.g., Reference LeviLevi 1988; Reference Weingast, Morris, Oppenheimer and SoltanWeingast 2004). Analytic narratives require testable implications derived from the comparative statics that the narrative helps reveal.

The narrative of analytic narratives establishes the principal players, their goals, and their preferences while also illuminating the effective rules of the game, constraints, and incentives. Narrative is the story being told but as a detailed and textured account of context and process, with concern for sequence, temporality, and key events. By meeting these criteria, the narrative offers a means to arbitrate among possible explanations for observational equivalences – that is, two distinct processes that lead to the same outcome.

Comparative statics are crucial for comparative research because they generate hypotheses of what could have taken place under different conditions. Comparative statics therefore clarify the relationship between the key endogenous and exogenous variables. Moreover, the consideration of “off-the-equilibrium-path” behavior typically reveals reasons and reasoning for why actors took one path and not another. Indeed, what actors believe will happen were they to make a different choice typically influences the choices they do make. As Niall Reference FergusonFerguson (1999) observed in his study of the causes of World War I, to understand why Britain entered what would otherwise have been a continental war, we need to know what the British believed would happen had they not entered the war. Another important aspect of the game-theoretic approach is that the off-the-path behavior of an equilibrium disciplines each player’s beliefs, for they must be consistent with all the other player’s strategies (see Reference Weingast, Tetlock and BelkinWeingast 1996).

For example, consider the illustrative “deterrence game” in the appendix of Analytic narratives (Reference Bates, Greif, Levi, Rosenthal and WeingastBates et al. 1998). Two countries interact: the home country and an opponent. The home country maintains a large, expensive army; the opponent does not attack. Is the large army the reason for peace as it deters the opponent from attacking? Or is it a waste of resources because the opponent has no interest in attacking? These two hypotheses are observationally equivalent: both offer explanations for why the opponent does not attack, yet they differ dramatically as to the reason for the observation. Different people have different beliefs that can only be understood contextually:

This approach provides the researcher with some discipline. As the deterrence game illustrates, absent a game and an equilibrium structure, it is possible to posit a wide range of beliefs that motivate action. How do we choose among these different accounts? In the context of a game, beliefs about another player’s actions are part of the equilibrium. Not just any sets of beliefs will work. In the deterrence game, the opponent must have a belief about how the home country will react to an attack; and, in equilibrium, this belief must hold in practice.

This form of explicit theory provides criteria to enable the researcher to distill the narrative and ensure that the explanation need not rely too much on factors outside the model.

11.4 Overcoming the Limits of Analytic Narratives

The analytic narrative approach, at least in its original formulation, had several potential limitations, some recognized by the authors and others revealed by various critiques.

11.5 Analytic Narratives for Use in Development Policy and Practice

In this section we turn, briefly, to suggest the implications of the analytic narrative approach to problems of development.

Economists have long proposed an economic role for political institutions, such as the market infrastructure embodied in the provision of secure property rights, enforcement of contracts, and, generally, the provision of justice and the rule of law (Reference WeingastWeingast 1995). Governments that use violence against minorities and opponents, confiscate citizens’ wealth, and create economic privileges (such as dispensing monopoly rights) fail to provide adequate market infrastructure. As Adam Smith recognized more than two centuries ago, the risk of violence and of plunder leads men to avoid hard work, initiative, and investment. In discussing settings in which “the occupiers of land in the country were exposed to every sort of violence,” Smith argued that “men in this defenceless state naturally content themselves with their necessary subsistence; because to acquire more might only tempt the injustice of their oppressors” (Reference SmithSmith 1776: III.iii.12:405). Further, a “person who can acquire no property, can have no other interest but to eat as much, and to labour as little as possible” (Reference SmithSmith 1776: III.ii.9:387–88).

But, if secure property rights, enforcement of contracts, and the provision of justice are necessary for economic development, how are such institutions built and, especially, sustained?

Reference North and WeingastNorth and Weingast (1989) developed the hypothesis of credible commitments to answer this question. Governments seeking to implement the economists’ prescriptions for political institutions had to commit to honoring rights of citizens and to use agreed upon political procedures to make political decisions. They developed their hypothesis in the context of a unique case: the English Glorious Revolution of 1688–1689. Although this revolution and its institutional consequences were unique to that case, North and Weingast provided some important evidence favoring their larger, general argument about credible commitments. Focusing on public finance, they showed that the ability of the English government to borrow money changed dramatically. Government debt had never been much above 5 percent of estimated GDP in the seventeenth century. But in the eight years following the Glorious Revolution, it rose by nearly an order of magnitude, to 40 percent of estimated GDP. Because debt repayment depends critically on credible commitments, the massive increase in debt in a short time suggests that a new mechanism for making credible commitments had emerged.

Sure enough, subsequent studies have identified some of the devices used to create credible commitments and have leant support to the hypothesis.Footnote ² First, Reference CoxCox (2012) has shown that a number of other variables also increased dramatically, consistent with the credible commitment hypothesis. Specifically, per the North and Weingast narrative, Cox demonstrates that parliament gained control over taxation and the issuance of public debt. Similarly, the ministerial responsibility system emerged: while parliament faced difficulties in holding the king accountable for public decisions, they could hold the king’s ministers accountable, forcing them to honor parliament’s interests. Second, scholars have undertaken a range of studies of public debt at similar events. For example, Reference SummerhillSummerhill (2015) has shown that nineteenth-century imperial Brazil provided the institutions for credible commitment to public debt, yet it failed to provide the institutional foundations for private financial markets and hence this fundamental basis for economic development. Reference Mo and WeingastMo and Weingast (2013: ch 4) reveal the means by which the South Korean regime under President Park Chung provided credibility to its promises to honor property rights and a range of other programs, such as education, underpinning that country’s economic development.

As a second illustration, consider political stability, another element widely agreed as important for economic development. Coups, civil wars, ethnic conflict, and other forms of disorder cripple a country’s ability to develop. Reference Cox, North and WeingastCox, North, and Weingast (2019) show that disorder in the form of violent takeover of regimes occurs surprisingly often in the developing world: the median regime of the poorest half of countries lasts only seven years. Just how do a minority of countries provide for political stability?

Reference Mittal and WeingastMittal and Weingast (2012) provide three conditions for political stability, one of which they call the “limit condition”: the idea that all successful constitutions reduce the stakes of power, for example, by providing incentives for political officials to honor a range of citizen rights. Limited government does not imply small government (as modern political debate suggests), but a government that can honor restrictions on its behavior, such as abiding by election results, refraining from the use of violence to repress enemies, and, generally, honoring citizen rights.

The logic of the limit condition is that high stakes make it much more likely that people who feel threatened by the government will support coups. For example, landowners in Chile under the presidency of Salvadore Allende supported the military coup in 1973 to protect themselves. Similar events led to disorder in Spain (1936–1939) and Kenya (2007–2008). One way that constitutions reduce the stakes of politics is through various forms of countermajoritarian institutions.

Mittal and Weingast develop their hypothesis in the context of the American case, where the institutional features of the US Constitution are unique. Subsequent work has revealed similar features in a range of cases of stable constitutions. Countermajoritarian provisions serve two valuable roles in preserving political stability. First, they often aid in the instantiation of democracy. When groups see themselves as potentially worse off under democracy, they are likely to resist democratization. The reality is that powerful – and sometimes inimical – groups often have the power to hold up democratization, such as slaveholders in the early American Republic, Whites in South Africa in the 1990s, the supporters of the authoritarian regime of Francisco Franco in Spain following the death of the dictator in 1975, the military dictatorship in Chile in the late 1980s and early 1990s, and the communist regime in Poland in 1989. In each of these cases, countermajoritarian provisions to protect powerful groups aided democratization and, eventually, the lightening of the relevant constraints. Similarly, electoral laws often bias elections in favor of constituencies that favor the previous regime (Chile and Spain). These provisions also become part of the limit condition once democracy has been initiated.

Others have also used analytic narratives to discuss the difficulties of achieving transitions to democracy. Reference FerraraFerrara (2003) suggests conditions under which widespread protest and uprising has little effect, given the strategic use of coercion. His case is Burma, but the implications are more general. On the other hand, Reference NalepaNalepa (2010) considers the conditions under which pacts and negotiated settlements among elites facilitate the transition from autocracy to more open access regimes. She finds that the transition will prove unstable (if it even takes place) unless a specific type of limit condition holds: namely, that key players receive credible commitments that the “skeletons in their closets” will not be revealed or that they will receive amnesty for politically problematic behavior during the old regime. Her analytic narrative focuses on Eastern Europe but is applicable to a wide range of cases where transitional justice is at issue.

Analytic narrative approaches are also useful in understanding why some reforms succeed while others do not in countries experiencing development. Methodologically, this demands explicit recognition of the comparative statics, on the one hand, and the off-the-path-behavior, on the other. These features distinguish analytic narratives from other case studies, enabling them to reveal processes and causal mechanisms that might otherwise go undetected and to provide the bases for generalizations that might otherwise not be possible. Some authors already self-consciously attempt analytic narratives (e.g., Reference HosmanHosman 2009 on Nigeria’s failed oil policies), but many do not. Even so, we could get additional leverage on reform by transforming existing case studies into analytic narratives.

Among the many accounts of reform, we have selected two categories of cases where analytic narratives clearly increase explanatory power for the particular instance as well as making the findings transportable to other situations.

The first set is where the same laws have different effects in different places. The study of the Sierra Leone decentralization reforms (Reference Clayton, Noveck and LeviClayton, Noveck, and Levi 2015) display not only varying impact but also reasons for that variation. Sierra Leone has a long history of tension among elites at different levels of government and a more recent history of tensions among key local elites. This case explores the consequences of the latter for effective public service delivery once decentralization is introduced.

A rich literature (cited in the case study) reveals contradictory expectations of the effects of local interelite dynamics. The narrative reveals considerable county-level variations in power sharing between the traditional power-holders, the paramount chiefs, and the newer power-holders, the elected councilors. This variation provides an opportunity to derive expectations specific to the case and then assess their plausibility. Indeed, from the narrative the authors hypothesized that competitive relationships among the two improve services while collusion reduces their quality.

The first challenge was to offer a measure of elite dynamics to be used in a statistical investigation of the implications of differences. To test the impact of this relationship required, first, a measure of the nature of their interaction. The probability of collusion was operationalized by using data that reports on the following direct relationships: the median number of times councilors report having contacted a chief in the previous month, the percentage of councilors that report having had a dispute with a chief during the past month, and the percentage of councilors that report that they are related to a Paramount Chief either through blood or marriage.

But what accounts for the distinctiveness of power-sharing arrangements? The strategic interaction underlying the implementation of the reforms reveals two possible equilibria of collusion and competition. Digging deeper uncovers factors that possibly change the strategic interactions among the key actors and, thus, the comparative statics. The authors considered the gender, age, and party of the councilors as well as the degree of electoral competition. But the most telling explanatory factor was one that could only be known by knowing the case in depth: the proportion of councilors who were in the Civilian Defense Force (CDF) during the war. Former CDF councilors were likely to have forged deep ties with the paramount chiefs who were the primary sources of funding.

The next step was to determine the extent to which these different kinds of power-sharing arrangements – and the probable causes of them – actually influenced service delivery. Drawing out testable implications that could in fact be explored with the available survey material enabled the authors to provide additional confidence in the hypotheses they had derived. Further statistical tests indeed suggested that collusion, particularly that produced by the proportion of councilors who had been in the CDF, leads to far poorer service delivery than does competition.

Bangladesh, Honduras, China, and the United States all have similar labor laws on their books but very distinctive actual protection and enforcement of labor protections in the supply chains of global brands. Reference Berliner, Greenleaf, Lake, Levi and NoveckBerliner et al. (2015a, Reference Berliner, Greenleaf, Lake, Levi and Noveck2015b) investigate the clusters of stakeholders and what transforms the relations of power among them. To do this, they consider the strategic interactions among key players and what transforms the current equilibrium or status quo. Using the logic of game theory but not formalizing it, they are aware that it is off the equilibrium path for workers to organize and make demands unless they are assured that they will not be punished for their actions by losing their jobs or being sent to jail.Footnote ³ That only happens when brands find it in their interest to improve worker rights and benefits, and this only occurs if government is upholding its laws or the reputation of brands among consumers is being threatened.

Unfortunately, both of these circumstances are most likely to occur when there is an unexpected (if predictable) catastrophe such as a major fire or building collapse where workers’ lives are tragically lost. Reflecting comparative statics, such a shift leads to reform, but whether the commitments are credible depends on the creation of legal institutions that are hard to change and that incorporate sufficient administrative capacity to implement the rules. The testable implications may differ among the cases, but they are the organizing principles of the cases. The findings are not promising for labor rights. Honduras and Bangladesh lack the government capacity to maintain a positive labor rights regime over a long period of time. In China, the government has the capacity but not the will to establish meaningful labor rights, although it does ensure some protections. The United States, which once had both the will and capacity, now lacks the first and possibly the second and so has undergone a reversal. The result in all these instances is that the pressure on brands to discipline their supply chains is episodic or nonexistent.

The second set of cases document instances where societal interests come to trump private interests, making it possible to actually implement policies that will serve the population as a whole. As we saw with the labor illustration, it is difficult enough to ensure the protection of the interests of a neglected group within the society. It is arguably harder to protect general interests, as the case of corruption in Indonesia (Reference KurisKuris 2012a, Reference Kuris2012b) documents. And it is arguably harder still to implement policies where the interests of the world at large are at issue, as the case of deforestation in Brazil (Reference JacksonJackson 2014, Reference Jackson2015) details.

Of course, in each of these cases a range of stakeholders are the beneficiaries or losers from policy change. To transform past practice required some combination of leadership, interests, expert knowledge disseminated widely, trust relationships, monitoring, new forms of direct enforcement, credible commitments, and mobilizations that changed the incentives of both government officials and recalcitrant stakeholders. All of these features are documented in these cases, and documented well. Lacking is a structure to the accounts that makes it possible to observe the causal mechanisms and derive testable implications. The comparative statics are not sufficiently explicit. While the Sierra Leone decentralization and labor standards cases do not provide an actual formalization of the game, the presentation of the material makes it possible not only to derive but also to test implications that enhance confidence in the claims of the authors and make them generalizable to other cases.

11.6 Creating Take-Aways

Multiple interpretations are inherent in the traditional case study method. Moving beyond traditional approaches, analytic narratives provide two methods for establishing the generalizability of findings from case studies. First, the model in an analytic narrative often affords a range of explanations and predictions. Although the main account of a unique case may not be testable, the model may yield other predictions that can be tested, either in this case or in other cases. Second, as with other methods, out-of-sample tests constitute an important route to generalization. The presumption today in social science research is that the authors will provide those tests themselves. However, seldom does the level of knowledge for the out of sample case rival the detailed understanding of the original case that puzzled the author. The demonstration of generalizability must rest on a larger community of scholars who take the findings applicable to one place and time to illuminate a very different place and time. Each case then becomes a case among many that are grist for the mill of scholars, experts on particular countries and sectors, and policy-makers who must work collaboratively to sort out the lessons learned.

In this chapter, we have outlined the analytic narrative approach and, in Section 11.5, suggested the potential value of the approach for problems of development. Reflecting the interest of the authors employing the approach, the applications tend to focus on political issues, such as political stability and violence. The approach also applies to case studies of particular economic reforms, and we believe it will produce valuable results in this area.

In summary, the goal of analytic narratives is to provide several forms of discipline on the structure of case studies, such as a game, with emphasis on comparative statics and on off-the-path-behavior, and on predictions that can be tested on aspects of the case even if the main assertion about the case cannot.

12.1 Introduction

This chapter considers how different types of development-focused organizations have introduced case studies into their operations, and explores the lessons from these experiences for other development organizations interested in using case studies to enhance their own implementation effectiveness.Footnote ¹ At one level, of course, case studies will be used differently depending on the organizational context; as such, to fully exploit a case study’s potential it must align with an organization’s specific reality: its history, mission, mandate, and capability. Actually doing this, however, requires undertaking the complex task of integrating cases into idiosyncratic organizational structures, rules, regulations and processes, and aligning it with a corporate culture that, at least initially, may or may not be favorably disposed to ‘learning’ in this way. In the sections that follow, we provide a comparative analysis of how this task has been conducted in four different development organizations, focusing in particular on how they select, prepare, and utilize case studies for collective learning.

A concern from the outset, and one that some regard as a pervasive weakness of case studies, is how to prepare cases that are both faithful to the unique particularities of each intervention and yet potentially usable by practitioners working elsewhere, perhaps even in different sectors, regions, and scales of operation. Indeed, “But how generalizable is that?” is a common critique levelled against case studies as a research method, where the concern is that the case itself is neither randomly selected nor “representative” of a larger population, but rather “cherry picked” to support predetermined conclusions. As methodological and empirical issues, these concerns are addressed elsewhere in this volume.Footnote ² For present purposes, we consider case studies not as “qualitative evaluations” nor as small-scale “impact assessments” of projects, but focus instead on their roles as diagnostic and pedagogical instruments within (and between) development agencies. In this sense, we consider how case studies are prepared and read in ways akin to their use in medicine, law, and public policy – which is to say, as instances of broader phenomena, wherein professionals use their seasoned experience (and, where appropriate, scientific knowledge) to learn from specific instance of how, why, where, and for whom particular outcomes emerged over the course of a project’s or policy’s implementation. If formal impact evaluations are concerned with assessing the “effects of causes” (e.g., Did this rice subsidy, on average, benefit the poor? Did that text message invoking sacred precepts increase credit card repayments?), then in this instance case studies primarily seek to discern the “causes of effects” (How was this village able to solve its water disputes so much more effectively than others? Why did that program for improving child nutrition fare so much better with younger mothers than older ones? Where were the weakest and strongest links in the implementation chain of this immunization program? Why do some development organizations seemingly learn more effectively than others?).Footnote ³ It is in responding to these latter concerns that case studies have a distinctive comparative advantage; in this sense they should be seen as a key complement to, not a substitute for, more familiar evaluation tools used to engage with and learn from development interventions.

In this spirit, our concern here is to work backwards from broader concerns about the conditions under which development organizations ‘learn’ (or seek to learn), with a view to considering the role that case studies play in this process. Our discussion proceeds as follows. Section 12.2 considers four broad factors that seem especially important for understanding how organizations (not just their individual staff members) learn – that is, modify and/or improve their procedures and products in the light of experience and evidence. Section 12.3 then considers how these four factors have been deployed in case studies as used by four different organizations engaged with development issues: the World Bank, Germany’s GIZ (Gesellschaft für Internationale Zusammenarbeit), the Brookings Institution, and China’s Ministry of Finance. Section 12.4 concludes by categorizing how these different organizations are using case studies to learn across four organizational levels.

12.2 Organizational Learning Within Development Organizations

How do development organizations learn? A reading of the literature suggests that four broad factors seem to be especially important for understanding whether and how such learning takes place: motivation, environment, knowledge type, and practical use. We explore each of these factors by responding to four related questions.

12.3 Using Case Studies for Organizational Learning in Four Development Agencies

Organizations have different ways of curating, documenting, and mobilizing knowledge. Generating and using case studies as a tool for organizational learning requires a considerable investment of an organization’s time and resources, and different organizations have deployed different approaches. This section presents the experiences of four different organizations engaged with development issues – a multilateral agency (the World Bank), a major bilateral agency (Germany’s GIZ), a leading think tank (Brookings Institution), and a key national ministry of a large developing country (China’s Ministry of Finance) – as they have developed their use of case studies within their individual contexts. Reflecting on the experience of these different types of organizations may assist other organizations in their decisions about whether and how best to incorporate case studies.

The organizations were selected on the basis of their participation in the Global Delivery Initiative (more on this below) as well as the type of organization they represent. They were assessed via oral interviews as well as complementary desktop research of secondary material. Based on this assessment, the chapter will now examine how the motivation for organizational learning, managing knowledge, and the use of case studies in managing knowledge can vary among different types of development organizations.

All of the four organizations are linked through their involvement in the Global Delivery Initiative (GDI; described below – see Box 12.2) and all have developed case studies and shared them through the GDI network, which allows for some comparison between methods and approaches used.

12.4 Lessons Learned in Aligning Case Studies with an Organizational Learning Agenda

In the previous section we noted that case studies on development practice are used in different ways and with different levels of systematization for the purpose of organizational learning. Here we can make use of our IGOIL categorization to explain how case studies from these different organizations tap into different levels of learning.

As we see from Table 12.2, different organizations use case studies for learning purposes, but such purposes serve different objectives. We can use the MoF of China and the World Bank as two examples with different purposes. For China’s MoF, learning is external facing, with partners that want to learn from the experiences captured in the Chinese case studies. This external interest may come typically from other governments that want to learn how the Chinese government dealt with a particular development challenge. Learning is done mainly at the interorganizational level: the MoF selects and systematizes experiences to be disseminated, and this external demand is what guides the capture and systematization of knowledge by the MoF.

The World Bank’s approach is also very much about interorganizational learning, by sharing experiences among institutions on how to address development challenges. However, at the same time there is a specific focus on knowledge retention and organizational learning, with the goal of interpreting and using the knowledge collected through the case studies to support the organization’s business practices and improve performance. The GDI approach focused on contacting particular partners and using group discussion to advance this learning agenda; it also provided training for practitioners to not only become case writers, but to develop capacity at the individual level for transformational change by better understanding the change process.

Table 12.2 also points to some of the different motivations for using case studies as a learning tool. In the case of MoF China and Brookings, for instance, case studies are shown as exemplars of how to do things or ‘what and how things work’ in the spirit of sharing such knowledge outside the boundaries of the organization. At GIZ the focus is to provide practitioners, within and outside the organization, with examples of good practices. Finally, GIZ understands itself as a convener of experiences on transformational processes, with the role of promoting dialogue not only at the practitioner level but also across organizations and countries.

Table 12.2 and the preceding discussion shows that case studies do not need to use the same knowledge-sharing strategy or audience to inform development processes. Case studies can be used as a learning tool to improve performance and implementation in internal practices. They may never be shared directly with other practitioners or stakeholders outside of that organization, but this approach may still spread lessons indirectly through changes in behavior and practices as a consequence of insights captured in the case study. On the other hand, case studies can be used directly to inform counterparts of experiences that provide insights on what works and how. In this instance cases may have more impact on an external organization receiving such knowledge.

Finally, the use of case studies as a learning tool also generates some knowledge value in the process of developing the case study itself, in addition to the output. As has been shown with MoF China, the GDI, and to some extent GIZ, case writers are trained to focus on a problem-driven approach to tackle case studies. These case writers are also practitioners involved in development projects who may be keen to incorporate this approach in future development practices. Further capacity building at an individual level may also take place among the key stakeholders involved. As a case study’s interviewees, they play a role in articulating their experiences, which are captured as knowledge on the “how to” of implementation. As experienced through the preparation of case studies by the four organizations discussed in this chapter, such engagement provides these key stakeholders with a new perspective on how to tackle challenges throughout the implementation cycle, and in the process perhaps generates a change of mindset.

13.1 Introduction

Former UK prime minister Tony Blair once said: “The problem isn’t vision. Often we know what to do. The real problem is getting things done.”Footnote ¹ In 2013, the World Bank Group embraced this challenge as a part of a new “science of delivery” initiative championed by its president,Footnote ² building on an ambition that Sir Michael Barber articulated in the service of the Blair government, manifest most conspicuously in his deployment of dedicated delivery units (see Reference BarberBarber, 2015). At issue was whether organizations could develop and formalize reliable guidance about how best to translate good ideas into real impact.

As part of this effort to improve implementation, the qualitative case study has a special place. Randomized controlled trials and other tools used to assess program design or evaluate the effectiveness of specific interventions provide little leverage or practical insight when the breakdown between ideas and impact lies in the hows – the specific steps taken to deliver a service or change an institution. A case study can help improve the translation of policy into results by tracing these pathways, illuminating the effects of context, process, politics, and capacities on intermediate achievements and broader outcomes.

But practitioners can also use case studies to improve performance in a variety of other ways. While previous chapters have laid out a social scientific rationale for the use of qualitative case studies, proposed standards for assessing rigor, and offered examples, this chapter focuses on employing case studies for adaptation and learning, especially in governments or organizations that seek to promote economic growth and development. It proposes that case studies useful for this purpose have seven specific qualities, though they may differ widely in other respects. Additionally, it offers a brief user’s guide for policy planners, managers, and instructors.

Our observations build on insights from two programs: the World Bank’s Development Research Group and its leading operational unit deploying case studies, the Global Delivery Initiative (GDI), and Princeton University’s Innovations for Successful Societies (ISS) program, which develops policy-focused case studies of development.Footnote ³ Both programs worked for many years with people leading change in different contexts. From 2008 through 2021, the Princeton program helped a rising generation of leaders address the institution-building challenges facing governments in fragile states and neighborhoods, low-income countries, and crisis situations. Case studies were, and remain, the program’s medium for enabling public servants to share experience with each other in an accessible manner. Similarly, the World Bank-based GDI, which launched in 2014, began as a collaboration among various development partners to help practitioners build a more systematic understanding of program implementation, promote policy dialogue, and improve operational effectiveness. The Global Delivery Library, one of the GDI resources, became an open repository of cases that tapped the tacit knowledge of field-level practitioners about how to navigate delivery challenges, enabling future operations to draw upon wisdom from past interventions.

13.2 From the Science of Delivery to Adaptive Management

Blair’s observation – it’s not the vision but the how that’s the problem – had its roots in a prime minister’s struggle to improve service delivery across different sectors, especially education, health, and policing. In the United Kingdom, as in every country, implementation is often the great bugaboo on which great ideas stumble. But offering reliable generalizations to help guide the work of front-line providers, managers, and ministers poses many challenges. The social world cannot be reduced to a set of laws or principles as easily as the natural world.

Efforts to frame a science of delivery exposed two different policy worlds: one in which it was possible to base generalizations on credible evidence, and another in which tracing the influence of actions on impact was more difficult, though still valuable. In medicine and education, for example, there were some strong points of agreement about measures that could have a big impact on broad outcomes, as Reference WagstaffWagstaff (2013) has correctly noted. Take the example of vaccination against childhood diseases. There is mounting evidence about how best to scale vaccination campaigns. Though not completely reducible to a formula – at least not to one that works the same way to the same extent in every setting – it is possible to think systematically about how to achieve results, including estimates of the participation rates needed to create herd immunity and innovations to help maintain the cold-chain when lack of electricity threatens vaccine viability. Reference WagstaffWagstaff (2013) points out that it is unsurprising, then, that champions of a science of delivery – the testable, relatively stable understanding of cause and effect within the implementation process – often started their careers in a field such as public health and that journals such as Implementation Science were specific to this policy area.

This science came together as the confluence of many strands of research and multiple methods of investigation. It is notable that the contributions in the pages of the Centers for Disease Control’s Morbidity and Mortality Weekly Report have included not only analysis of epidemiological data, but also case studies based on field interviews.Footnote ⁴ The qualitative case studies help identify the nature of the many gaps between the release of a vaccine to a health worker and actual protection of an individual against the disease, and often to point to remedies. By tracing the breakdowns in the process, they spur adaptation that could help improve the match between the numbers of people a campaign aimed to protect and actual levels of vaccine administration. Through multiple cases, as well as larger tracking studies, practitioners are able to come closer to answering the key “How?” questions that Reference BehnBehn (2017: 94) rightly highlights as essential elements of a science of delivery: “How does this strategy produce results? What exactly are the causal connections between the strategies employed and the resulting outputs or outcomes?”Footnote ⁵

Case studies have also aided understanding by enabling us to probe why outliers – exceptional successes or failures – differed from the patterns normally observed, thereby illuminating possible ways to improve performance across the board. This was the approach adopted by Reference Brixi, Lust and WoolcockBrixi, Lust, and Woolcock (2015) to learn from local service success stories in parts of the Middle East and North Africa. Household survey data from several countries in the region indicated that student performance was often poor, despite the fact that school access and facilities had improved. If all schools in a country operated under the same set of regulations, these authors asked, why do some areas perform so much better than others, controlling for demographics? Did the differences stem from a condition outside the control of managers, or was it something that principals and teachers in one area just decided to do differently – a practice that, at least in principle, others could replicate? The household surveys did not contain the type of information that allowed them to answer these questions, so the team went to the successful schools and studied them. One hypothesis was that degree of parental engagement affected both teacher behavior and student performance. The questions the team posed therefore included several about interaction between school officials and the community. The case studies found that the successful schools were those where principals and teachers met with residents and there was more communication with families. The challenge was then to figure out how to generalize a practice that was at least partially sensitive to the orientations and aptitudes of school leaders. In this instance, qualitative case studies supported development of alternative explanations and illuminated a potential solution to the problem of low-performing schools.

Not all policy spheres look like either of these examples, however. In some, policy arenas, implementation involves multiple changes at once, which means there are several possible causal explanations for outcomes. In Reference BehnBehn’s (2017: 96) words: “Thus, the manager’s ability to assign causal credit is difficult. And if the management team is just starting out – if this is the team’s first effort to improve performance – which of the team’s multiple actions deserves how much of the credit?” The answer to this question cannot be called “science,” he says. “It could, however, be an intelligent guess.”

An intelligent guess is a step in the right direction, a hypothesis rooted in facts, though it isn’t the same as an evidence-based handbook, the kind of product Reference BehnBehn (2017) suggests a science would produce. Where it is hard to winnow out which conditions, circumstances, or actions carry the most weight in delivering a development outcome, and where we are therefore likely to have a high ratio of intelligent guesses in decision-making, implementation may adhere to a different model. Continual review, learning, and mid-course correction become essential. Though long practiced, this approach has more recently gone under names such as “adaptive management” or AdaptDev, which now has its own Google Group,Footnote ⁶ “Doing Development Differently” (DDDFootnote ⁷), and “Problem-Driven Iterative Adaptation” (PDIAFootnote ⁸). The common idea across these new platforms is that where a traditional after-action review, for example, is conducted at the end of an initiative, the push instead should be for feedback and learning to occur throughout an effort to implement a policy or institutional change. Reference Booth, Balfe, Gallagher, Kilcullen, O’Boyle and TiernanBooth et al. (2018: 8) point to a process in which implementers, in response to complex challenges, “deliberately set themselves up to learn by trial and error, testing initial approaches and adjusting rapidly as evidence on possible avenues of change is acquired.” Matt Reference AndrewsAndrews (2018: 1), one of the key contributors to this approach, has written on the basis of his long experience: “We always ask of PDIA in practice: What did we do? What results emerged? What did we learn? What did we struggle with? What was next?”

Although both policy learning and learning-by-doing have a long history, the ambition of the Doing Development Differently and AdaptDev communities that have emerged in this space is to expand the practice of experimenting, learning, and adjusting in domains where broad evidence-based generalizations about implementation are out of reach. In these areas, the people responsible for translating ideas into practice will almost certainly encounter challenges and unexpected obstacles (Reference SchonSchon 1983; Reference Pritchett, Samji and HammerPritchett, Samji, and Hammer 2013). If they do not step back, reflect, learn, and adapt, they risk persisting with interventions or strategies that are not well suited to the situation that they face. Therefore, these teams must be ready and willing to adapt mid-course, to experiment and scale up what works, and to iterate and integrate feedback into implementation. Together with careful planning and the elaboration of a clearly articulated theory of change, the incorporation of “rapid feedback loops” into an endeavor is crucial, as is using these processes for “learning in response to ongoing challenges” (Reference Pritchett, Samji and HammerPritchett, Samji, and Hammer 2013: 1).

In this corner of the policy world, where causal relationships are less straightforward than they are in public health (and elsewhere), case studies help practitioners pool observations, recognize what has worked, identify where things aren’t turning out as anticipated, flag surprises, and open up space for adaptation. They help make the tacit knowledge practitioners have accumulated as explicit possible. Although they may draw on focus groups, surveys, and quantitative evidence, they employ interviews to help trace the steps taken, departures from the roadmap, and intermediate results in order to help us better address both anticipated and unexpected circumstances and increase the probability of generating intended impacts.

In early experiments, embedding case development and data collection directly into projects not only strengthened the quality of evidence produced but also enabled managers to make mid-course corrections and secure stronger buy-in from other stakeholders. Innovative elements have sometimes included smartphone surveys to check whether a service reached intended beneficiaries or assess satisfaction, geotagged information displayed on maps to help spot service coverage issues, satellite photography to track crop conditions, and other information generated with relatively low-cost and flexible tools that have a broad variety of applications (e.g., see Reference Danquah, Hasham and MacFarlaneDanquah et al. 2019 on Sierra Leone). Workshops to document and review implementation steps taken to date help staff members spot omissions and bottlenecks and discuss creative ways to surmount unanticipated obstacles.

The World Bank’s Global Scaling Up Rural Sanitation program aptly illustrates this kind of effort. With the goal of making a dent in the 2.5 billion people worldwide without access to improved sanitation, the project launched pilots in three countries, which served as learning laboratories for developing a theory of change. After this pilot phase concluded, the project then made the necessary adjustments and scaled up to a further 10 countries; to date, it has provided some 22 million people in 13 countries with improved sanitation.

The use of pilots in the initial “learning laboratory” countries provided crucial knowledge about what worked and what did not. This information was then disseminated through a global network, allowing team members to reflect on and analyze the results of their actions. Team leaders were able to learn from these initial lessons in real time, allowing for quick adaptation. An iterative and adaptive approach was also hardwired into the program, giving task team leaders both the freedom and the mandate to apply lessons learned in their countries or areas of responsibility, while also adapting and correcting course as they scaled up and collecting their own evidence locally to target effective behavior changes and interventions.

A second example from the GDI illustrates a slightly different approach, this time in the context of improving access of Nigerians to sustainable, clean, potable water. A case study indicated that governance reforms were difficult to implement, trust in the system was low, and monitoring was weak – with the result that progress had stalled. It was crucial to establish trust, build networks, and enhance relationships with a wide variety of stakeholders. To design a new phase of the project, the World Bank decided to share the case study and solicit ideas from each major stakeholder. It organized a series of meetings to invite observations and proposals. The first convened its Nigeria task team leaders. The subsequent meetings took place in Abuja and involved participation from representatives of more than sixty agencies, including the head of the Federal Program Implementation Unit, the high representative of the Federal Ministry of Finance, State Ministers of Water, State heads of the program implementation units, and the World Bank Country Director. Participants had a chance to discuss the case itself and introduce other information, then they charted out concrete recommendations.

13.3 Seven Qualities That Make a Case Useful for Practitioners

For purposes of learning and mid-course adjustment, not all case study formats are created equal. Moreover, the information and format needed are not always the same that academic colleagues seek. The GDI and Princeton’s ISS program both ambitiously tried to tailor what they do to serve three distinct audiences: practitioners who want to improve implementation success, policy researchers or scholars who want to ground a (social) science of delivery, and aspiring leaders completing courses of study in universities and staff colleges or executive education programs. The jury is still out on whether it is possible to serve three masters equally well. Nonetheless, the experience to date has generated some wisdom – not yet formally tested! – about what helps a case to meet the needs of practitioners. This wisdom can be spelled out in seven principles:

13.4 Putting Cases to Work: Moderating a Case Discussion

A case is not usually a stand-alone document, though it can be so. If an important purpose of case studies is to promote learning and adaptation, then much rides on their capacity to stimulate group reflection, deliberation, and innovation. This in turn raises another question: How does one effectively moderate a case discussion?

Coming forward to the present, in our experience, the tone, sequence, and focus vary depending on whether the aim is to teach – to introduce key concepts and ways of thinking about a problem – or to help people who have participated in implementation reflect on their work. For the first purpose, the moderator may play a strong role in directing the discussion so that a group reaches key points, pausing to elaborate these. By contrast, for adaptive learning, where the point of a discussion is to help the people who carried out the work reflect and solve problems, the moderator may stand back a bit more to give participants a bigger opportunity to shape the agenda and to get into specific operational details in more depth than one might in a classroom setting. In both situations, however, there are some shared objectives, most importantly stimulating creative thinking about ways to: overcome obstacles that continue to impede success; mitigate the downsides of a generally successful response; reach difficult (isolated, marginalized) communities; take the intervention to scale; or adapt an approach for different circumstances.

To use a case for classroom purposes, we usually begin by reminding the group of the broader issues at stake. Every case has a development challenge at its core, the public value the people at the center of the action seek to create: the desired impact on citizens’ lives. Every action also has an author, so naming names is important, or at least naming offices: “Minister Marina da Silva wanted to reduce the rate of deforestation in order to adhere to a new climate regime and preserve water quality and availability in her country”; “Sudarsono Osman wanted the land registries in Kuching to serve citizens faster, with fewer errors.” The discussion leader may want to add some additional facts to situate the issue, identify what created the space for change, and add some more detail about the lead decision-makers.

Next comes the dramatic moment: “But … something stood in the way.” The discussion leader then poses a series of questions, beginning with “What was the main problem, the main delivery challenge?” At this stage, it is important to ensure that everyone can identify the general form of at least the major implementation problem in a case – process efficiency, aligning the interests of a principal and an agent, collective action, or coordination, for example: “Mr. X is responsible for making the program work, but he’s stuck. At the start, what is his main problem? What is the general form of this problem?” Knowing the general form enables the case user to link to a general toolkit and consider whether solutions often considered in other settings might be useful in the circumstances at hand. The ability to abstract in this way enlarges problem-solving capacity. It is important to pause and sharpen familiarity with the general concept and the standard toolkit at this point.

Third, we help users connect with the context: “What do we know about the setting and the elements of context that might shape which tactics Mr. X can deploy?” Context is something that will come up throughout the discussion but especially at the end, when the focus is often on scale, scope conditions, and adaptations required to help a similar approach work in another setting. Context may include resource levels, diversity, socioeconomic conditions, government structure, legal authority, and many other conditions or circumstances, some of which may be malleable, while others remain fixed.

The real focus of the discussion comes after this point: “What options did they consider? Were there other possibilities and, if so, do we know why they weren’t considered? What motivated the choices they made?” And then: “Let’s work through the steps the team takes … ” The central objective is to develop a clear outline of the strategy and tactics employed. If the real issue the instructor wants to use as a focal point occurs later in the case, then it may be perfectly acceptable to expedite the discussion and simply throw the key elements of the initial response into a Powerpoint slide. “So here are the steps they initially took … Have I got it right?” Usually, however, the aim is to pause to consider the purpose of each step, the appropriateness of the design, what proved difficult to do, any pleasant surprises, and how sensitive the actions taken were to the aptitudes of team leaders or context.

In the classroom, the instructor’s job is to help participants identify concepts useful for analyzing problems that emerge at each step, as well as to bring external information to bear, where warranted. One of Princeton’s Ebola response cases, for example, focuses on carrying out contact tracing in a very difficult context. If the group is unfamiliar with the key elements of contact tracing, it is helpful to call a short “time out” and explain these in some detail. Even if the elements are in the case text, pausing to reinforce the ideas is often helpful for nonspecialists.

Sometimes the focus of the discussion is not on the strategy or the main steps taken, but on an unanticipated obstacle a team confronts: “There is a big unanticipated obstacle in this case … They struggle to adapt. Put yourself in their shoes. How would you deal with this situation?” If the obstacle is minor and the response is successful, it is possible to fold this discussion into the previous stage of the conversation. If the obstacle is significant and incompletely resolved, the major part of the discussion could focus on this matter. The aim is then to help participants identify possible solutions by abstracting from the specific – giving the problem a general name that links to a toolbox – or by inviting each person to tap his or her own experiences and intuitions about how to solve the problem.

At this stage the moderator’s role is to ensure everyone has a chance to contribute and to provide two or three alternative ways to structure the problem under discussion, in the event that everyone is stuck. For example, in one Smart City case, a public health unit used sophisticated math modeling to identify households at risk of lead poisoning, but the effort temporarily ground to a halt over the question of whether it could enter houses at risk and intervene, given concerns for privacy, personal autonomy/consent, and data security. Did it matter that those most at risk were too young to make informed choices on their own behalf? Would the answer to these questions be different if the issue was secondhand cigarette smoke or some other kind of risk – and if so, why? The moderator stimulated thinking by highlighting the ethical principles at issue and inducing participants to think about the implications by pointing to analogous issue areas where the same quandary was a matter of settled law or procedure.

The discussion moderator may want to summarize the results actually achieved and move on, but it is also possible to craft two important conversations around this segment of the case: one focused on causation and the other focused on metrics. Often the conversation will jump to the impact on the broad development challenge, the outcome highlighted in the beginning. In most instances many things affect this type of outcome, so it is important to identify the other things that contribute – the potential confounders – and then try to identify the specific lines of influence through which policy implementation shaped this “public value.” To establish these lines of influence, we usually have to focus on intermediate outcomes or outputs: faster delivery times, lower rates of error, more inclusive coverage, etc.: “Were these the right metrics? Can you think of better metrics? If your office didn’t have much money, is there a way to assess effectiveness inexpensively?” “What contributed most to these improvements?” “On one important dimension, there was little improvement … Why?”

Finally, if the purpose of the discussion is to assess the extent to which lessons from the case are applicable in other contexts, then it is possible to skim through some of the other stages and focus on this matter. Identifying the scope conditions, or the central factors and processes shaping the effectiveness of the solution case protagonists deploy, is central to this task. It is also possible to focus this part of the discussion on ways to improve further, to mitigate the downsides of the tactics selected, or to borrow from other fields to get around some of the limitations associated with the tactics actually used.

Some moderators subdivide the cases, asking participants first to read just the opening sections that outline the problem and the delivery challenges (possibly also the options considered and framing), so that the group has a chance to think about tactical toolkits available and how to proceed. The moderator then hands out further sections of the case, and the next phase of the conversation picks up with what the decision-makers actually did and the pros and cons of the approach, improvements, etc. A third handout might focus on an unanticipated obstacle or on results, prompting another turn in the conversation.

Over the years we have come to share the view of Harvard Business Case Publishing that providing moderators with teaching notes or discussion guides improves usage and enhances the quality of discussion. These notes provide some of the general concepts, toolkits, conceptual puzzles, options, and additional background information that moderators often need to move a conversation forward and inspire creative thinking. Generating them should become a part of the case development process, and they usually flow well from the initial research design and the cross-cutting analysis produced at the end, if there is such.

13.5 Using Case Studies as Part of Adaptive Management

Using cases for problem-solving or improvement within an organization entails a slightly different approach. In this setting, the case study becomes part of a participatory process designed to improve problem identification, foster development of solutions, and win agreement on accompanying changes in practice, including monitoring results. Since 2012, this form of adaptive management, long practiced in many major companies, has attracted a following in public sector development organizations. The United States Agency for International Development’s adaptive management principles, treated as requirements in some of its assistance packages or awards, include elements such as regular monitoring of results; practices to support mid-course review of strategy and implementation and course correction; rewarding “candid knowledge sharing” and collaborative learning; and sharing results widely.Footnote ¹¹

The qualitative case study can play an important role in this approach. In some instances, the case writer’s role is to conduct interviews before a mid-course review begins and to assemble observations of individual team members and beneficiaries in a form the moderator can use to structure discussion of what has worked, why some steps did not succeed to the degree anticipated, and what to do next. The project manager may then use the results of the conversation to revise the program so that there is a record for comparison after the next attempt to improve delivery. Alternatively, a designated writer may skip the first step and become the recorder for the group discussion, creating a case as a record or after-action report. Qualitative cases drawn from other settings may also enter the moderated discussion at various points in order to spur reflection and creative thinking about what decision-makers should do next.

Those developing the PDIA approach have given this issue a lot of thought. In their experience, one of the challenges associated with learning and adaptation is to induce team members to think hard about the sources of success and difficulty. For this purpose, they employ some of the tools of the trade that Toyota has developed – for example, the “Five Whys” exercise that asks participants to push themselves beyond an initial statement about the proximate cause of a problem to deeper reasons: If A was the cause, why did A happen? If B caused A to happen, what caused B?Footnote ¹² They go through this exercise at multiple points, creating a “fish diagram” to help provide a record of the discussion (see Figure 13.1). This discussion generates information and insight to incorporate in the next case draft, rendering the case study a collective, participatory product.

Figure 13.1 Example of fishbone diagram in adaptive management and participatory case study

Source: Reference Andrews, Pritchett and WoolcockAndrews, Pritchett, and Woolcock (2015: 21)

The next step is to encourage people to think outside the box in generating solutions for each problem the process identifies. The aim at this stage is to encourage people to draw on their own thinking. At this point it may be helpful to consider what others who have faced similar problems have done, drawing on cases from the libraries that the GDI and the ISS offer, or some other source. These stories take people out of their circumstances and surroundings, reduce defensiveness, and trigger new lines of thought. These conversations about other places usually quickly lead back to a more open discussion about the issues on the table. The moderator may summarize what another government or agency tried and then simply ask, “Would that work here, in your view?” “What would you do differently?” “What is the theory of change behind this idea?” “How will we know if this idea works?” Again, this part of the discussion can go into the case draft, if the case serves as the collective record.

But there is also a further step in adaptive learning. The PDIA authors ask participants to identify the space for change in connection with each problem identified in the previous step. That space includes three elements: Authority (who has the authority to act?), Acceptance (Do the people who will be affected recognize the need for change?), and Ability (Is there capacity – time, money, skill – to act?). This phase of the discussion may help set priorities – if the suggestion is to move where there is space or leverage – or it may lead to creative thinking about how to expand the space for change. This information may also become part of the case record.

Both the GDI and the Princeton ISS program have contributed to learn-and-adapt initiatives. In its first years, the GDI’s Science of Delivery team helped more than sixty different projects use cases to broaden or deepen thinking during review of the initial concept note, decide how to address operational challenges, or present results. Participants sometimes convened their project staff to discuss and record their experiences as their work moved forward, resulting in the gradual development of a case, or they assembled at the conclusion of a project to develop an after-action report that documents the steps they took.

The GDI described its method, the Delivery Lab, as an opportunity to bring together thematic experts with specific operational knowledge from GDI’s partner organizations and other invited guests who are working to overcome the obstacles and bottlenecks that can impede development efforts. Each lab began with a practitioner (the challenge holder) sharing an operational challenge that he or she currently faced in the context of an ongoing project. This brief presentation was followed by a facilitated group discussion and brainstorming session where experts shared relevant experiences. Ultimately, participants worked together to cocreate actionable solutions. The sessions allowed for peer exchange of experience-based knowledge, as practitioners explore problems and think through potential solutions.

13.6 Conclusion

Implementation-focused case studies play a vital role within the development community in the three key respects described here: (a) helping to develop better understanding of implementation dynamics (a science of delivery), (b) training, and (c) supporting adaptive management. But both the GDI and ISS program observe that practitioners have often employed qualitative cases for other purposes too.

Sometimes the aim is simply to help a manager or public servant structure a problem and think about the menu of options others have tried. A case study can provide a quick guide to key issues and enough operational knowledge to enable the decision-maker to figure out what s/he needs to know so as to pose the right questions in a more detailed person-to-person follow-up conversation. For instance, Princeton’s ISS program has documented the efforts of a number of governments to improve cabinet office coordination and support for policy decisions. These cases have helped chiefs of staff and deputy ministers learn from each other without having to take valuable time to travel abroad in search of ideas. But they have also facilitated face-to-face small group meetings that have matched those who have led impressive reforms with those who are just beginning to think about what to do.

To take another, similar example, the GDI used a case on accountability for mineral royalty funds to support Colombia’s peace process. In Colombia, royalty funds from mining and natural resources held potential for financing local projects and building legitimacy. However, early experiences in managing natural resource funds were unsuccessful in part because local governments lacked capacity to avoid misallocation, corruption, and poor planning, and the central government had no mechanism to remedy this problem. As a result, instead of building peoples’ confidence in their governments, the initial program undermined trust and the sense of government efficacy. The National Planning Department then created a new program that had flexibility to help local governments to build their capacity to implement projects, while also mobilizing community members to carry out “citizen visible audits.” The case study on this program, which helped ensure that money was not stolen or misplaced and that projects met the real needs of the citizenry, helped foster agreement among parties to the peace process. In Colombia, an actual example of how to build local accountability and legitimacy and equitably use natural resources to develop the country moved policy conversations forward.

Apart from this kind of use, the programs have also found that people who have played important roles in the changes a case documents value the record of achievement. Those who labored hard to make something happen often immediately move on to the next project or crisis. The case provides welcome recognition and helps them explain their own contributions to others. They say the acknowledgment helps fuel another round of effort. Indeed, organizations often ask the programs whether they will commit to develop a case study on a specific program so that managers can say to team members, “If we do well, we will become a model … ”

In other instances, people have written to say that they have used a case as a briefing to prepare for deployment to a new post. Operations documents and technical reports rarely contain names, but cases often do, thereby helping newcomers know to whom they can reach out for additional information while also offering historical context and an implicit heads up about sensitivities.

Finally, the case study is a vital tool for communicating to a wider audience what purpose a development initiative serves, the human story that unfolds around and within it, and the results achieved. It gives form and spirit to the numbers we often use to analyze policies. In an era when trust in governments and international organizations is low, the case study is a way to make the work practitioners do more accessible to fellow citizens and to rebuild shared understandings about the missions we pursue.