Hostname: page-component-cd9895bd7-jkksz Total loading time: 0 Render date: 2024-12-23T15:48:36.792Z Has data issue: false hasContentIssue false

Varieties of Empiricism in Political Science

Published online by Cambridge University Press:  15 December 2015

Rights & Permissions [Opens in a new window]

Abstract

Type
From the Editor
Copyright
Copyright © American Political Science Association 2015 

Perspectives on Politics is a scholarly journal of political science. Our mission is to publish excellent political science research and writing that engages matters of real political consequence, bridges conventional methodological and theoretical divides in the discipline, and is framed and written in a manner that speaks to our broad political science readership. This is what we mean in calling the journal “A Political Science Public Sphere.”

I have spoken and written the above words so often over the past six years that I can probably recite them in my sleep. They constitute my “official” representation of the journal in all contexts in which I communicate with authors, reviewers, publishers, colleagues, students, and other interested parties, including our journal’s reading public itself.

Our journal publishes political science in a variety of formats. Our two most prominent formats center on rigorous peer review.

We publish research articles that pass through our double-blind peer review process, which typically involves the following steps: internal review, at least two and usually three rounds of double-blind external review and revision, at least one round of line-editing by me, and then a round of professional copy-editing (which always involves queries and some further revisions).

We publish discussions of books in a variety of formats—standard reviews, review essays, symposia, and Critical Dialogues. All of these discussions also involve peer review, in three ways: the books under review are almost always university press books that have only been published after undergoing pre-publication peer review; all of the reviews we publish are peer reviews, written by political science peers, scholars who have Ph.D.’s and at least one related publication; and all of the book reviews are professionally reviewed and edited by me, and then subjected to professional copy-editing.

It is important for our readers to be reminded of these logistics.

But I have a more substantial point in noting these things: as a political science journal, at the heart of everything we publish is vigorous peer-review, criticism, revision, and editing. In this respect our journal is similar to most other political science journals. And the fact that there are a great many such journals means that political science is comprised of a very extensive and intensive web of journals, forms of communication, forms of scholarly conjecture and refutation, and forms of peer review.

Two other things about the kind of political science publishing we do are worth noting explicitly.

The first is that there is an ineliminable, irreducible element of editorial judgment involved in all publication decisions. Anyone who has had any experience either editing a journal or a press, or submitting a piece for review at a journal or press, knows that it is very common, and indeed almost inevitable, that a piece will receive a range of reviews. Some will be more or less critical. Some will be more or less favorable. Some will come with a recommendation of “revise and resubmit,” some will come with a recommendation of “decline.” Almost all substantial reviews are not reducible to these simple recommendation terms. They offer “ifs,” “ands,” and “buts.” What is an editor to do? What I do is this: I read the reviews carefully, looking for common threads and also differences, and trying to assess them; I then re-read the paper under review, very carefully, in light of the reviews; I then sit back and take some time to think; I then craft a very careful letter to authors; and if this letter contains an invitation to Revise and Resubmit, authors are asked to submit a memo explaining their revision strategy, and authors are informed that this memo, along with the revised paper, will be shared with the reviewers. With articles that are accepted, this process is repeated at least twice, and sometimes as many as four times. At a certain point a reviewer consensus emerges, or at least a balance of opinion. This consensus almost never involves universal assent to the paper’s key claims. Reviewers often write reviews that end with lines like these: “This is an excellent paper. I still remain unconvinced about X or Z. But the author has worked hard to revise the paper, and it offers a valuable perspective even if I disagree with it, and it deserves to be published.” Or these: “Given my own well known work, I cannot agree with this paper. But it is interesting, and it ought to reach the light of day.”

Most of the most careful and favorable reviews are laced with reservations or dissents. The most honest reviewers often admit that it would be hard for any author to satisfy all of their reservations. But on balance they recommend publication. And as editor, I am charged with the responsibility of reading all of the reviews, and again re-reading the paper, and then weighing the reviews on balance, and on balance coming to an editorial decision.

On balance.

Even the best, most rigorous, review processes in political science involve such judgments. And if we think about this for but a minute, it should come as no surprise. For we all know, both intellectually and experientially, that political science is a very complex and diverse discipline that consists of many very smart and very opinionated people who agree on some things and disagree on many others, and that the debates among us are what drive our discipline. It doesn’t matter what the topic or the method. There will be disagreement. Another way of saying this is to say that even the best research is imperfect, limited, and contestable. And invariably contested.

Max Weber said precisely this in his famous 1918 lecture “Science as a Vocation”: “In science, each of us knows that what he has accomplished will be antiquated in ten, twenty, fifty years. That is the fate to which science is subjected; it is the very meaning of scientific work, to which it is devoted in a quite specific sense… Every scientific ‘fulfilment’ raises new ‘questions;’ it asks to be ‘surpassed’ and outdated. Whoever wishes to serve science has to resign himself to this fact… We cannot work without hoping that others will advance further than we have. In principle, this progress goes on ad infinitum.”

And this leads me my second point: When a political science article, or book, is published, what happens is that it is given a particular space, and then set free in the public realm of inquiry, dialogue, debate, and further inquiry. Ad infinitum. Publication is not sanctification. It is not a signal, to the scholarly community or to the broader public world, that the published work is True, Final, Perfect. It is a signal that the work has been evaluated by a range of scholars, and has been found to rise to a level of plausibility, validity, and excellence that merits publication and that warrants being treated with particular seriousness by scholarly colleagues. That is all. People are then free to read it or ignore it—we have all experienced both. They are free to engage it, critique it, build on it or tear it down. And every single publication will experience every one of these possible responses.

Every excellent research article that is published in any excellent political science journal takes its bearing from previous excellent research articles that are found in some way wanting—some published years ago, some published months ago—it contests these others, and it advances an alternative explanation. An alternative perspective on politics. Every excellent political science book that is reviewed in a political science journal is subjected to a serious discussion that balances criticism and appreciation. And the critical review of one book often feeds the writing of another, sometimes even by the reviewer. Articles engender more articles. Books engender more books. There is no end to this process.

These observations are commonplace. We all know them to be true. But at the same time, when many political scientists talk, with a spirit of great seriousness, about “science,” they forget such things, and act as if there is some method, or set of rules, or system of bureaucratic requirements of “data access and research transparency,” that can mitigate the inherently interpretive and inherently contestable and provisional character of even the best contributions to political science. We want to see all the data. We want to know all the steps in the process of reasoning. We want to be able to subject everything to an ultimate test, to an essential judgment of Truth or Falsity. But there is no such test, no such judgment. There is no Archimedean point from which our complex and constantly changing political world can be apprehended, and no vantage point from which our truth claims can be insulated from provisionality.

Of course this does not mean that “anything goes.” At every step in the ongoing process of scholarly inquiry—and such inquiry consists of a never-ending recursive cycle of pre-publication and post-publication review—political scientists are liable to questioning. “Why do you say this?” “What is your evidence for this?” “Are you sure you have interpreted this evidence properly?” “What about this alternative interpretation of your evidence?” “What about this alternative evidence?” “Are you so sure that an alternative explanation doesn’t work better?” Whether one’s work involves multivariate analysis or formal modeling or descriptive case studies or detailed ethnographic description or constitutional analysis or textual exegesis or normative argument, one is always liable to questions such as these. Different kinds of evidence or argumentation may be relevant in different situations. Scholars will often disagree about the kinds of evidence or argumentation that are relevant. A level of meta-argument ensues, sometimes even followed by a deeper level of meta-argument. This is the life of scholarship, and every experienced editor knows that while this life can be facilitated, and in some ways regulated, it cannot be purified or perfected. Every good editor also knows that there is a difference between editing—an intellectual activity involving engagement, communication, facilitation, and the cultivation of spaces of argument—and policing.

Publication is not the end of critique, contestation, and critical review by peers. It is one step in an iterative and interminable process. Publication does not confer a final editorial Good Housekeeping Seal on a piece of research. It merely signals that a piece of research has been deemed important by a range of peers and by the editors who are publishing it, and that it has been through serious processes of review and revision and editing, and that it thus merits special attention.

All modern political science is empirical in this sense: It involves explanatory or interpretive claims about the political world that draw on experience—whether this be the experience of sensing, reading, observing first-hand, coding or measuring or mathematically analyzing—and that are fallible, contestable, and replaceable. No piece of research is above or beyond criticism. All research is perpetually subject to critique. To submit one’s research for review and publication is to presume its fallibility and submit it to critique. To publish it is to submit it to the gaze of a broader community of critical readers, who will evaluate it in light of their own experiences as readers, observers, and analysts.

At the same time, political science is a pluralistic discipline, characterized by a great variety of empiricisms. Indeed, a case could be made that the farther along the spectrum of quantification or formalism we go, the farther we move from the empirical world as experienced to something more refined, processed, computed, systematized—and thus more esoteric and perhaps even unintelligible. But that is a philosophical suggestion, and Perspectives is not a journal of philosophy.

The current issue of our journal contains empirical research in a variety of forms.

One is the more or less conventional form of most empirical research in U.S. political science—the quantitative analysis of empirical data. This is the kind of empirical political science best suited to many of the most insistent demands for “Data Access and Research Transparency.” For if the core of your research is the report of new empirical findings or the analysis of new statistical data, then it makes perfect sense to expect that your data and the methods of analyses are well explained and that your most important data are accessible for further analysis. And this is an expectation that we have always taken seriously at Perspectives. At the same time, even the most conventionally “empirical” work that we publish is grounded in larger theoretical frameworks and normative concerns—Why do we care about inequality? Why do we care about civil wars? What’s the problem?—and thus draws on political theory broadly construed.

In our September 2014 issue we published a piece of quantitative empirical research that quickly went viral: Martin Gilens and Benjamin I. Page’s “Testing Theories of American Politics: Elites, Interest Groups, and Average Citizens.” This article offered a statistical test of diverse models of political influence in the United States by developing a large data set of proposed policy changes between 1981 and 2002, and analyzing 1,779 cases of proposed policy change in which it was possible to assess reported survey support for the changes by income level. As Gilens and Page summarized the research: “Multivariate analysis indicates that economic elites and organized groups representing business interests have substantial independent impacts on U.S. government policy, while average citizens and mass-based interest groups have little or no independent influence.”

These findings generated extensive discussion both within political science and beyond it. Indeed, we received a number of critiques centering more or less on the replication of the Gilens and Page data. As a matter of current editorial policy, we do not publish straight replications, because our journal’s mission is to publish research and writing that engages diverse perspectives and advances theoretical arguments about important political questions. No article that simply reports findings, and that rests on the veracity of such findings, can satisfy our journal’s distinctive editorial profile and expectations. We receive many submissions that proceed in this “normal science” manner, some of them very good, and we expeditiously review them internally and typically return them to their authors with encouragement to submit their pieces to one of the many journals in our discipline—APSR, AJPS, JoP, CPS—that does tend to publish “findings” in the standard, normal science format. For the same reason, no piece that simply replicates or reanalyzes the data furnished in an earlier article meets our editorial expectations. For we focus our energies on the publication of original research that addresses big political questions, and the methodological drawbacks of any previously published article do not in and of themselves constitute a big political question.

At the same time, many of the articles that we publish are strongly grounded in empirical data. And given the attention that Gilens and Page’s data-centered analysis generated, it seemed appropriate to make an exception to our rule in this case, so long as the critique in question did more than simply reanalyze the Gilens and Page data. Peter K. Enns’s “Relative Policy Support and Coincidental Representation” is such a critique, and so we decided to run it as a “Reflection” along with a response from Gilens and then a final word from Enns. Their exchange centers on the interpretation of data—and both are accomplished scholars who believe that their reading of the data is the right one. But the exchange also centers on certain concepts—“relative policy support,” “coincidental representation,” and of course “democracy”—and on the “fit” between both the data and the concepts, on the one hand, and important features of contemporary political life on the other. Each author draws on a range of sources of evidence and support. Enns cites at least two other recent efforts to reanalyze Gilens and Page’s data, which is widely accessible, having been posted online as “supplementary materials” when their piece was published. The disagreement relates in part to the interpretation of the empirical record, and in part to the implications to be drawn from this record. Gilens maintains that the record of policy outcomes ought to greatly trouble those who subscribe to democratic values. Enns rejects “theories of economic-elite domination” and defends a more classically “pluralist” perspective.

Needless to say, this disagreement does not take place in a vacuum, and implicates much other scholarship on the theme of inequality, including work published in these very pages (Leo Panitch’s “Capital and Politics,” a review essay in this issue centering on four important books, among them economist Thomas Piketty’s Capital—a data-driven work if ever there was one—is also tangentially relevant. So too is Jeffrey Frieden's review of Leo Panitch and Sam Gindin's The Making of Global Capitalism: The Political Economy of American Empire and David Kotz's The Rise and Fall of Neoliberal Capitalism.) Also needless to say, such disagreement will not be resolved here.

Kay Lehman Schlozman et al’s, “Organizations and the Democratic Representation of Interests: What Happens When Those Organizations Have No Members?” is clearly relevant to this discussion. The essay extends the long-standing research agenda on interest groups and inequality developed by Schlozman, Sidney Verba, and Henry Brady (their The Unheavenly Chorus was reviewed in our December 2013 issue). In this piece they draw largely on descriptive statistics to document the changing profile of the U.S. interest group landscape. As they write: “We use data from the Washington Representatives Study, a longitudinal database containing more than 33,000 organizations active in national politics in 1981, 1991, 2001, 2006, and 2011. The share of membership associations active in Washington has eroded over time until, in 2011, barely a quarter of the more than 14,000 organizations active in Washington in 2011 were membership associations, and less than half of those were membership associations with individuals as members. In contrast, a majority of the politically involved organizations were memberless organizations, of which nearly two-thirds were corporations. The dominance of memberless organizations in pressure politics raises important questions about democratic representation.”

James Fishkin, Thad Kousser, Robert C. Luskin and Alice Siu’s “Deliberative Agenda Setting: Piloting Reform of Direct Democracy in California,” reflects on the Deliberative Poll that the authors organized in connection with a 2011 initiative called What’s Next California. Drawing on survey research, descriptive statistics, and transcripts of small group discussions, they argue that the California experiment demonstrates the potential of deliberative forums to improve both citizen knowledge and civic competence, and thus to enhance the quality of agenda-setting and problem-solving in liberal democracies. Here too empirical data is developed, and analyzed, in the service of a broader inquiry into the normative deficits of existing democracy and the possible ways of remedying these deficits.

In “The Influence of Private Health Care Financing on Citizen Trust in Government,” Melani Cammett, Julia Lynch, and Gavril Bilev draw on comparative empirical data to analyze the relationship between inclusive policy outcomes and levels of citizen trust. As they write: “Using individual-level data from the 2008 European Social Survey and country-level health care financing data we analyze the influence of private financing of health care on political trust in twenty-five European countries. Net of known predictors of trust at the individual and country level, we find that trust in government is significantly lower where the health system is financed to a greater degree by private sources.” And in “Reflections on The Civic Culture and The Civic Culture Transformed,” Sidney Verba sympathetically reviews the “post-materialist” research agenda of Ronald Inglehart while reflecting broadly on the progress in the development of cross-national survey research since the 1963 publication of Almond and Verba’s classic The Civic Culture.

Each of the above pieces contributes generally to what we might call the development of empirical research, as conventionally understood, in political science. At the same time, our issue contains other work that is also empirical, though in a broader sense, through the employment of qualitative and historical methods and approaches. Dina Bishara’s “The Politics of Ignoring: Protest Dynamics in Late Mubarak Egypt,” argues that scholars of authoritarianism need to pay attention to ruling strategies of “ignoring” as well as to strategies of repression and concession. Bishara writes: “Drawing on research on the role of emotions in protest politics and on framing and social movements, I argue that ignoring protests can trigger emotional responses that encourage people to engage in protest, such as anger, indignation, and outrage.” She supports this argument through a careful case study of a 2007 protest episode in Egypt involving public employees—tax collectors. The piece draws on extensive fieldwork conducted in Egypt before and after the 2011 uprising, and on semi-structured interviews with more than 60 labor activists, activists in non-governmental organizations, members of officially-sanctioned unions, and academics.

Michael Bernhard’s “Chronic Instability and the Limits of Path Dependence” is a major contribution to “historical institutionalist” research in comparative politics. Bernhard argues that this research “suffers from a certain stability bias in explaining macro-level phenomena such as regime change. Given that large swaths of the globe periodically suffer from periods of chronic instability and weak rule (witness the situation in the contemporary Middle East or Africa, as well as historical epochs in other regions), it is essential for practitioners of comparative historical analysis to develop a set of tools to understand chronic instability… My purpose here is to explore and theorize about the possibility that institutional settlements made at a series of junctures do not generate sets of institutions that are capable of managing the inherent logic of social cleavage and conflict in society. Actors face serial episodes where they have strong autonomy, but may find that stable institutional outcomes elude them.” Bernhard outlines a conception of chronic instability “as multiple, frequent, and connected episodes of disjunctive change”; considers the role of external and internal factors; and then expands on the implications of this conception for understanding a range of cases of instability in twentieth century European history, including Poland and Czechoslovakia during the interwar years, and Germany from Empire to Cold War. As he writes: “The material presented here is a ‘plausibility probe,’ intended to show that the causal mechanisms outlined earlier are useful in analyzing a well-known sequence of chronic regime instability. It is an exercise in congruence testing that connects the presence of the causal mechanisms specified in my theory to a series of failures to establish path dependence. On its own it cannot conclusively validate the theory but can increase confidence that further testing is warranted and worthwhile.” This form of empirical analysis is central to much comparative historical research, even if it does not satisfy the requirements of strict falsification.

Eileen McDonagh’s “Ripples from the First Wave: Monarchical Origins of the Welfare State” also offers an empirically grounded contribution to grand theorizing about regimes and regime transformation. McDonagh argues that: “Most theorists of the welfare state … see its development as grounded in economic imperatives and distributive political conflicts rather than as shaped, determined, or at least grounded in patrimonial histories based on monarchical legacies.” Drawing on a wide range of methods and approaches—from the history of political thought (Machiavelli, Tyrell. Locke) to classical social theory (Weber) to historical institutionalism to large-N quantitative analysis—she proceeds to delineate the “institutional logic” of patrimonialism, outline its likely impacts on the development of state social welfare provision, and then develop and statistically test a set of empirical hypotheses about the importance of monarchical legacies.

If McDonagh theorizes about the long sweep of modern European history and Bernhard focuses on mid-twentieth century European politics, the issue’s three book review essays—Steven Wilkinson’s “Uncertain Glory? India’s Achievements in Perspective,” Maya Tudor’s “Pakistan’s Security State of Mind,” and Aqil Shah’s “States, Organizations and Warmaking: Pakistani Exceptionalism in Comparative Perspective”—center on India and Pakistan, two more recent cases of post-colonial state-building in the global South that loom large in the geopolitics of the twenty-first century.

With Andrea Louise Campbell’s “Family Story as Political Science: Reflections on Writing Trapped in America’s Safety Net,” we arrive at a very different kind of political science empiricism: a personal reflection on the experience of writing a deeply personal, and semi-autobiographical, ethnography of the U.S. welfare state. In her previous work (including How Policies Make Citizens: Senior Citizen Activism and the American Welfare State and, with Kimberly J. Morgan, The Delegated Welfare State: Medicare, Markets, and the Governance of Social Provision) Campbell has employed a range of empirical methods to analyze the welfare state from developmental and public policy perspectives. In her own words: “Trapped in America’s Safety Net tells the story of my brother and sister-in-law as they attempt to navigate the incredibly complex world of social assistance, following a car accident that left my sister-in-law a quadriplegic. It is what anthropologists term an ‘intimate ethnography,’ a work that explores the lives of family members while ‘finding ways of linking the individual stories to larger social processes.’”

Campbell’s comments on the methodological implications of her work are important, especially in light of the objections to the DA-RT initiative recently voiced by many ethnographers (see, for example, the many contributions to the Spring 2015 issue of the Qualitative and Multi-Method Research Newsletter). And they are worth quoting at length: “Political science is an empirical discipline. But as my colleague Regina Bateson has pointed out, although the ‘word “empirical” is commonly defined as “based on observation or experience,” somehow the “experience” side of things has disappeared from empirical political science.’ Having now experienced through close family members various programs I had studied for years, I can say that the lived reality taught me innumerable details and countless profound lessons that no technical analysis could ever reveal … Witnessing my family’s struggles has been a searing experience—and a revelatory one, even if the resultant narrative is far from a standard research design.”

Campbell is not a partisan of one particular style of research, and as a serious scholar she is acutely aware of the tradeoffs involved in most research choices: “There was the analytical challenge of writing an intimate ethnography about American social policy. How representative are Dave and Marcella’s experiences? What do we learn from their situation that we would not learn from a standard academic research design? Conversely, what would a traditional analysis reveal that a single case study cannot?” Her very mode of raising these questions signifies a recognition that different kinds of research can disclose different dimensions of a phenomenon of interest, and that it is thus a good thing that a range of research approaches and perspectives are available. (Frances Fox Piven’s review essay in this issue comments on the limits but also the strengths of Campbell’s book as a contribution to social science knowledge.)

Many centuries ago a rather important political scientist offered a very similar reflection on the nature of political inquiry:

“Our discussion will be adequate if it has as much clearness as the subject-matter admits of, for precision is not to be sought for alike in all discussions, any more than in all the products of the crafts. Now fine and just actions, which political science investigates, admit of much variety and fluctuation of opinion, so that they may be thought to exist only by convention, and not by nature. And goods also give rise to a similar fluctuation because they bring harm to many people; for before now men have been undone by reason of their wealth, and others by reason of their courage. We must be content, then, in speaking of such subjects and with such premises to indicate the truth roughly and in outline, and in speaking about things which are only for the most part true and with premises of the same kind to reach conclusions that are no better. In the same spirit, therefore, should each type of statement be received; for it is the mark of an educated man to look for precision in each class of things just so far as the nature of the subject admits; it is evidently equally foolish to accept probable reasoning from a mathematician and to demand from a rhetorician scientific proofs.”

This conception of political science is no doubt quaint in a great many ways. Yet Aristotle was onto something and we would do well to remember it: The methods of inquiry ought always to be determined by the objects and the purposes of inquiry. Political science is neither mathematics nor physics, neither rhetoric nor poetics. At its best, it furnishes partial and provisional understandings of a dense, opaque, and elusive social world. And just as the world of politics contains a variety of discourses, practices, and institutions, the world of political science contains a variety of empiricisms. There are many ways of experiencing the political world, and of interrogating this experience, and of carefully, rigorously, and critically subjecting it to considered analysis and theoretical scrutiny. The best work in political science sheds new light on experience. One standard by which the best work can be judged to be excellent is because of the ways it draws upon previous work, and offers new insight through the engagement with long-standing ideas and concerns. Such concerns are explanatory. But they are also normative. For the best work in political science, whatever its methodological orientation, is work that is motivated by a question about or problem with the actual political world that matters. To frame and perform any analysis in political science is to make judgments about what features of the world and problems in the world are worthy of study. Such judgments of significance ground all important work in political science. To this extent, our discipline is characterized not only by a variety of empiricisms, but by a variety of political theories which rests on a variety of interpretations of what matters and why. Perspectives on Politics is a journal committed to recognizing this variety, featuring it, and fostering productive conversations in which colleagues can more reflexively approach their work and more constructively understand both the agreements that join us together and the disagreements that make our scholarly life vital and interesting.