Production and Pre-publication Evaluation

Broadening the Definition of Research and the Style of Publication: We encourage APSA and journal editors to treat the scholarship of teaching and learning as a legitimate area of scholarly inquiry in political science, and to consider how to incorporate such studies into APSA's family of publications. For instance, the ASPR and Perspectives on Politics do not prohibit manuscripts on the scholarship of teaching and learning, but they are seldom encouraged or solicited. New publications, whether electronic or print, provide an excellent opportunity to integrate the scholarship of teaching and learning into the rest of the discipline by publishing high quality empirical studies and teaching materials such as research-based simulations. (See also our discussion of the possible future of PS.)

The committee also encourages further development of journal editors' ongoing efforts to ensure that the opportunity to publish is equally accessible to all substantive fields, methodological approaches, normative stances, or research questions. Correctly or not, some political scientists worry that political science journals privilege quantitative and formal methods, the study of American politics, or mainstream political science (however defined) over interdisciplinary work, topics in emerging or smaller subfields, conservative viewpoints, or studies focusing on race, class, gender, and sexual orientation. Some believe that faculty at PhD institutions receive better treatment than do independent scholars or faculty at non-PhD-granting institutions.

Such perceptions or actual biases can create a self-perpetuating cycle if they affect decisions about submission to journals, make it more difficult to find reviewers, or encourage reviewers to recommend that a submission go to a specialized journal or even one outside political science. Any of those outcomes can, in turn, further reduce the visibility of some kinds of scholarship within the discipline. The committee therefore recommends that APSA develop an assessment of these perceptions and publicize its findings widely. We have not developed a detailed strategy for such an assessment; perhaps experts outside the discipline (or political scientists outside the academy) should conduct it, given the sensitivity of the issue and the difficulty in reaching a persuasive conclusion. If an assessment does show unrecognized bias at any stage of the publication process, APSA might consider providing training for editors, editorial assistants, and reviewers to work against it. With or without a systematic assessment, we endorse editors' continued efforts to broaden the array of articles they publish, along a variety of dimensions.

Finally, we urge more attention to the range of possibilities encompassed by an “article” or a “publication.” On one end of the spectrum, on-line materials as part of or accompanying an article could include reproductions of archival material, maps, interview transcripts, musical selections, videos, or other types of evidence impossible to include in a printed journal. On the other end of the spectrum, journals might experiment with brief formats—a several thousand word report of a new finding; a 1200-word “viewpoint” paper that addresses an important topic in political science, politics, governance, or public policy; or even a 600-word “research letter” reporting original research.Footnote ²

Enhancing Peer Review: Pre-publication reviews both help to improve draft manuscripts and provide an essential element of quality control. The value, equity, inclusiveness, and timeliness of reviews speak to core publication values—opportunities for authors and quality—and to the missions of creating scholarship and supporting career development. The peer-review process also is a crucial element of broadening the definition of research, since new pools of reviewers might be essential for evaluation of different kinds of work. As expectations for publication in top disciplinary journals rise and as the number and variety of political science publications increases, however, the process of pre-publication evaluation comes under greater pressure. Three related issues—reviewer fatigue, reviewer exclusion, and double-blind reviews—seem especially worthy of consideration.

We know of no data to determine the accuracy of this perception, but many political scientists are concerned that some scholars are asked to review disproportionately while others are not invited to do so. Journal editors sometimes cannot get the scholars considered most distinguished in the field to review because they are overcommitted; editors sometimes have to make repeated requests before getting a sufficient number of reviews. The problem may be most acute in emerging fields, specialized methodologies, and interdisciplinary research. Conversely, some active political scientists are not requested to do reviews despite being willing and able to do so. These members are perceived to be disproportionately in lower-ranked or teaching-oriented departments; some also may have removed themselves from the publication process because they believe there to be biases against particular topics.

The committee sees four remedies that are relatively easy to implement. We also encourage journal editors to experiment with other responses to the apparent problems of overinclusion and underinclusion. First, APSA could facilitate a more systematic search for qualified and underused reviewers. The association, or a set of journal editors, could establish a joint archive of scholars who are promising reviewers, for example, those who have recently published in the field or who have been recently tenured. Recruitment at annual or regional meetings and in sections could be a source of names for the archive. Editors and graduate editorial assistants could be trained to look in nontraditional places—for example, in community colleges or public agencies, and among independent scholars. Identifying all APSA journal articles and all potential reviewers through a common set of keywords would help in establishing and developing this archive. APSA's membership renewal form could also include a screen inviting people to suggest themselves as reviewers, using the common set of keywords. The archive could include information about how often a person has been asked to review, has accepted or declined, has completed reviews on time, and has submitted manuscripts to journals for review.

Second, editors could engage in more extensive internal review, including a more aggressive use of desk rejects.Footnote ³ This implies a greater burden on editors and more editorial control; a more extensive use of broad editorial teams or an active editorial board could help with both concerns. Since more internal editorial review challenges the traditional assumption of external, blind peer review, it should be used judiciously and with a transparent set of decision rules. Journal editors might agree to some sort of external review of their procedures and decisions after a certain period of time to assure themselves and others that decisions about desk rejects are appropriate and fair. The association might investigate the training programs used by the NSF and National Institutes of Health (NIH) to decrease unintentional biases.

Third, publications could seek further ways to simplify and streamline the review process. Most editors already remind reviewers about deadlines and take considerable pains to get reviews in on time. Editors might experiment with more closed-ended reviews, although this risks approaching a multiple-choice format—which would be more efficient but less effective. We urge the association or the Publications Committee to canvass other disciplines and academic book publishers, and develop a set of suggestions or best practices from them.

Finally, the committee encourages APSA to articulate a set of norms for departments about pre- and post-publication evaluation to incorporate into their tenure and promotion criteria. Training as peer reviewers, conducting peer reviews, and contributing commentary or online dialogue are increasingly valued professional responsibilities. The goal here is to encourage departments and universities to expand their understanding of faculty productivity and disciplinary engagement; doing so would contribute to the mission of career support and development as well as improving the quality of evaluation in APSA publications.

The committee also considered more substantial changes in the review process. Some journals in other disciplines, for example, are experimenting with asking authors of rejected manuscripts if they want the reviews passed on to another journal. The logic here is that an article might be well suited for a specialized journal or a different journal than the one to which it was submitted, and the same reviews might yield a “revise and resubmit” or possibly even acceptance in a different publication context. Committee members raised concerns about this practice, so we would recommend proceeding with caution. Another possible innovation would be transforming reviews, where appropriate, into published commentary on an article. That would encourage high-quality reviews and may provide an incentive for reviewers to respond more quickly. However, it would require editors to make an additional set of complicated decisions, and could slow publication if the review being transformed into a commentary requires revision, or itself calls for review. Again, this is a suggestion rather than a committee recommendation.

The issue of double- or single-blind peer review is probably a question for journal editors or the Publications Committee to decide, but the planning committee offers two plausible positions for further consideration. On the one hand, while the system of double-blind reviews has problems, it has offsetting virtues. Reviewers might know authorship of some manuscripts owing to presentation of a paper at conferences, an article's availability on SSRN or other websites, or simply a Google search. Nevertheless, studies have documented that gender and ethnic bias in evaluation can occur even when only names differ, for example, an article by “Tanya Juarez” might be evaluated differently than one by “Thomas Jones.” Given this, it would seem that allowing reviewers to see an author's name could introduce bias without any clear benefit.

On the other hand, APSA journals might consider going to a single-blind review process, as the economists have done. Since the identity of most authors is only a Google search away, an author's name and identifying information are rarely in doubt to a reviewer who wants to know. The issue of gender, ethnic, and other biases, which is real, could be better addressed through the editorial teams and the monitoring of some key factors, much as is already done with field coverage and balance. Furthermore, moving to a single-blind review process would eliminate confusion and ambiguity. At present, editors and the author(s) must guess whether the reviewer did the Google search or not, and then must guess about how much that possible Google search did or did not influence the review. A single-blind review process would leave no ambiguity about what reviewers know about an author; after a few years, the association could analyze whether gender, ethnic, small-school, or other biases have been introduced or exacerbated. More speculatively, the author(s)' identity conveys some useful information to reviewers regarding at least the prior perspective and biases of the author(s) or the quality of the analysis and interpretations; again, knowing that this information is available to a reviewer can help all parties evaluate the context from which a review is written. Note that it would be important for journal editors to make clear to writers and readers alike just what is “blinded” and to whom.

In general, pre-publication peer review remains the gold standard for enhancing the production of scholarship and assuring the quality of articles and reports. Peer review will only become more complicated as the association moves toward a more differentiated set of publications or engages with new technologies for presenting material, so some discipline-wide discussion of how to do it effectively, efficiently, and fairly would be valuable.

Registration Standards: Research analyzing the distribution of test statistics in published articles suggests that political science suffers from severe reporting or publication bias. Results are much more likely to make it into print if they reach significance levels just above critical cutoffs of 90% or 95%, and less likely if they fall just below these thresholds. This raises concerns about the quality of articles in the discipline's journals.

For one thing, individual published results may be wrong; if estimated effects are only published conditional on being “statistically significant,” then these estimates are likely to be overestimates. In addition, even if all individual published estimates were correct, the systematic underreporting of null results would produce an inaccurate record of general effects. That is, scholars, students, and external audiences will start believing that X causes Y because they only read studies in which X causes Y, not studies in which it does not. Furthermore, pressures to generate significant or counterintuitive findings can induce researchers to condition their analysis strategies on patterns they observe in the data; that, in turn, can lead to intentional or unintentional data fishing. The result is false and fragile findings.

One response is registration standards, in which researchers describe and make public the analyses they plan to conduct before seeing the results of those analyses. Creating a public record of intended analyses reduces the scope for adjusting analysis plans to select for significant findings, and creates a record of studies in a given area even if the published record is not representative of that population.

Registration is now the norm for experimental research in medical sciences; major medical journals now treat registration as a precondition for publication of results from a randomized control trial. The American Economic Association has recently launched a registration tool (socialscienceregistry .org). The research funding group 3ie has launched a registry in the area of development economics. Informally, individual researchers have been registering designs in political science through the website of EGAP (Experiments in Governance and Politics, e-gap.org) or through Harvard University's Dataverse (http://dvn.iq .harvard.edu/dvn/).

This committee recommends that APSA develop a set of principles on registration for adoption by the Publications Committee or individual journal editors—and, hopefully, by other political science journals. To this end we highlight several considerations that will help to ensure that APSA's registration principles are consistent with the needs of the broader discipline:

Dissemination, Engagement, and Post-publication Evaluation

Expanding Channels for Dissemination, and Incorporating Technological Innovation in Presentation: We encourage everyone involved with APSA publications to continue working to expand channels for disseminating publications to new audiences or to extend dissemination and integration among current APSA members. However, the association does not have a mandate to develop the technology for innovative distribution and, given inevitable budget constraints, the committee considered but does not recommend hiring an in-house technology expert (beyond those already on the staff). Therefore we suggest an alternative way to promote technological innovation in broadening and deepening the audience for publications—a small grants program or annual awards (one each at the two annual conferences) for the most exciting or useful innovation in enriching APSA publications' readership. The goal of the grants program or awards would be to facilitate and reward experiments in dissemination and outreach, highlight their importance to authors and users, provide a signal to employers, and spread the use of especially promising ideas.

We also encourage the continued expansion of online publication of APSA products. “Online” does not mean that editors give up the important curatorial task of putting issues together with themes having coordinated content and sustained treatment; it does, however, open possibilities for expanded readership and electronic integration across items and journals. Editors might choose to post articles online on a rolling basis (see, for example, Oxford University Press's new strategy for presenting chapters in its Handbooks series), or they might decide that the journal issue goes live only when all parts are in place and connected through a table of contents or editor's introduction articulating their connections. Different publications might experiment with different choices about the timing of release, and about the appropriate iteration of online and print publication.

Incorporating technological innovations in dissemination is closely linked to, but not identical with, incorporating technological innovation in presentation of ideas and findings. On the one hand, scholarly content increasingly employs formats beyond the traditional modes of journals and books (and even these traditional publication routes can be enhanced with new media and online features). On the other hand, APSA has neither the responsibility nor the resources to go very far down the road of developing these new formats or features. Thus the committee comes down in roughly the same place with regard to new forms of presentation as it did with new forms of dissemination—a suggestion for small grants or awards at the two annual conferences, with the goals of signaling quality to readers and employers, encouraging new ideas and experiments in presenting research findings, and making the best innovations widely visible. For both dissemination and presentation, we note the importance of the forthcoming Presidential Task Force on Public Engagement as a source for ideas, expertise, and research findings.

The Future of PS: The planning committee recommends consideration, presumably by the Publications Committee and APSA staff, of possible restructuring of PS. The journal's current mission combines research on teaching and learning in political science (some items peer-reviewed), symposia and peer-reviewed research articles about currently important political and policy issues, articles about the discipline of political science (some items peer-reviewed), and news about individuals, departments, and the association. Each component serves an important function, but each is sufficiently distinct that they need not be united in one publication. Arguably each element of PS's mission would be best served by separating them.

First, news about individuals, departments, and the association should remain within APSA staff oversight. That publication could become online only, with electronic links to the websites of the individuals or organizations mentioned or to other online material. It could also include “columns” (that is, regular contributions) or guest reports by an array of political scientists—inside and outside the academy—that surveyed, reviewed, and generally kept track of the blogosphere and other technological innovations. Ensuring that many of the contributors are located outside the United States might expand APSA's international reach.

Second, components that focus on teaching and learning could become a stand-alone journal, with an editor and editorial assistants at a teaching-oriented college or university analogous to the editorial set-up of Perspectives on Politics and the APSR. Such a journal should be available online, and perhaps be online only; along with peer reviewed articles, it could usefully include links to syllabi collections, simulations or other teaching tools, and datasets or other resources useful in a classroom. Further details of developing that publication would evolve during the recruitment of an editor and editorial board.

Third, the excellent contributions to the research section in PS may not be receiving the readership they deserve since they appear in a publication whose profile is mixed. If PS were reconfigured, research articles and symposia could be incorporated into one of the existing journals, or—since they tend to be brief, accessible, and current—they could become part of a new publication focusing especially on public visibility and impact. Alternatively, if the association creates its own blog or develops ties with extant political science blogs (see discussion below), the research section of PS could be migrated there. Finally, some or all of the research symposia now in PS might be appropriate for the APSA's new e-journal, depending on whether it is developed and how it is configured.

The Future of Book Reviews: The question has been raised about whether book (and other) reviews should remain within Perspectives on Politics or be located in a stand-alone journal. The committee did not come to a clear conclusion on that issue. We have several reasons for abstaining from a recommendation: some disagreement and ambivalence within the committee; deference to the current editor's clear vision, combined with a recognition that future editors might develop a different vision; a sense that this decision belongs in the purview of the Publications Committee and possible future editors; and recognition of the interdependence of several choices about APSA publications. There is no immediate reason to decide about a change, and other publication issues arguably have greater priority or urgency.

A decision about the appropriate place for reviews should be put into the context of a broader debate about the identity and purpose of Perspectives. The current motivating vision for the journal is as a “political science public sphere”; the review essays, critical dialogues, and research articles are all aimed at creating a rich discourse within the broad and fragmented field of political science. In this vision, book reviews and symposia enable members of the discipline to publicly discuss big ideas presented at book length, and they are often the main way for political scientists to stay abreast of new topics and debates. Removing reviews from Perspectives arguably would diminish and narrow its vision and effectiveness.

An alternative identity and purpose of Perspectives might be to pursue a broad readership outside as well as inside political science, i.e., to seek to engage a politically or policy-oriented public. In this conception, Perspectives would focus more on publishing articles of broad public interest and relevance, and less on cultivating an internal political science public sphere. An extensive set of reviews of academic books is less essential to the mission of public engagement, and could even deter potential readers outside the academy. (We note, nonetheless, that magazines such as The Economist and The New Yorker publish a few reviews or review essays in each issue, so a newly configured Perspectives might well include selected symposia or review essays.)

This sort of redirection of Perspectives, or other alternative visions, would likely imply shifting the composition and focus of the articles published in the journal. Any decision about Perspectives' future, and therefore about the role and location of reviews, needs to be interdependent with current APSA discussions about launching an e-journal, or future discussions about engaging more actively in the blogosphere or moving research articles and symposia from PS to another APSA journal. Even if the current model remains in place, the committee urges even more attention to innovative ways of presenting book and other reviews online. Debate and dialogue might be further encouraged, for example, through moderated comments and structured discussions involving reviewer(s), author(s), and readers. As we noted above, other material could be reviewed, such as museum exhibits, movies, websites, datasets, and archives. Investigation for users as well as integration across items and journals could be enhanced by imaginative and more extensive use of the capability for electronic links. Note that these or other innovations should occur in the context of a broader move toward creating online interaction as a core part of APSA journals, not as something directed at Perspectives in particular.

Perspectives is a healthy and well-liked journal, which is not in evidently urgent need of reform. The decision about reviews in Perspectives, therefore, depends mainly on how the association decides to pursue broader public engagement and how it understands the mission of the journal—as well as, of course, preferences of current and future journal editors and authors. This is another arena in which attention to the choices of other disciplines could be illuminating, since they cover a wide range of options. Sociology, economics, education, and psychology have stand-alone review journals (the latter is online only); history has both a stand-alone journal of review essays and reviews in the two flagship journals. Reviews are integrated into the flagship journals of anthropology, American studies, and public administration. Anthropology is considering development of a digital platform for book reviews. Furthermore, APSA sections and other political science professional associations or publishers are experimenting with various forums for reviews, ranging from inclusion in the central journal to stand-alone online reviews that roll out when complete.

Stand-alone journals have more space for review essays, symposia, or additional reviews and can be more flexible with regard to timing and format. But it would be important to determine how widely they are read, whether they are read in relation to each other, and how much intradisciplinary engagement they generate. Financial considerations will also matter, although they are outside the purview of this report.

Open Access: Scholars like open access because they want others to find, disseminate, and use their work. Open access to journal articles can enhance teaching and learning, generate public visibility and impact, and foster a career if an author can show interest in his or her writings to deans and departments. Open access enables students and scholars around the world to benefit from the resources enjoyed by political scientists in well-off colleges and universities, and can spur international dialogues even wider than Perspectives' political science public sphere can currently hope to do. The United States' federal government and the national governments of other countries are making increasingly strong demands that data or publications from publicly funded research be made publicly available.

Under current financial arrangements, however, open access can be cost prohibitive. The rough rule of thumb is that the average revenue to journal publishers per article in traditional publications is $5,000; publishers will not, and cannot, relinquish all of that revenue. Neither can scholarly associations, whose income depends in large part on contracts with publishers that, in turn, rely on subscriptions to journals. In short, the question of open access raises broad and deep issues, leading one expert to claim that “a comprehensive OA paradigm is pretty much a frontal assault on professional societies” (http://scholarlykitchen.sspnet.org/2013/08/01 /open-access-and-professional-societies/).

APSA and its members need a much more developed analysis of the financial, legal, and intellectual implications of open access than this report can offer before concrete recommendations are made. We therefore limit ourselves to endorsement of the principle of open access—but also to an equally strong urging of the Publications Committee, Council, APSA staff, and others to systematically analyze the opportunities and risks attendant on a regime of open access, especially in light of the other worthy demands on APSA resources.

Ungating Selected Articles: Ungating provides a short-term solution to the desire for open access, before a more extensive policy is developed. Most generally, public engagement as a core ASPA value entails making the highest quality political science research available and relevant to an interested public. But it is unrealistic to expect even interested nonacademics to seek out scholarly journal articles and to overcome significant logistical or commercial barriers to access. Thus real impact will require appropriate platforms and approaches to drive attention to these articles and to convey their significance even in an intensely competitive, information-saturated marketplace of ideas.

Ungating access to journal articles when appropriate to their public discussion is the most obvious step that APSA can take. APSA does work with publishing partners to selectively ungate articles for marketing and promotional purposes; we recommend that it work even more closely with journal publishers to create an easy, smooth, fast, and egalitarian process to allow bloggers to temporarily ungate any APSA journal article about which the blogger is writing. The committee suggests consideration of a two-tiered process: a top tier of pre-approved, established blogs with “green light” power to temporarily ungate articles discussed on their blog; and a second tier that involves a relatively simple and rapid process for noncertified blogs to request ungating.

Blogs: A more extensive response to the issues involved in open access—and an important innovation in its own right—would be for APSA to encourage and support the trend toward political science blogs. They can translate disciplinary findings into topical, readable prose in a timely fashion and stimulate informed public discussion of issues. We see three potentially viable models for APSA to support blogs, each with its own problems and advantages. The models are not mutually exclusive: two or three could be pursued simultaneously. Each model provides access to writers and readers, and takes advantage of APSA's ability to brand a publication as high quality. They differ in the degree of commitment of associational resources of time, reputation, and money. These three models are:

The committee does not take a position on which model is most desirable or feasible; among other things, there are complex tradeoffs in budgetary and staffing decisions. We do, however, strongly recommend that the association move toward choosing which of these (or some other) model will best foster quality, career advancement, opportunities for writers and the capacity for readers to investigate, and public engagement and influence.

Regardless of how the association moves with regard to blogs, we further recommend development of a short course or panel to be regularly offered at each of the annual conventions. For scholars who want more public connection, a panel with people who are regularly engaged or who write on blogs could be useful and interesting. Panelists could explain how they got started, ways they make their work more accessible, and mistakes to avoid. A panel could be offered at low cost, and a more extensive short course would pay for itself if there is demand. The association also might see if there is sufficient commitment to develop a public engagement section, which could produce a regular newsletter and be allocated a panel or two at each annual convention.