1. I will use “American political development (APD) community” to denote those who self-identify with that subfield and “NOMINATE community” to designate those who routinely use the algorithm. Although these communities are, as the contributors to this symposium generally recognize, distinct in some respects, they also overlap because some scholars belong to both groups and thus share a common interest in historical analysis that accurately portrays and interprets the political past. While my use of these labels indicates the respective general theoretical and empirical orientations of the two communities, my critique of the NOMINATE system is intended to address their shared theoretical and empirical concerns.

2. I will refer to the articles in this symposium by the names of their authors. For example, “Bateman and Lapinski” will refer to David A. Bateman and John Lapinski, “Ideal Points and American Political Development: Beyond DW-NOMINATE.” On p. 169, Bateman and Lapinski write, “If the policy space were stable, the liberalizations to the [Social Security] program should be accompanied by midpoints that moved to the liberal side of the space.” On p. 144, Devin Caughey and Eric Schickler similarly ask, “How should congressional scholars, particularly those with a historical bent, choose an approach to measuring spatial change over time?” Only after extensive induction into the norms and conventions of the NOMINATE community could someone fully understand the metaphorical reasoning in these passages. Even after this induction, the actual content of these passages would remain metaphors with an unspecified (and perhaps unspecifiable) connection to the empirical reality they purport to represent. The induction into the NOMINATE research community thus requires both training in how to use the terminology of the community and the acceptance of an obscuration of how that terminology actually applies to empirical reality.

3. Thus, Phil Everson, Rick Valelly, Arjun Vishwanath, and Jim Wiseman (hereinafter Everson et al.) are only partially correct when they state that “NOMINATE … reliably scales legislators by their locations in so-called issue space within each and every Congress (p. 98).” Although the algorithm is certainly reliable, the question is whether these locations actually have meaning.

4. How the algorithm actually processes votes need not detain us here. As the other articles in this symposium illustrate, there are many alternative ways to tweak the algorithm, each of which alters the manner in which individual voting records are related to one another.

5. For example, see Figure 8 on p. 110 of Everson et al., and imagine that the points in the two-dimensional display did not have partisan labels.

6. Those with long experience with the patterns produced by roll call votes in the U.S. Congress might object that they would be able to recognize those patterns when juxtaposed against those derived from weather stations. But they would simply be assuming that rainfall would not produce the same patterns while failing to demonstrate that the algorithm had conveyed political meaning in producing the roll call displays. To understand why that is the case, imagine that the same analysts were presented with patterns produced by roll calls in the Israeli Knesset and asked to distinguish them from rainfall patterns arising from monsoons in India. If the analysts still insisted that they could tell the difference, they would be either supremely overconfident or had already studied displays that they knew were produced by one or the other of these data sets.

7. If the analysts knew the data set contained roll call votes, they would “see” the array one way. If they knew the set contained rainfall records, the array would be “seen” another way.

8. This is probably what Everson et al., mean when they say that “NOMINATE scores are not true values of anything real” (p. 106). This statement, however, leaves readers in a quandary as to what they think the scores actually are.

9. There is, in fact, no correct labeling decision because the statistical artifacts produced by the algorithm do not directly correspond to any observable aspect of social and political reality. However, as discussed toward the end of this article, that is a slightly different point.

10. One way of thinking about this problem is to imagine a group of people attending a cocktail party. Now further imagine that these people are arrayed in clumps in a room according to their social relationships with the other people, with individuals occupying certain spots depending on their relationships with the others. Now even further imagine that one of these people walks out of the room (i.e., is removed from the data set). At least some of the remaining people would change positions in the room because the social relationships available to them had been altered. If a cocktail party algorithm could recalculate the positions of the remaining individuals on the basis of their social relations to one another, this would be a very interesting and potentially fascinating exercise in understanding how cocktail parties evolve. However, it would be the dynamics of the group that would be explained, not the individual identities and personal histories of people attending the party. One of the major problems with the NOMINATE algorithm is that it behaves as if it is analyzing evolving social interaction in a cocktail party but purports to interpret the points occupied in the room as if they were rigidly autonomous identities.

11. Everson et al., contend that the “fact” that there are two dimensions “was a discovery of the NOMINATE algorithm” (p. 107). To label the dimensions produced by the algorithm a “discovery” is to claim that these artifacts were and are somehow immanent facts in the structure of the world, as opposed to the tautological products of statistical processing. They thus misidentify an essentially arbitrary definition as something independently existing in social and political reality.

12. The necessity of linking theoretical concepts to empirical reality has frustrated model builders for decades. For example, in “Modernizing Political Science: A Model-Based Approach,” Perspectives on Politics 5, no. 4 (December 2007)Google Scholar, Kevin Clarke and David Primo maintain that models should be regarded as a “representational objects and less like linguistic entities” (p. 742). If we do this, they contend, models are not true or false but, instead, more or less useful. The problem is both (a) that models are linguistic entities (because they are described in language and have no meaning or existence outside of language) and (b) that their application through use is also thoroughly ensconced in language as well, with the consequence that they are also true or false in at least two senses. First, to the degree to which they purport to be, in fact, models of reality, they must represent that reality in language. Second, to the degree to which they allow successful prediction, the models must link their analytical lexicon to terms depicting empirical reality (pp. 741–53).

13. On p. 149 of their article, Bateman and Lapinski note that one of the “obstacles for integrating” the NOMINATE system “into historical analyses” is that “the meaning of the dimensions estimated by DW-NOMINATE is unclear and unstable over time.” However, in the rest of the article they appear to hold out hope that it might still be possible to link the first dimension to language describing empirical political, including historical, reality.

14. As in this case, most work utilizing the NOMINATE system assigns the label “score” to the coordinates produced by the algorithm and thus, somewhat surreptitiously if unconsciously, converts the bare statistical artifacts into another metric such as “ideology.” However, while Lee does call the artifacts “scores” in this passage, she is otherwise calling the practice into question.

15. For example, the existence of an ideological continuum along which “conflict over economic redistribution” can be arrayed is an assumption that arises entirely outside of the algorithm itself. If we introduce such assumptions, we can always attach meaning to the statistical artifacts generated by the algorithm. The question is whether these assumptions are logically entailed by the construction of the algorithm itself. If not (and they are not), then they are essentially arbitrary. As a result, we should search for a meaning that is logically entailed by the algorithm. To have empirical meaning, that entailment must both be logically deducible from the construction of the algorithm and be deployable in an analytical statement (e.g., in language that describes actual human behavior). As we shall see, the algorithm does not have these properties.

16. Although Lee does not actually say so, her critique strongly implies that mixing together different kinds of roll calls into one data set results, when processed by the algorithm, in coordinates that have no meaning whatsoever. See, for example, this passage on p. 126:

This passage more or less concedes that the algorithm will generate statistical artifacts regardless of whether or not those artifacts mean anything. Perhaps more importantly, note the reference to an apparently Platonic notion of a “true dimensionality of the space.” How would we come to know this true dimension? What could it possibly reference with respect to the social reality of politics and political contestation?

17. I actually think this is impossible because party members often believe and act on the belief that ideological consistency maximizes the prospects for electoral victory. I go along with Lee here solely for the purpose of interrogating her argument.

18. And, even then, Lee has been selective in the weight she places on the works in this literature, ultimately placing more emphasis on those that agree with her thesis. The literature itself, as she notes, is divided and contentious.

19. See pp. 125–126 of Lee's article.

20. See, for example, Caughey and Schickler, pp. 138. While Caughey and Schickler offer many interesting and important observations on the relationship between progressivism and party organizations in Congress during the 1920s, I would contend that their discussion of the algorithm itself, including their attempt to distinguish NOMINATE coordinates from ideology, does not appear to contribute to a better understanding of that period.

21. The tautology arises because the definition of the terms is mutually self-referential: Polarization is this particular relation between numerical coordinates and this particular relation between numerical coordinates is polarization. The pattern defines the term and the term defines the pattern. Within the social conventions observed within the NOMINATE community, this identity is commonly accepted. For example, Everson et al., contend that NOMINATE generates “measures … that reveal and track party polarization over time” (p. 98). However, as Lee also contends, most interpretive practice attempts to distinguish between statistical artifacts and the reality that they purportedly represent. To do so, we need an independent conception of polarization arising out of some other body of empirical evidence.

22. This is a reference to the Tractatus, not Wittgenstein's later work. See Wittgenstein, Ludwig, Tractatus Logico-Philosophicus (New York: Routledge & Kegan Paul, 1961; originally published 1921)Google Scholar.

23. Lee, pp. 125, 126.

24. In Lee, Frances, Beyond Ideology: Politics, Principles, and Partisanship in the U.S. Senate (Chicago: University of Chicago Press, 2009)CrossRefGoogle Scholar, Lee has offered a definition of ideology in connection with her critique of the NOMINATE system (pp. 26–27, 49–53). However, that discussion does not directly address the problems raised in the text.

25. Caughey and Schickler suggest that “it is often possible to make use of the scores without giving them an ideological interpretation” (pp. 128–129). However, they do not say what the coordinates mean if they are not given an ideological interpretation or when and if they might be meaningless. The latter is particularly important to consider because the analyst must demonstrate that the coordinates can have meaning before meaning is assigned.

26. Poole, Keith T. and Rosenthal, Howard, Ideology & Congress (New Brunswick, NJ: Transaction, 2007), 3 Google Scholar.

27. Bateman and Lapinski undercut the reference to Converse in their article by observing that “where Converse [emphasized] ‘ideas and attitudes,’ Poole and Rosenthal define ideology as the stable continuum that enables a prediction of who will vote with whom—it is ‘fundamentally the knowledge of what-goes-with-what’—rather than a set of ideas or issue positions that are characterized by their functional interdependence” (p. 153). Also see note 22. Converse, Philip E., “The Nature of Belief Systems in Mass Publics,” in Ideology and Discontent, ed. Apter, David E., (New York: Free Press, 1964), 206–61Google Scholar.

28. Poole and Rosenthal, Ideology & Congress, 5.

29. Bateman and Lapinski, p. 152. Also see an unpublished manuscript by David Bateman and Josh Clinton, “A House Divided? Roll Calls, Polarization, and Policy Differences in the U.S. House, 1877–2011,” p. 11. Caughey and Schickler similarly note that “NOMINATE scores provide a statistical summary of legislators’ voting behavior. The scores themselves do not have any inherent meaning independent of the theoretical and substantive framework that we use to interpret them” (p. 128). This is equivalent to saying that we should impute meaning to the scores by accepting the conventions of the NOMINATE research community. While they encourage members of that research community to be more critical in their interpretation of the coordinates within those conventions, they still endorse the prior assumption that NOMINATE scores must mean something more than “a statistical summary of legislators’ voting behavior.”

30. Bateman and Lapinski, p. 153.

31. For example, Everson et al., contend that “the NOMINATE algorithm … produces scores akin to an imaginary interest group rating machine that had been in operation since the First Congress” (p. 98). We would have to wonder what ideological qualities this imaginary interest group would have to possess in order for this clearly trans-historical claim to be valid.

32. Bryan and McKinley never served together in the House of Representatives. However, the former was in the 52nd Congress (1891–1893), and the latter belonged to the 51st Congress (1889–1891). Because those using the NOMINATE system routinely compare members in adjacent or nearly adjacent Congresses by relating their voting patterns to other members, the fact that their periods of service did not coincide does not present a problem in this comparison.

33. It is not at all clear what the coordinates would mean if they did not somehow measure ideology. The most common alternative explanation would be that they measure party loyalty (see Caughey and Schickler, p. 135), which would, from an analytical perspective, be rather pedestrian because much simpler and less ambiguous statistical measures of party loyalty in congressional voting are available. So we will stick with the ideological interpretation in this example.

34. Prior to and during the Civil War, policies such as the Homestead Act, federal aid to land grant colleges, and the creation of a national banking system, not to mention opposition to the expansion of slavery, would have been deemed “conservative” because, as an empirical pattern, the supporters of these policies were located on the right side of the array. (For displays on the alignment of slavery, see Poole and Rosenthal, Ideology & Congress, pp. 121–27. The other policies during the Civil War period generate very similar displays.) Similarly, during the 1890s, support for federal aid to education, federal pensions for Union soldiers, the national bank system, and federal enforcement of voting rights would have been deemed “conservative” positions. Without going into details, it should be obvious why deeming these policies “conservative” wreaks havoc with the pretense that ideology, as Poole and Rosenthal have defined the term, possesses significant policy consistency or attitudinal coherence throughout history.

35. Caughey and Schickler are on the right track when they “highlight the limitations of NOMINATE scores for analyzing ideological conflict and change over the broader sweep of American history” because they do not respect ideological alignments “as they were understood by political observers at the time” (p. 129). However, they still insist that the coordinates represent ideological positions if the ideological content is historically contextualized.

36. Bateman and Lapinski, p. 162.

37. Caughey and Schickler, p. 129.

38. To a much greater extent than in explanations of mass voting behavior, ideology has become an analytical “primitive” in the study of Congress: an explanatory variable that is either left unexplained or, when the analyst is compelled to offer an account of its origins and content, folds back upon itself in the form of a tautology. For the increasingly salient role of ideology as an explanation for congressional behavior, see Lee, Beyond Ideology, chap. 2.

39. Fowler, Linda L., “How Interest Groups Select Issues for Rating Voting Records of Members of the U.S. Congress,” Legislative Studies Quarterly 7, no. 3 (August 1982): 405, 406CrossRefGoogle Scholar.

40. Ibid., 405, 406, 412.

41. Mayhew, David, “Events as Causes,” in Political Contingency: Studying the Unexpected, the Accidental, and the Unforeseen, ed. Shapiro, Ian and Bedi, Sonu (New York: New York University Press, 2007), 124 Google Scholar.

42. Converse, “The Nature of Belief Systems,” 257, n. 15.

43. Although he accepts interest group ratings as “measures of legislative preferences” that can be used in formal models of legislative behavior, Keith Krehbiel does not accept an ideological interpretation the ratings. However, he does find them useful for determining the “ideal points” of members of Congress. Although we might, as indicated in the next section, quibble with his conception of ideal points, his interpretation of the ratings as indicators of preference clusters is otherwise consistent with the social interpretation in the text. See Krehbiel, Keith, “Deference, Extremism, and Interest Group Ratings,” Legislative Studies Quarterly 19, no. 1 (February 1994): 61–77 CrossRefGoogle Scholar.

44. ADA scores are thus not the empirical manifestation of ideological beliefs that are somehow immanent in reality. They are, instead, a reflection of the political priorities of an organization that, among other things, presents its interpretation of “liberalism” to the public as a claim on the attention of an audience. We should not, for that reason, be attempting to discern whether NOMINATE or ADA ratings or any other scoring method more accurately and completely encompasses “ideology” but, instead, be studying how and why these interpretations are deployed in political contention.

45. Caughey and Schickler recognize that the ratings of progressivism that they compare with NOMINATE coordinates play a role in congressional social and political reality in a way that the latter do not. However, they also write, “This is not to say that political actors’ perceptions of the relevant cleavage necessarily trump other potential conceptualizations” (p. 132, n. 19). Maybe so, but should not the social perceptions of progressivism by members of Congress trump the brute imputation of ideology as the substantive meaning of NOMINATE scores? For example, when the Conference for Progressive Political Action (CPPA) “identified seventy-five roll call votes in the Senate from 1919 to 1924 that it used to evaluate the progressive bona fides of senators” (p. 132) should not this be accepted as evidence that (1) the relevant individuals in the CPPA possessed social knowledge of group relations that, in turn, dictated their selection of these roll calls; and (2) that this same social knowledge of group relations shaped the policy positions for those senators who oriented themselves, positively or negatively, toward the CPPA? And, for this reason, is not this evidence superior to that derived from the brute imputation of ideological significance to NOMINATE scores?

46. Their dual commitments to historical context and the NOMINATE system, however, sometimes entangle their argument in circularity. For example, Caughey and Schickler write, “Clearly, the linear conservative trajectory implied by first-dimension DW-NOMINATE does not match Bilbo's biographer's assessment that the senator's liberalism ‘waxed rather than waned’ in the late 1930s before he turned sharply to the right in the 1940s” (pp. 141–142). They then note that the numerical coordinates produced by their alternative item response theory (IRT) algorithm does emulate the pattern described by Theodore Bilbo's biographer. However, if we are evaluating “fit” according to the account provided by Bilbo's biographer, then we already know that Bilbo's trajectory was something like the IRT model. That makes the model redundant. Even more importantly, the successful emulation certainly does not imply that the IRT model is a better or more sophisticated measure of Bilbo's ideological trajectory than that derived from the biographer's thick knowledge of the senator's historical context and social relations. In fact, in terms of then contemporary ideological alignments, Bilbo might even be (and probably was) a deviant instance and the success of the emulation would, in that case, be deceptive as evidence that the IRT model is generally superior to the NOMINATE algorithm. Caughey and Schickler also say that the “point of the foregoing analysis is not that the IRT estimates are correct and the DW-NOMINATE scores are not” (pp. 142–143). In this case, we should ask what is the meaning of “correct”? The self-understanding of political actors (including the interpretation of their biographers if they had one) or brute deductions from the algorithm? If the latter, how would we choose between the IRT and NOMINATE algorithms?

47. Bateman and Lapinski make a somewhat similar point when they say that DW-NOMINATE “scores are agnostic to other sources of information, and thereby impose an assumption that legislator voting patterns are invariable to historical context, to institutional change, and to policy development” (p. 149). This indifference to historical context thus ignores all those social relations that position the member in an ongoing legislative process. Lee similarly hints at the social context of legislative deliberations when she notes that “congressional voting is highly structured, not subject to much cycling or instability. Put more simply, members of Congress are members of political parties, and parties are coalitions of societal interests” (p. 126). Here Lee suggests that the social context within which members of Congress deliberate discourages the appearance of otherwise theoretically possible forms of preference revelation (such as those underpinning Arrow's impossibility theorem).

48. In some respects the theoretical status of an ideal point is similar to that of a market price in a rational expectations perspective. Both assume the processing of perfect information over an infinite range of possibilities and alternatives in which no one, not even the analyst, can identify what those possibilities and alternatives might be. For that reason, the cognitive mechanisms through which individuals evaluate legislative choices in order to maximize the utility of their policy bundle necessarily remains a mystery. That mystery parallels the rational expectations approach to markets where traders are said to rationally maximize both the individual and social utility of a product through the mental processing of information that is not fully available to any of them.

49. Bateman and Lapinski propose, as an alternative, that each member “has a comparable ideal point for every week in which he or she was a sitting member and in which Congress was in session” (p. 166). Following that logic, each member could also be assigned an ideal point for every moment in which they cast a vote. This would more exactly align the NOMINATE coordinates with their ostensible theoretical meaning but would also basically maintain the tautological identity between the scores and the content of an ideal point.

50. Everson et al., skirt this issue when they state that “legislators, being professional politicians, know rather precisely what they want—and by the same token what they do not want out of the legislative process. In other words, they have what is known as an ideal point when they go about their business of negotiating” (p. 99). The question is not whether members of Congress have plans and priorities when they legislate (they certainly do), but whether they know precisely what they want and have perfect knowledge with respect to how to get it. If they do not have these things, then an ideal point is nothing but a Platonic fiction.

51. When a member weighs the utility of present and future policy choices, he or she must consider the uncertainty that necessarily attends future policy possibilities. We can assume that the member somehow discounts future utilities. But we cannot assume that the member can do anything more than assign a range of probabilities to these future possibilities as he or she calculates that discount. That makes the bundle of utilities itself a range. The conception of an ideal point is thus rendered incongruous with reality because it purports to precisely identify the summed utility of a member's bundle of present and future policy choices. Bateman and Lapinski implicitly recognize this problem when they say, “The DW-NOMINATE scores show the southern Democrats beginning to drift rightward during the 1920s, but … these scores are problematic because information from later in southern Democrats’ careers will bleed into [the] earlier period” (p. 158). This bleeding into the earlier period is not problematic because members of Congress do not anticipate a future when considering their votes in the present but because they anticipate that future imperfectly, and for that reason, their behavior in the future cannot be read backwards as a precise representation of how they anticipated that future in the past. Their solution to this problem is simply to statistically assume that members do not anticipate a future. Caughey and Schickler tackle the problem from a different angle but make even less tenable assumptions (pp. 139–140).

52. We would have to first calculate the ideal point of the leader and then assign it to the follower as well. However, in this social relationship, the leader must also take into account the policy desires of the follower because amassing and maintaining a substantial following maximizes the leader's ability to reach his or her own legislative goals. This dialectical relationship has no stable solution if we (reasonably) assume that the leader is also attempting to attract new followers as well. In fact, as we well know, one of the most common traits of effective leaders is the ability and willingness to respond opportunistically to unanticipated events.

53. Caughey and Schickler say that the NOMINATE system's “summarization” of legislators’ voting behavior “takes on additional meaning to the extent that the statistical assumptions of the scaling model faithfully represent the decision-making process of legislators” (p. 143). While the nod toward the social experience of legislators is welcome, the implication that the statistical assumptions of the NOMINATE algorithm could, even theoretically, “faithfully represent the decision-making process of legislators” is, as shown by their own analysis, untenable.

54. The rules and precedents of the House of Representatives, for example, operate as a kind of algorithm, in that they simultaneously (a) produce the units of analysis (i.e., votes) and (b) process those units into an outcome (i.e., an institutional decision). The first step translates individual behavior into a statistic, and the second translates the statistic back into behavior (e.g., shared recognition of the decision). However, these steps are tautological in the sense they are the product of formal definitions (as opposed to hypothetical statements about empirical reality).

55. For a sampling of the precedents that have strictly defined the practice of voting, see Hinds, Asher C., Hinds’ Precedents of the House of Representatives (Washington, DC: Government Printing Office, 1907), volume V, sections 5970, 5981, 6046, 6047, 6048Google Scholar.

56. It is only the behavior (i.e., the vocal announcement of a vote or the electronic recording of a pushed button) that is unambiguous, not the relation between the casting of a vote and the intention of the voter. Whether or not the member makes a mistake in translating his or her preference into these ritual behaviors or whether or not the member actually understands the decision upon which he or she is voting is not empirically evidenced. When votes are properly cast within this ritually defined setting, the institution simply does not care (in the sense of recognizing this behavior for the purposes of making a collective decision) whether members are accurately translating their personal preferences into the formal votes that they cast. In other words, whether or not members actually know what they are doing when they cast a vote is irrelevant to institutional recognition as long as the formal proprieties are observed.

57. This is perhaps one of the reasons that the statistical artifacts produced by the NOMINATE algorithm are much more frequently used as a descriptive adjunct to interpretive narratives, as opposed to employment as independent or dependent variables. The “look at this” invocation in a descriptive mode evades the impossibility of explaining what it is we are looking at which would inevitably arise if the artifacts were to enter into a hypothetical statement.

58. This is the problem that, in various ways, Bateman and Lapinski, Lee, and Caughey and Schickler have attempted to address. While their critiques of the problem are extraordinarily acute and perceptive, the solutions they recommend only recreate, in each case, the problem in another (albeit more sophisticated) form.

59. The NOMINATE algorithm has sparked a vigorous and often fruitful debate over the general question of how we might interpret congressional behavior over the course of American history. For this and other things, we should heartily thank Keith Poole and Howard Rosenthal. However, the algorithm itself has now become a bit like Homo erectus in the evolution of congressional research. Despite the creative efforts of many supremely talented analysts to devise a viable mutation so that the lineage might continue, it has basically reached a dead end and is now biding time as we wait for the next rung on the evolutionary ladder to appear. From that perspective, the algorithm has become as much an impediment to historical research as it was once a useful tool.

60. Bateman and Lapinski offer similar reasons why APD scholars have generally resisted integration into the NOMINATE community (pp. 149–156).

61. See, for example, Orren, Karen and Skowronek, Stephen, Searching for American Political Development (New York: Cambridge University Press, 2004)Google Scholar.

62. Use of the NOMINATE algorithm, for example, is almost universally acknowledged as the preeminent statistical tool in the analysis of legislative behavior. As Bateman and Lapinski note, NOMINATE has “fundamentally transformed the analysis of congressional politics” (p. 147). Caughey and Schickler similarly remark that “no data source has had a greater impact on the study of legislative politics—both historically and in the contemporary period—than the NOMINATE project” (p. 128). Everson et al. are even more effusive in their praise: “anyone interested in understanding current affairs should care about NOMINATE” (p. 98).

63. Kuhn, Thomas, The Structure of Scientific Revolutions (Chicago: University of Chicago Press, 1962)Google Scholar.