Scholarship is a risky activity, in which there is always the possibility of failure. Many scholars fail honorably, and sometimes tragically, if they have devoted their career to pursuing an hypothesis that was finally disproved. The topic of this article is dishonorable failures. In other words, the thrust of the argument will be overwhelmingly negative.^{Footnote 1}

I shall argue that there are many schools of thought in the humanities and the social sciences that are obscurantist, by which I mean that one can say ahead of time that pursuits within these paradigms are unlikely to yield anything of value. If something of value is produced, as may happen, it does so in spite of, not because of, the adoption of the paradigm. Moreover, I believe that with honorable exceptions these approaches have acquired an almost dominant or hegemonic status. As I cannot document that claim, it will have to stand as an unsubstantiated assertion. The article as a whole, in fact, may seem long on assertions and short on arguments. In a brief discussion that covers a vast ground, this defect is inevitable. With the exception of my objections to data analysis, for which I rely on second-hand authorities, I am nevertheless confident that I could back up my statements with detailed and specific criticism.

There is a less polite word for obscurantism: bullshit. Within Anglo-American philosophy there is in fact a minor sub-discipline that one might call bullshittology.^{Footnote 2} A weakness of this literature is that it mostly does not engage directly with specific writers or schools or provide any causal mechanisms to explain the emergence and persistence of a large body of scholarly nonsense. In my opinion, bullshittology ought to be located within cognitive psychology and the sociology of science, not within philosophy. For instance, psychologists have shown that the human mind constantly and excessively engages in pattern-seeking, seeing connections and relations where none exist. As I argue below, social scientists are prone to the same tendency. Also, it is not surprising that some social scientists are subject to the social mechanisms of pluralistic ignorance, groupthink and herding that other social scientists have unraveled.

Unlike these philosophers, therefore, I am not going to try to nail down the essence of obscurantism, if there is such as thing. Instead I shall try to document and illustrate my claim that in the humanities and social sciences, obscurantism is far from a marginal or minor problem. Many critics have already made this claim with respect to soft obscurantism, a term I shall explain shortly. Until recently, critics of hard obscurantism – essentially mathematical modeling and data analysis – have been less numerous and vocal. The current economic crisis has, however, brought a heightened awareness of this phenomenon.

Soft obscurantists tend to criticize hard obscurantists, and vice versa. When I criticize both sides, each of them may easily assume that I belong to the other side. Alternatively, each side may take me for an ally in their fight against the other side. Neither situation is particularly pleasant. Fortunately, I am not alone in this enterprise. A number of public-spirited scholars have taken time off from their own work to criticize and denounce obscurantism in detail, sentence by sentence or equation by equation.^{Footnote 3}

Abstractly speaking, obscurantism may cause either waste or harm. The waste mainly involves scholars and students who could have devoted their life to work that would have been more useful to society as well as more fulfilling to the individuals themselves. The harm can occur when obscurantist theories are used as premises for action. Hence we get four categories:

I cannot illustrate or discuss all of these, so I have to limit myself to some examples.

I begin in the bottom right-hand corner with some interpretations of Jane Austen's Mansfield Park. Consider first Edward Said's comments on her description of Sir Thomas Bertram's reaction to the chaos and frivolity he finds upon coming back from the West Indies:

True, there is nothing in the work to contradict this interpretation. Nor is there anything that supports it. It is absurd to argue, as Said does, that a reading that is not explicitly contradicted by the text can ipso facto be imputed to the author as a “clear” expression of her intentions.

A more egregious example can be taken from two claims that Fanny Price, the heroine of the novel, is essentially a prostitute. In Jenny Davidson's interpretation of Mansfield Park, she argues that Fanny Price has “much in common” with Lucy Steele from Sense and Sensibility (Reference DavidsonDavidson 2004: 155, 164). Although some readers have found Fanny Price insufferably priggish and obsequious, she would appear on a naive reading (the one I would defend) to have nothing in common with the selfish and scheming Lucy Steele. Davidson, however, views Fanny's modesty as instrumental (150), and claims that her “ability to conceal her thoughts turns out to be a highly effective stratagem” (161) in the conquest of Edward Bertram. These claims fail two tests of intentionality. First, there is no evidence in the novel for imputing scheming intentions to Fanny Price. Although her modesty is in fact rewarded, that consequence of her behavior cannot explain it.^{Footnote 4} Second, there is no evidence for imputing to Jane Austen an intention to make readers view Fanny Price as similar to Lucy Steele. Davidson cites the facts that “Fanny” evokes Cleland's novel Fanny Hill (a “woman of pleasure”) and that “Price” has obvious monetary connotations (163–4). Although the text may cause these associations to be produced in some modern readers, Davidson offers no evidence that Austen intended her readers to associate Fanny Price with the heroine of a pornographic novel. Her reading amounts in fact to a functional explanation of the text (see below).

An even more extravagant effort along the same lines is offered by Jillian Heydt-Stevenson. Her argument amounts to a claim of an almost systematic attempt by Austen to deceive naive readers of Mansfield Park into believing in the innocence of Fanny Price, while allowing more perspicacious readers to understand that she is “little more [sic] than a fetishistic commodity, essentially [sic] bought and sold by members of her family, encouraged to prostitute herself for rank and wealth” (Reference Heydt-StevensonHeydt-Stevenson 2000: 328). Like Davidson she ignores the inconvenient fact of Henry Crawford's rejected offer, and like her she cites the use of “Fanny” and “Price” as evidence for the sex-for-money interpretation of Fanny Price. This procedure is little better than numerology or astrology.

From these functionalist readings, I now turn to an instance of structuralist literary criticism in a famous article by Roman Jakobson and Claude Lévi-Strauss. Although Lévi-Strauss's early work on kinship systems was not obscurantist, his later work on analysis of myth seems to me impenetrable and lacking in intersubjective validity. The same comment applies, I believe, to the analysis of Baudelaire's poem “Les chats” from which I shall quote a brief excerpt:

The whole article is an orgy in arbitrary binary oppositions whose significance is never stated. Here, I only want to direct the reader's attention to the last sentence, where the authors claim that the unvoiced fricative, after its appearance in the sixth line, is thereafter carefully avoided. How do they know that the absence is deliberate? What is the significance of the absence? What about all the other absences that one could list? What is the significance of the repetition of the fricative? Once again, numerology seems to be a close cousin.

The heyday of structuralism is over, and the theory seems to have left few traces. Functionalism, however, is alive and well, as I discovered upon my recent return to France. More specifically, the ideas of Louis Althusser, Michel Foucault and Pierre Bourdieu retain an immense and immensely pernicious influence on French scholarship in the social sciences and in philosophy. According to these writers, social practices of the most varied kind can be explained by their tendency to maintain the hegemony of dominant groups. If some practices appear to have a different effect, that fact is itself explicable by the “need of the system” to maintain a facade of impartiality. As in the second passage this functionalist mind-set is readily identified by the pervasive use of verbs (or verbal nouns) without subjects. The following excerpt from Foucault's enormously influential Discipline and Punish is a perfect illustration of this practice:

As I have discussed this passage in previous work (Reference ElsterElster 1983: 104–5) and as its vacuousness is, I hope, evident, I shall not examine it any further. I cite it only because I am confident, on the basis of serving on French dissertation committees and of intellectual encounters of other kinds, that the general attitude it expresses is still very pervasive. Currently, the main term of art is that of a “dispositif” – any kind of institution or behavioral pattern that can be claimed to have the effect or function, directly or indirectly, in the short run or in the long run, of perpetuating the rule of the elites. We can learn from Foucault's own words how he erected this apparatus into some kind of supraindividual agent:

Among the other entries in the bottom right-hand cell of the table, let me briefly mention psychoanalysis. In France and Argentina and to a lesser extent in other countries, this approach to the mind is still taken seriously. As a theoretical enterprise, it lacks any scientific foundation. With some important exceptions, the study of psychoanalysis is a waste of time. This is particularly true of the more extreme representatives, such as Jacques Lacan and Melanie Klein. Since I cannot in this short paper offer evidence, let me just cite from a recent editorial in Nature (2009):

The two phrases I have italicized correspond to my charges of obscurantism and waste. Although psychoanalysis as a form of therapy may have fallen out of fashion in many countries, many scholars still rely on its theoretical framework.

I shall not dwell on multiculturalism and post-modernism as sources of waste, but refer readers to Barry, Sokal and Bricmont. The flaws of Marxism as a theoretical enterprise were exposed so thoroughly by the “non-bullshit Marxist” group that non-bullshit Marxism was revealed to be an empty set (a rare case of intellectual autophagy). In my own view, the only contribution of Marx left standing is his normative conception of the good life as one of active and joint self-realization. Marxist theory has nothing to contribute to empirical social science (which is not to decry the work of individual Marxist scholars). In spite of its intellectual bankruptcy, Marxism continues to be taught in universities all over the world, in the characteristically obscurantist style of shared verbal reflexes masquerading as arguments.

From waste, let me turn to the harm that can be caused by soft obscurantism. In the 1960s and beyond, Bruno Bettelheim persuaded many that the cause of autism lay in uncaring parents, especially in the “refrigerator mother”. The view, which had no basis in evidence and is today known to be false, caused needless feelings of guilt in parents of autistic children (Reference Severson, Aune, Jodlowski and OsteenSeverson, Aune & Jodlowski 2007). Similarly,

In a Norwegian case in which a father was accused of sexual abuse of his daughter, on the basis of her statements, an expert psychologist testified that the sharp fence posts in the child's drawing of a house surrounded by a fence very likely had a sexual significance (Aftenposten, Oslo, October 9 1999). She affirmed, moreover, that the number of posts in the fence very probably indicated the number of occasions on which the child had been abused. The child's father spent two weeks in jail, in a security cell, was barely acquitted of incest, but his life was ruined. Later, the child confessed that it was all an invention.

It goes without saying that in one sense, Marxism has caused untold harm. Whether Marxism as a theory, rather than a practice or a rationalization for actions undertaken on other grounds, has done actual harm is not obvious. The Stalinist idea of “objective complicity,” while consistent with some functionalist aspects of Marxism, probably reflects a very general tendency of the human mind. The more specific idea of “crisis maximization” or la politique du pire – things have to get worse (be made worse) before they can get better – does perhaps have a more direct Marxist ancestry. Here again, however, there are many non-Marxist examples, such as the behavior of Louis XVI in 1790–91. On a more general level, however, Marx's successors took over his intellectual hubris and notably his belief in an historical progression with a desirable, inevitable and foreseeable outcome. The most serious charge that can be laid against Marx is that his contempt for what he called “bourgeois rights” probably had a direct influence on the massive human rights violations that his successors carried out. I refer readers to Reference KolakowskiLeszek Kolakowski (1978) for unsurpassed analyses of these attitudes and their disastrous consequences.

I now move on to defend what may be more controversial and surprising claims about hard obscurantism.

There have been many attempts to explain human behavior using quantitative models. The most prominent ones are perhaps rational-choice theory (including game theory), agent-based modeling, and network theory. What little I know about the last two tends to make me skeptical, but I cannot make any strong claims about them. A long familiarity with rational-choice models has, however, fostered a skepticism that is, I hope, better grounded.

Before I state the grounds for my skepticism of the theory, let me emphasize that for some purposes it is highly valuable. In my opinion, the development of rational choice theory, as a conceptual tool for understanding human action and interaction, has been the greatest achievement of the social sciences to date. It is, moreover, an indispensable policy tool for changing behavior, for the simple reason that people usually respond to incentives in predictable ways. What I deny is that it has a privileged status as an explanatory tool.

To achieve explanatory success, a theory should, minimally, satisfy two criteria: it should have determinate implications for behavior, and the implied behavior should be what we actually observe. (These are necessary conditions, not sufficient ones.) Rational-choice theory often fails on both counts. The theory may be indeterminate, and people may be irrational. In what was perhaps the first sustained criticism of the theory, Keynes emphasized indeterminacy, notably because of the pervasive presence of uncertainty. His criticism applied especially (but not only) to cases where agents have to form expectations about the behavior of other agents or about the development of the economy in the long run. In the wake of the current economic crisis, this objection has returned to the forefront. Before the crisis, going back to the 1970s, the main objections to the theory were based on pervasive irrational behavior. Experimental psychology and behavioral economics have uncovered many mechanisms that cause people to deviate from the behavior rational-choice theory prescribes.

Disregarding some more technical (and important) sources of indeterminacy, the most basic one is embarrassingly simple: how can one impute to the social agents the capacity to make the calculations that occupy many pages of mathematical appendices in the leading journals of economics and political science and that can be acquired only through years of professional training? Here are some frequently made answers to this question, with my rejoinders:

1. 1. We should accept a theory if it produces correct predictions even if we do not understand how it does so (quantum mechanics). Rejoinder: the social sciences produce very few correct predictions, except for the short-term impact of small changes in economic variables.

2. 2. Even though agents are incapable of intentional maximizing in complex situations, natural or social selection will eliminate non-maximizers. Rejoinder: this is mere hand-waving. No explicit model exists.

3. 3. Individual errors will cancel each other in the aggregate. Rejoinder: there is no reason to think they are symmetrically distributed around the correct answer.

4. 4. An expert billiard player whose experience helps him figure out the angles would be utterly incapable of solving the relevant equations; yet he acts as if he could. Rejoinder: nobody can be an expert in all the situations where economists apply rational-choice theory. Development of expertise in any field is said to require 10,000 hours of practice.

With regard to irrationality, I shall only produce a (short!) list of robust mechanisms:^{Footnote 5}

• • hyperbolic time discounting

• • loss aversion

• • the sunk-cost fallacy and the planning fallacy (especially deadly in conjunction)

• • the tendency of unusual events to trigger stronger emotional reactions (an implication of “norm theory”)

• • the cold-hot and hot-cold empathy gaps

• • trade-off aversion and ambiguity aversion

• • anchoring in the elicitation of beliefs and preferences

• • the representativeness and availability heuristics

• • the conjunction and disjunction fallacies

• • the certainty effect and the pseudo-certainty effect

• • choice bracketing, framing, and mental accounting

• • cases when “less is more” and “more is less”

• • sensitiveness to changes from a reference point rather than to absolute levels

• • status quo bias and the importance of default options

• • meliorizing rather than maximizing

• • motivated reasoning and self-serving biases in judgment

• • flaws of expert judgments and of expert predictions

• • self-signaling and magical thinking

• • non-consequentialist and reason-based choice

• • overconfidence and the illusion of control

• • spurious pattern-finding.

Many scholars – not just rational-choice theorists – find the rapidly expanding array of mechanisms uncomfortably large. Some are also worried by the fact that unlike rational-choice economics, behavioral economics does not rest on a unified theory. Rather, it consists of a bunch of theories or mechanisms that are not deductively linked among themselves. Human behavior seems to be guided by a number of unrelated quirks rather than by consistent maximization of utility. In fact, there are so many quirks that one suspects that for any observed behavior, there would be a quirk that fits it. While these qualms are understandable, they do not amount to reasoned criticism. Behavioral economics is here to stay.

With regard to data analysis, I cannot make claims to first-hand competence. I rely mainly on the work of David Freedman, an acknowledged master of pure and applied statistics. He carried out detailed analyses of six articles published in leading academic journals: four from American Political Science Review, one from Quarterly Journal of Economics and one from American Sociological Review. One of them was named “best article of the year” by the journal where it appeared. The number of mistakes and confusions that he finds – some of them so elementary that even I could understand them – is staggering.

Friedman demonstrated the many pitfalls, fallacies and temptations that arise in regression analysis as it is routinely carried out.^{Footnote 6} Let me cite two of my favorite among his statements. The first is taken from a famous article on “Statistical models and shoe leather”:

When I have presented my objections to data analysis to various audiences, my critics have usually located themselves at point (1) of this scale.

The second statement I shall cite offers a caricature – which like any good caricature reveals important features of its object – of the responses that modelers might make to his criticism:

I shall return to the question of harm. First, however, there is the issue of waste. I believe that much work in economics and political science that is inspired by rational-choice theory is devoid of any empirical, aesthetic or mathematical interest, which means that it has no value at all. I cannot make a quantitative assessment of the proportion of work in leading journals that fall in this category, but I am confident that it represents waste on a staggering scale. I believe, but less confidently, that the same is true is true of many instances of data analysis.

The amount of harm is more difficult to assess. I do not know of any detailed study discussing the importance of modeler hubris in the 1998 collapse of Long Term Capital Management, costing investors 4.5 billion dollars, or in the current financial crisis. Greed, short-termism and deregulation may have been more important than unwarranted confidence in the Nobel-prize-winning models. Although one can cite many examples of fund managers telling their clients that according to their models a crisis of the magnitude of what has happened since 2007 would only occur once in n years, n being some very large number, it remains to be shown that these managers actually believed in the models and used them as decision premises. After all, as we have seen, they had very little to lose if the models got it wrong.

This being said, I find it hard to believe that excessive belief in the efficiency of markets and the rationality of market participants did not play some role in generating the crisis. Since the information that is reflected in prices is a public good, nobody has an incentive to produce it (Reference Grossman and StiglitzGrossman & Stiglitz 1980). This free-rider problem generates mechanical diversification of assets as a substitute for due diligence (Reference BhidéBhidé 2009). The “Warren Buffett remedy,” similar to the “shoe-leather sociology” that David Freedman proposed to keep data analysis honest, is shoe leather investment.

Turning now to the possible harm caused by data analysis, we may consider the impact of Chicago-style economics on legislation concerning the death penalty and gun control. In one summary,

Although Ehrlich's work was discredited, a new wave of studies claimed to have found similar effects. Commenting on an analysis that challenged this conclusion, Ehrlich said that “[i]f variations like unemployment, income inequality, likelihood of apprehension and willingness to use the death penalty are accounted for, the death penalty shows a significant deterring effect” (quoted in Reference FessendenFessenden 2000). In sobering contrast, Reference AchenChristopher Achen (2002: 446) warns that “[a] statistical specification with more than three explanatory variables is meaningless.” There are just too many ways to fiddle with the numbers.

Similar remarks apply to the claim by Reference LottJohn Lott (2000) – cited by John Ashcroft as Attorney General in the Bush administration – that the right to carry concealed handguns saves lives. Commenting on Lott's work, Hashem Dezhbakhs writes that “[t]he academic survival of a flawed study may not be of much consequence. But, unfortunately, the ill-effects of a bad policy, influenced by flawed research, may hurt generations” (cited after Reference Donohue and WolfersDonohue & Wolfers 2005: 745). In other words, we can tolerate waste, but we should not accept harm.

The advocates of the death penalty and right-to-carry laws will respond that these measures tend to reduce harm. In their discussion of life-life tradeoffs, Cass Sunstein and Adrian Vermeule defend the death penalty on these grounds. To be sure, they qualify their argument as a conditional one, valid only if the empirical claims stand up. Yet they display considerable hubris when, as legal scholars with no econometric training, they assert that “[o]ne leading study finds that as a national average, each execution deters some eighteen murders” and proceed to discuss the implications (Reference Sunstein and VermeuleSunstein & Vermeule 2005: 708). If they had read David Freedman, they would have understood that the idea of executing people on the basis of any kind of statistical analysis is morally unacceptable. There is just too much uncertainty about the facts to warrant this irreversible action. (I am not implying that this is the only reason for opposing the death penalty.) I also believe that the “leading study” is wrong, but that is a second-hand opinion that has no particular weight.

Readers who are willing to grant me the claim that soft and hard obscurantism is sufficiently widespread to justify worry may have asked themselves about its causes and possible remedies. To the speculations I have offered elsewhere about the psychological and sociological mechanisms that may sustain the production of nonsense on a large scale (Reference ElsterElster 2009a; see also Reference Gambetta and OriggiGambetta & Origgi 2009), I shall add the following.

When we believe we have found the cause of a phenomenon, there occurs a mental click of satisfaction. Other mental operations can also trigger clicks. There is the click of observing a similarity between the phenomenon and another one, the click of observing that the phenomenon has consequences that benefit someone or something, and the click of observing that some other phenomenon could have been the cause of the phenomenon we are studying. Because all these observations satisfy our deep-seated need to find order, patterns and meanings in the universe, they trigger a click that is easily mistaken for the real thing, the click of explanation. I offer this not as an established truth, but as a hypothesis that might explain the grip that analogies, functional explanations and just-so stories have on the mind. These attempted shortcuts to knowledge also owe a great deal of their attraction to our unwillingness to admit ignorance. As Michel de Montaigne said (Reference de Montaigne1991: 1165): “Many of this world's abuses are engendered – or to put it more rashly, all of this world's abuses are engendered – by our being schooled to be afraid to admit our ignorance and because we are required to accept anything which we cannot refute.”

As for remedies, I am mostly pessimistic. Both soft and hard obscurantists are so numerous that they will always be able to muster the large number of citations that is threatening to become a necessary and sufficient condition for funding and survival. One lever for change could be careful and detailed critical reading – shoe leather criticism – of the kind exemplified by the public-spirited scholars that I cited at the outset. Let us hope there are enough scholars who have both the competence and the public-spiritedness to follow their example. At the same time, of course, we can hope that obscurantism will recede or at least be kept at bay as good non-obscurantist scholarship flourishes.