Hostname: page-component-78c5997874-t5tsf Total loading time: 0 Render date: 2024-11-08T20:28:06.120Z Has data issue: false hasContentIssue false
  • English
  • Français

Explaining Systematic Bias and Nontransparency in U.S. Social Security Administration Forecasts

Published online by Cambridge University Press:  04 January 2017

Konstantin Kashin ,
Gary King  and
Samir Soneji
Konstantin Kashin
Affiliation:
Institute for Quantitative Social Science, Harvard University, Cambridge, MA 02138
Gary King
Affiliation:
Institute for Quantitative Social Science, Harvard University, Cambridge, MA 02138
Samir Soneji*
Affiliation:
The Dartmouth Institute for Health Policy & Clinical Practice, Dartmouth College, Lebanon, NH 03766
*
e-mail: [email protected] (corresponding author)
Rights & Permissions [Opens in a new window]

Abstract

Core share and HTML view are not available for this content. However, as you have access to this content, a full PDF is available via the ‘Save PDF’ action button.

The accuracy of U.S. Social Security Administration (SSA) demographic and financial forecasts is crucial for the solvency of its Trust Funds, other government programs, industry decision-making, and the evidence base of many scholarly articles. Because SSA makes public insufficient replication information and uses antiquated statistical forecasting methods, no external group has ever been able to produce fully independent forecasts or evaluations of policy proposals to change the system. Yet, no systematic evaluation of SSA forecasts has ever been published by SSA or anyone else—until a companion paper to this one. We show that SSA's forecasting errors were approximately unbiased until about 2000, but then began to grow quickly, with increasingly overconfident uncertainty intervals. Moreover, the errors are largely in the same direction, making the Trust Funds look healthier than they are. We extend and then explain these findings with evidence from a large number of interviews with participants at every level of the forecasting and policy processes. We show that SSA's forecasting procedures meet all the conditions the modern social-psychology and statistical literatures demonstrate make bias likely. When those conditions mixed with potent new political forces trying to change Social Security, SSA's actuaries hunkered down, trying hard to insulate their forecasts from strong political pressures. Unfortunately, this led the actuaries into not incorporating the fact that retirees began living longer lives and drawing benefits longer than predicted. We show that fewer than 10% of their scorings of major policy proposals were statistically different from random noise as estimated from their policy forecasting error. We also show that the solution to this problem involves SSA or Congress implementing in government two of the central projects of political science over the last quarter century: (1) transparency in data and methods and (2) replacing with formal statistical models large numbers of ad hoc qualitative decisions too complex for unaided humans to make optimally.

Type
Articles
Information
Political Analysis , Volume 23 , Issue 3 , Summer 2015 , pp. 336 - 362
Creative Commons
Creative Common License - CCCreative Common License - BY
This is an Open-Access article, distributed under the terms of the Creative Commons Attribution license (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted re-use, distribution, and reproduction in any medium, provided the original work is properly cited.
Copyright
Copyright © The Author 2015. Published by Oxford University Press on behalf of the Society for Political Methodology

Footnotes

Authors' note: For helpful advice or comments, we are grateful to Bill Alpert, Jim Alt, Steve Ansolabehere, Neal Beck, Nicholas Christakis, Mo Fiorina, Dan Gilbert, Alexander Hertel-Fernandez, Martin Holmer, David Langer, and Theda Skocpol. Thanks also to the many participants in the forecasting and policy process for information and advice. Replication data are available on the Political Analysis Dataverse at http://dx.doi.org/10.7910/DVN/28323. Supplementary materials for this article are available on the Political Analysis Web site.

References

Altman, Nancy J. 2005. The battle for Social Security: From FDR's vision to Bush's gamble. Hoboken, NJ: John Wiley & Sons.Google Scholar
Autor, David H., and Duggan, Mark G. 2006. The growth in the Social Security disability rolls: A fiscal crisis unfolding. Journal of Economic Perspectives 20(3): 71–96.Google Scholar
Banaji, Mahzarin R., and Greenwald, Anthony G. 2013. Blindspot: Hidden biases of good people. New York, NY: Delacorte Press.Google Scholar
Beland, Daniel. 2005. Social Security: History and politics from the New Deal to the privatization debate. Lawrence, KS: University Press of Kansas.Google Scholar
Beland, Daniel, and Waddan, Alex. 2012. The politics of policy change: Welfare, Medicare, and Social Security reform in the United States. Washington, DC: Georgetown University Press.Google Scholar
Blahous, Charles P. III 2007. Have the Social Security Trustees been too conservative? Paper presented at the American Enterprise Institute.Google Scholar
Blahous, Charles P. III 2010. Social Security: The unfinished work. Stanford, CA: Hoover Institution Press.Google Scholar
Edwards, George C III. 2007. Governing by campaigning: The politics of the Bush presidency. New York, NY: Longman Publishing.Google Scholar
Gilbert, Daniel T. 1998. Ordinary psychology. In The handbook of social psychology, eds. Gilbert, Daniel T., Fiske, Susan T., and Lindzey, G., Vol. 2, 89–150. New York: McGraw Hill.Google Scholar
Girosi, Federico, and King, Gary 2008. Demographic forecasting. Princeton, NJ: Princeton University Press. http://gking.harvard.edu/files/smooth/(accessed April 29, 2015).Google Scholar
Human Mortality Database. 2015. University of California, Berkeley (USA), and Max Planck Institute for Demographic Research (Germany). or (data downloaded on 31 January 2015) http://www.mortality.org or http://www.humanmortality.de (accessed April 29, 2015).Google Scholar
Kahneman, Daniel. 2011. Thinking, fast and slow. New York, NY: Farrar, Straus and Giroux.Google Scholar
Kashin, Konstantin, King, Gary, and Soneji, Samir 2015a. Replication data for: Explaining systematic bias and nontransparency in U.S. Social Security Administration forecasts. Harvard Dataverse [Distributor] V1 [Version]. UNF:6:967llFHgiywsHWWp1cVg9A== http://dx.doi.org/10.7910/DVN/28323 (accessed April 29, 2015).Google Scholar
Kashin, Konstantin, King, Gary, and Soneji, Samir 2015b. Replication data for: Systematic bias and nontransparency in U.S. Social Security Administration forecasts. Harvard Dataverse [Distributor] V1 [Version]. UNF:5:1oerGFXQ0Bu9bcMFU5/t2A== http://dx.doi.org/10.7910/DVN/28122 (accessed April 29, 2015).Google Scholar
Kashin, Konstantin, King, Gary, and Soneji, Samir 2015c. Systematic bias and nontransparency in U.S. Social Security Administration forecasts. Journal of Economic Perspectives. In press.CrossRefGoogle Scholar
King, Gary. 1995. Replication, replication. PS: Political Science and Politics 28(3): 443–99. http://j.mp/jCyfF1 (accessed April 29, 2015).Google Scholar
King, Gary, and Zeng, Langche 2004. Inference in case-control studies. In Encyclopedia of biopharmaceutical statistics, ed. Chow, Shein-Chung, 2nd ed. New York: Marcel Dekker. http://gking.harvard.edu/files/abs/1s-enc-abs.shtml (accessed April 29, 2015).Google Scholar
King, Gary, and Soneji, Samir. 2011. The future of death in America. Demographic Research 25(1): 1–38. http://j.mp/iXUpBv (accessed April 29, 2015).Google Scholar
Laursen, Eric. 2012. The people's pension: The struggle to defend Social Security since Reagan. Edinburgh, Scotland: AK Press.Google Scholar
Lee, Ronald D., and Carter, Lawrence R. 1992. Modeling and forecasting U.S. mortality. Journal of the American Statistical Association 87(419): 659–75.Google Scholar
Office of the Chief Actuary. 2012. The long-range demographic assumptions for the 2012 Trustees Report. Technical report, Social Security Administration. http://j.mp/OCACT12 (accessed April 29, 2015).Google Scholar
Office of the Chief Actuary. 2013. The long-range demographic assumptions for the 2013 Trustees Report. Technical report, Social Security Administration. http://j.mp/OCACT13 (accessed April 29, 2015).Google Scholar
Office of the Chief Actuary. 2014. The long-range demographic assumptions for the 2014 Trustees Report. Technical report, Social Security Administration. http://j.mp/OCACT14 (accessed April 29, 2015).Google Scholar
Rosenblatt, Robert, and DeWitt, Larry 2005. The role of Social Security's chief actuary. Contingencies (July/August):40–5. http://j.mp/ChiefA (accessed April 29, 2015).Google Scholar
Social Security Advisory Board Technical Panel. 2003. Technical Panel on assumptions and methods. Technical report, Social Security Advisory Board. http://j.mp/SSATech03 (accessed April 29, 2015).Google Scholar
Social Security Advisory Board Technical Panel. 2007. Technical Panel on assumptions and methods. Technical report, Social Security Advisory Board. http://j.mp/SSATech07.Google Scholar
Social Security Advisory Board Technical Panel. 2011. Technical Panel on assumptions and methods. Technical report, Social Security Advisory Board http://j.mp/SSATech11.Google Scholar
Soneji, Samir, and King, Gary. 2012. Statistical security for Social Security. Demography 49(3): 1037–60. http://j.mp/Qvla7N (accessed April 29, 2015).Google Scholar
Tetlock, Philip E. 2005. Expert political judgment: How good is it? How can we know? Princeton, NJ: Princeton University Press.Google Scholar
The Board of Trustees, Federal OASDI Trust Funds. 1982–2010. Annual report of the Board of Trustees of the federal old-age and survivors insurance and federal disability insurance Trust Funds. Technical report, Social Security Advisory Board.Google Scholar
Weaver, R. Kent. 1988. Automatic government: The politics of indexation. Brookings Institution Press.Google Scholar
Wilson, Timothy D., and Brekke, Nancy. 1994. Mental contamination and mental correction: Unwanted influences on judgments and evaluations. Psychological Bulletin 116(1):117.Google Scholar