Hostname: page-component-78c5997874-4rdpn Total loading time: 0 Render date: 2024-11-17T17:53:59.478Z Has data issue: false hasContentIssue false
  • English
  • Français

Assessing Threats to Inference with Simultaneous Sensitivity Analysis: The Case of US Supreme Court Oral Arguments*

Published online by Cambridge University Press:  10 December 2015

Jeffrey Budziak  and
Daniel Lempert
Get access Rights & Permissions [Opens in a new window]

Abstract

Political scientists relying on observational data face substantial challenges in drawing causal inferences. A particularly problematic threat to inference is the unobserved confounder. As a means to assess this threat, we introduce simultaneous sensitivity analysis to the political science literature. As an application, we consider the potentially confounded relationship between Supreme Court justice voting and oral argument quality. We demonstrate that this relationship is sensitive to the presence of a confounder, to a degree that threatens inference, and explore the confounder both theoretically and empirically. More generally, we show how sensitivity analysis can guide inquiry related to a covariate that cannot be directly measured.

Type
Original Articles
Information
Political Science Research and Methods , Volume 6 , Supplement 1 , January 2018 , pp. 33 - 56
Copyright
© The European Political Science Association 2015 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

*

Jeffrey Budziak is an Assistant Professor in the Department of Political Science, Western Kentucky University, 1906 College Heights Blvd., 300 Grise Hall, Bowling Green, KY 42101 ([email protected]). Daniel Lempert is an Assistant Professor in the Department of Politics, SUNY Potsdam, 44 Pierrepont Avenue, Potsdam, NY 13676 ([email protected]). A version of this paper was presented at the 2010 Conference on Empirical Legal Studies at Yale University. For helpful comments, discussion, and suggestions, the authors thank Larry Baum, Greg Caldeira, William Minozzi, and Dylan Small. The authors thank Timothy Johnson, James Spriggs, and Paul Wahlbeck for making their data available. To view supplementary material for this article, please visit http://dx.doi.org/10.1017/psrm.2015.74

References

Bailey, Michael A., and Maltzman, Forrest. 2008. ‘Does Legal Doctrine Matter? Unpacking Law and Policy Preferences on the U.S. Supreme Court’. American Political Science Review 102(3):369–384.Google Scholar
Bailey, Michael A., and Maltzman, Forrest. 2011. The Constrained Court: Law, Politics and the Decisions Justices Make. Princeton, NJ: Princeton University Press.Google Scholar
Bartels, Brandon L. 2009. ‘The Constraining Capacity of Legal Doctrine on the U.S. Supreme Court’. American Political Science Review 103(3):474–495.Google Scholar
Baum, Lawrence. 1997. The Puzzle of Judicial Behavior. Ann Arbor, MI: University of Michigan Press.Google Scholar
Berry, William D., DeMeritt, Jacquline H. R., and Esarey, Justin. 2010. ‘Testing for Interactions in Binary Logit and Probit Models: Is a Product Term Essential?’. American Journal of Political Science 54(1):248–266.Google Scholar
Black, Ryan C., Sorenson, Marion W., and Johnson, Timothy R.. 2013. ‘Toward an Actor-Based Measure of Supreme Court Case Salience: Information-Seeking and Engagement During Oral Argument’. Political Research Quarterly 66(4):804–818.Google Scholar
Black, Ryan C., Truel, Sarah A., Johnson, Timothy R., and Goldman, Jerry. 2011. ‘Emotions, Oral Arguments, and Supreme Court Decision Making’. Journal of Politics 73(2):572–581.Google Scholar
Black, Ryan C., Johnson, Timothy R., and Wedeking, Justin. 2012. Oral Arguments and Coalition Formation on the U.S. Supreme Court: A Deliberate Dialogue. Ann Arbor, MI: University of Michigan Press.Google Scholar
Bowers, Jake, Fredrickson, Mark, and Hansen, Ben B.. 2010. ‘RItools: Randomization Inference Tools’. R Package Version 0.1-11.Google Scholar
Caldeira, Gregory, and Wright, John. 1988. ‘Organized Interests and Agenda Setting at the U.S. Supreme Court’. American Political Science Review 82(4):1109–1127.Google Scholar
Caldeira, Gregory A., Wright, John R., and Zorn, Christopher J.W.. 1999. ‘Sophisticated Voting and Gate-Keeping in the Supreme Court’. Journal of Law, Economics, and Organization 15(3):549–572.CrossRefGoogle Scholar
Collins, Paul M. Jr. 2004. ‘Friends of the Court: Examining the Influence of Amicus Curiae Participation in U.S. Supreme Court Litigation’. Law and Society Review 38(4):807–832.Google Scholar
Cook, Thomas D., Shadish, William R., and Wong, Vivian C.. 2008. ‘Three Conditions Under Which Experiments and Observational Studies Produce Comparable Causal Estimates: New Findings from Within-Study Comparisons’. Journal of Policy Analysis and Management 27(4):724–750.Google Scholar
Corley, Pamela. 2008. ‘The Supreme Court and Opinion Content: The Influence of Parties’ Briefs’. Political Research Quarterly 61(3):468–478.Google Scholar
Ebbinghaus, Hermann. 1913. Memory. A Contribution to Experimental Psychology. New York, NY: Teachers College Columbia University.Google Scholar
Epstein, Lee, and Knight, Jack. 1998. The Choices Justices Make. Oxford: Oxford University Press.Google Scholar
Fisher, Ronald A. 1935. The Design of Experiments. Edinburgh: Oliver & Boyd.Google Scholar
Friedman, Barry. 2005. ‘The Politics of Judicial Review’. Texas Law Review 84(2):257–337.Google Scholar
Galanter, Marc. 1974. ‘Why the “Haves” Come Out Ahead: Speculations on the Limits of Legal Change’. Law and Society Review 9(1):95–160.Google Scholar
Gastwirth, Joseph L., Krieger, Abba M., and Rosenbaum, Paul R.. 1998. ‘Dual and Simultaneous Sensitivity Analysis for Matched Pairs’. Biometrika 85(4):907–920.Google Scholar
Gastwirth, Joseph L., Krieger, Abba M., and Rosenbaum, Paul R.. 2000. ‘Asymptotic Separability in Sensitivity Analysis’. Journal of the Royal Statistical Society, Series B 62(3):545–555.Google Scholar
George, Tracey E., and Epstein, Lee. 1992. ‘On the Nature of Supreme Court Decision Making’. American Political Science Review 86(2):323–337.Google Scholar
Gerber, Alan S., and Green, Donald P.. 2012. Field Experiments: Design, Analysis, and Interpretation. New York, NY: W.W. Norton.Google Scholar
Gibson, James L. 1983. ‘From Simplicity to Complexity: The Development of Theory in the Study of Judicial Behavior’. Political Behavior 5(1):7–49.Google Scholar
Gillman, Howard. 2001. ‘What’s Law Got to Do With It? Judicial Behavorialists Test the “Legal Model” of Judicial Decision Making’. Law and Social Inquiry 26(2):465–504.Google Scholar
Green, Donald P., and Gerber, Alan S.. 2002. ‘Reclaiming the Experimental Tradition in Political Science’. In Ira Katznelson and Helen V. Milner (eds), Political Science: The State of the Discipline. 805–832. New York, NY: W.W. Norton.Google Scholar
Guan, Weihua, Liang, Liming, Boehnke, Michael, and Abecasis, Gonalo R.. 2009. ‘Genotype-Based Matching to Correct for Population Stratification in Large-Scale Case-Control Genetic Association Studies’. Genetic Epidemiology 33(6):508–517.Google Scholar
Hainmueller, Jens, and Hangartner, Dominik. 2013. ‘Who Gets a Swiss Passport? A Natural Experiment in Immigrant Discrimination’. American Political Science Review 107(1):159–187.Google Scholar
Hansen, Ben B. 2004. ‘Full Matching in an Observational Study of Coaching for the SAT’. Journal of the American Statistical Association 99(467):609–618.Google Scholar
Hansen, Ben B. 2007. ‘Optmatch: Flexible, Optimal Matching for Observational Studies’. R News 7:18–24.Google Scholar
Hansen, Ben B., and Bowers, Jake. 2008. ‘Covariate Balance in Simple, Stratified and Clustered Comparative Studies’. Statistical Sciences 23(2):219–236.Google Scholar
Hansen, Ben B., and Klopfer, Stephanie Olsen. 2006. ‘Optimal Full Matching and Related Designs Via Network Flows’. Journal of Computational and Graphical Statistics 15(3):609–627.Google Scholar
Hansford, Thomas G., and Spriggs, James F. II. 2008. The Politics of Precedent on the U.S. Supreme Court. Princeton, NJ: Princeton University Press.Google Scholar
Haviland, Amelia, Nagin, Daniel S., Rosenbaum, Paul R., and Tremblay, Richard E.. 2008. ‘Combining Group-Based Trajectory Modeling and Propensity Score Matching for Causal Inferences in Nonexperimental Longitudinal Data’. Developmental Psychology 44(2):422–435.Google Scholar
Hitchcock, Christopher. 2007. ‘Prevention, Preemption, and the Principle of Sufficient Reason’. Philosophical Review 116(4):494–532.Google Scholar
Hosman, Carrie A., Hansen, Ben B., and Holland, Paul W.. 2010. ‘The Sensitivity of Linear Regression Coefficients’ Confidence Limits to the Omission of a Confounder’. The Annals of Applied Statistics 4(2):849–870.Google Scholar
Howard, Robert M., and Segal, Jeffrey A.. 2002. ‘An Original Look at Originalism’. Law and Society Review 36(1):113–138.Google Scholar
Ignagni, Joseph A. 1994. ‘Explaining and Predict Supreme Court Decision Making: The Burger Court’s Establishment Clause Decisions’. Journal of Church and State 36(2):301–327.Google Scholar
Imai, Kouske, Keele, Luke, Tingley, Dustin, and Yamamoto, Teppei. 2011. ‘Unpacking the Black Box of Causality: Learning About Causal Mechanisms from Experimental and Observational Studies’. American Political Science Review 105(4):766–788.Google Scholar
Imbens, Guido W. 2003. ‘Sensitivity to Exogeneity Assumptions in Program Evaluation’. American Economic Review 2(93):126–132.Google Scholar
Johnson, Timothy R. 2001. ‘Information, Oral Arguments and Supreme Court Decision Making’. American Politics Research 29(4):331–351.Google Scholar
Johnson, Timothy R. 2004. Oral Arguments and Decision Making on the U.S. Supreme Court. Albany, NY: State University of New York Press.Google Scholar
Johnson, Timothy R., Wahlbeck, Paul J., and Spriggs, James F.. 2006. ‘The Influence of Oral Arguments on the U.S. Supreme Court’. American Political Science Review 100(1):99–113.Google Scholar
Kantor, David. n.d. ‘Mahascores’. Accessed 23 January 2013.Google Scholar
Keele, Luke, and Minozzi, William. 2013. ‘How Much is Minnesota Like Wisconsin? Assumptions and Counterfactuals in Causal Inference with Observational Data’. Political Analysis 21(2):193–216.Google Scholar
Koenker, Roger, and Ng, Pin. 2012. ‘SparseM: Sparse Linear Algebra’. R Package Version 0.96.Google Scholar
Lax, Jeffrey R., and Rader, Kelly T.. 2010. ‘Legal Constraints on Supreme Court Decision Making: Do Jurisprudential Regimes Exist?’. Journal of Politics 72(1):273–284.Google Scholar
Lehmann, Erich Leo. 1975. Nonparametrics: Statistical Methods Based on Ranks. Oakland, CA: Holden Day.Google Scholar
Lempert, Daniel. 2015. ‘Simultaneous Sensitivity Analysis in Stata: Arsimsens and Pairsimsens’. Observational Studies 1(1):74–90.Google Scholar
Liu, Weiwei, Kuramoto, Janet, and Stuart, Elizabeth A.. 2013. ‘An Introduction to Sensitivity Analysis for Unobserved Confounding in Nonexperimental Prevention Research’. Prevention Science 14(6):570–580.Google Scholar
Long, J. Scott, and Freese, Jeremy. 2014. Regression Models for Categorical Dependent Variables Using Stata, 3rd ed., College Station, TX: Stata Press.Google Scholar
Maltzman, Forrest, Spriggs, James F., and Wahlbeck, Paul J.. 2000. Crafting Law on the Supreme Court: The Collegial Game. Cambridge: Cambridge University Press.Google Scholar
Martin, Andrew D., and Quinn, Kevin M.. 2002. ‘Dynamic Ideal Point Estimation Via Markov Chain Monte Carlo for the US Supreme Court, 1953–1999’. Political Analysis 10(2):134–153.Google Scholar
McAtee, Andrea, and McGuire, Kevin T.. 2007. ‘Lawyers, Justices, and Issue Salience: When and How do Legal Arguments Affect the U.S. Supreme Court’. Law and Society Review 41(2):259–278.Google Scholar
McGuire, Kevin T. 1990. ‘Obscenity, Libertarian Values, and Decision Making in the Supreme Court’. American Politics Research 18(1):47–67.Google Scholar
McGuire, Kevin T. 1995. ‘Repeat Players in the Supreme Court: The Role of Experienced Lawyers in Litigation Success’. The Journal of Politics 57(1):187–196.Google Scholar
Morgan, Stephen L., and Winship, Christopher. 2007. Counterfactuals and Causal Inference: Methods and Principles for Social Research. Cambridge: Cambridge University Press.Google Scholar
Richards, Mark J., and Kritzer, Herbert M.. 2002. ‘Jurisprudential Regimes in Supreme Court Decision Making’. American Political Science Review 96(2):305–320.Google Scholar
Rohde, David, and Spaeth, Harold J.. 1976. Supreme Court Decision Making. New York, NY: W.H. Freeman.Google Scholar
Rosenbaum, Paul R. 1989. ‘Sensitivity Analysis for Matched Observational Studies with Many Ordered Treatments’. Scandinavian Journal of Statistics 16(3):227–236.Google Scholar
Rosenbaum, Paul R. 1991. ‘A Characterization of Optimal Designs for Observational Studies’. Journal of the Royal Statistical Society 53(3):597–610.Google Scholar
Rosenbaum, Paul R. 2002. Observational Studies, 2nd ed., New York, NY: Springer.Google Scholar
Rosenbaum, Paul R. 2010. Design of Observational Studies. New York, NY: Springer.Google Scholar
Segal, Jeffrey A. 1984. ‘Predicting Supreme Court Cases Probablistically: The Search and Seizure Cases, 1962-1981’. American Political Science Review 78(4):891–900.CrossRefGoogle Scholar
Segal, Jeffrey A., and Spaeth, Harold J.. 1993. The Supreme Court and the Attitudinal Model. New York, NY: Cambridge University Press.Google Scholar
Segal, Jeffrey A., and Spaeth, Harold J.. 2002. The Supreme Court and the Attitudinal Model Revisited. New York, NY: Cambridge University Press.Google Scholar
Sehkon, Jasjeet S. 2009. ‘Opiates for the matches: Matching Methods for Causal Inference’. Annual Review of Political Science 12:487–508.Google Scholar
Sekhon, Jasjeet, and Titiunik, Rocio. 2012. ‘When Natural Experiments are Neither Natural Nor Experiments’. American Political Science Review 106(1):35–57.Google Scholar
Sen, Maya. 2014. ‘How Judicial Qualification Ratings May Disadvantage Minority and Female Candidates’. Journal of Law and Courts 2(1):33–65.Google Scholar
Sides, John, and Lax, Jeffrey R.. 2012. ‘More on the Persuasiveness of Oral Arguments’. The Monkey Cage, March 27. Available at http://themonkeycage.org/2012/03/27/more-on-the-persuasiveness-of-oral-arguments/, accessed 1 December 2015.Google Scholar
Small, Dylan, Cheng, Jing, Halloran, M. Elizabeth, and Rosenbaum, Paul R.. 2013. ‘Case Definition and Design Sensitivity’. Journal of the American Statistical Association 108(504):1457–1468.Google Scholar
Small, Dylan, Gastwirth, Joseph L., Krieger, Abba M., and Rosenbaum, Paul R.. 2009. ‘Simultaneous Sensitivity Analysis for Observational Studies Using Full Matching or Matching with Multiple Controls’. Statistics and its Interface 2(2):203–211.Google Scholar
Spaeth, Harold J. 2007. ‘The Judicial Research Initiative—United States Supreme Court Judicial Databases’. Available at http://artsandsciences.sc.edu/poli/juri/sct.htm, accessed 1 December 2015.Google Scholar
Spaeth, Harold J., and Segal, Jeffrey A.. 1999. Majority Rule or Minority Will: Adherence to Precedent on the U.S. Supreme Court. Cambridge: Cambridge University Press.Google Scholar
Spriggs, James F. II, and Wahlbeck, Paul J.. 1997. ‘Amicus Curiae and the Role of Information at the Supreme Court’. Political Research Quarterly 50(2):365–386.Google Scholar
Stone, Harlan F. 1936. ‘The Common Law in the United States’. Harvard Law Review 50(1):4–26.Google Scholar
Stuart, Elizabeth A., and Green, Kerry M.. 2008. ‘Using Full Matching to Estimate Causal Effects in Nonexperimental Studies: Examining the Relationship Between Adolescent Marijuana Use and Adult Outcomes’. Developmental Psychology 44(2):395–406.Google Scholar
Tanenhaus, Joseph, Schick, Marvin, Muraskin, Matthew, and Rosen, Daniel. 1963. ‘The Supreme Court’s Certiorari Jurisdiction: Cue Theory’. In Glendon Schubert (ed.), Judicial Decision Making. 111–132. New York, NY: Free Press.Google Scholar
Tomz, Michael, Wittenberg, Jason, and King, Gary. 2003. ‘CLARIFY: Software for Interpreting and Presenting Statistical Results. Version 2.1’. Available at http://gking.harvard.edu/clarify, accessed 1 December 2015.Google Scholar
Wang, Liangsheng, and Krieger, Abba M.. 2006. ‘Causal Conclusions are Most Sensitive to Unobserved Binary Covariates’. Statistics in Medicine 25(13):2257–2271.Google Scholar
Wedeking, Justin. 2010. ‘Supreme Court Litigants and Strategic Framing’. American Journal of Political Science 54(3):617–631.Google Scholar
Supplementary material: PDF

Budziak and Lempert supplementary material

Budziak and Lempert supplementary material 1

Download Budziak and Lempert supplementary material(PDF)
PDF 167.9 KB
Supplementary material: File

Budziak and Lempert supplementary material

Budziak and Lempert supplementary material 2

Download Budziak and Lempert supplementary material(File)
File 37.4 KB