Hostname: page-component-cd9895bd7-dzt6s Total loading time: 0 Render date: 2025-01-04T10:00:20.717Z Has data issue: false hasContentIssue false
  • English
  • Français

Achieving Statistical Significance with Control Variables and Without Transparency

Published online by Cambridge University Press:  16 November 2020

Gabriel S. Lenz Open the ORCID record for Gabriel S. Lenz [Opens in a new window]  and
Alexander Sahn
Gabriel S. Lenz*
Affiliation:
Travers Department of Political Science, University of California, Berkeley, CA94720-1950, USA. Email: [email protected], [email protected]
Alexander Sahn
Affiliation:
Travers Department of Political Science, University of California, Berkeley, CA94720-1950, USA. Email: [email protected], [email protected]
*
Corresponding author Gabriel S. Lenz
Get access Rights & Permissions [Opens in a new window]

Abstract

How often do articles depend on suppression effects for their findings? How often do they disclose this fact? By suppression effects, we mean control-variable-induced increases in estimated effect sizes. Researchers generally scrutinize suppression effects as they want reassurance that authors have a strong explanation for them, especially when the statistical significance of the key finding depends on them. In a reanalysis of observational studies from a leading journal, we find that over 30% of articles depend on suppression effects for statistical significance. Although increases in key effect estimates from including control variables are of course potentially justifiable, none of the articles justify or disclose them. These findings may point to a hole in the review process: journals are accepting articles that depend on suppression effects without readers, reviewers, or editors being made aware.

Keywords

reproducibilitytransparencyopen sciencecovariatescontrolssuppression effectsregression
Type
Article
Information
Political Analysis , Volume 29 , Issue 3 , July 2021 , pp. 356 - 369
Copyright
© The Author(s) 2020. Published by Cambridge University Press on behalf of the Society for Political Methodology

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

Edited by Jeff Gill

References

Achen, C. H. 2002. “Toward a New Political Methodology: Microfoundations and ART.” Annual Review of Political Science 5:423–450.CrossRefGoogle Scholar
Altonji, J. G., Elder, T. E., and Taber, C. R.. 2005. “An Evaluation of Instrumental Variable Strategies for Estimating the Effects of Catholic Schooling.” The Journal of Human Resources 40(4):791–821.CrossRefGoogle Scholar
Athey, S., Imbens, G. W., and Wager, S.. 2018. “Efficient Inference of Average Treatment Effects in High Dimensions via Approximate Residual Balancing.” Journal of the Royal Statistical Society-Series B 80(4):597–623.CrossRefGoogle Scholar
Bartels, L. M. 1997. “Specification Uncertainty and Model Averaging.” American Journal of Political Science 41:641–674.CrossRefGoogle Scholar
Bhattacharya, J., and Vogt, W. B.. 2012. “Do Instrumental Variables Belong in Propensity Scores?” International Journal of Statistics & Economics 9(A12):107–127.Google Scholar
Bobko, P. 2001. Correlation and Regression: Applications for Industrial Organizational Psychology and Management. Thousand Oaks, CA: SAGE.CrossRefGoogle Scholar
Brodeur, A., Lé, M., Sangnier, M., and Zylberberg, Y.. 2016. “Star Wars: The Empirics Strike Back.” American Economic Journal: Applied Economics 8(1):1–32.Google Scholar
Casey, K., Glennerster, R., and Miguel, E.. 2012. “Reshaping Institutions: Evidence on Aid Impacts Using a Preanalysis Plan.” The Quarterly Journal of Economics 127(4):1755–1812.CrossRefGoogle Scholar
Clarke, K. A. 2005. “The Phantom Menace: Omitted Variable Bias in Econometric Research.” Conflict Management and Peace Science 22(4):341–352.CrossRefGoogle Scholar
Cole, S. R. et al. 2010. “Illustrating Bias due to Conditioning on a Collider.” International Journal of Epidemiology 39(2):417–420.CrossRefGoogle ScholarPubMed
Conger, A. J. 1974. “A Revised Definition for Suppressor Variables: A Guide to their Identification and Interpretation.” Educational and Psychological Measurement 34(1):35–46.CrossRefGoogle Scholar
Crede, M., Gelman, A., and Nickerson, C.. 2016. “Questionable Association Between Front Boarding and Air Rage.” Proceedings of the National Academy of Sciences 113(47):E7348.CrossRefGoogle ScholarPubMed
Davenport, T. C. 2015. “Policy-Induced Risk and Responsive Participation: The Effect of a Son’s Conscription Risk on the Voting Behavior of his Parents.” American Journal of Political Science 59(1):225–241.CrossRefGoogle Scholar
Gelbach, J. B. 2016. “When Do Covariates Matter? and Which Ones, and How Much?” Journal of Labor Economics 34(2):509–543.CrossRefGoogle Scholar
Gerber, A., and Malhotra, N.. 2008. “Do Statistical Reporting Standards Affect What is Published? Publication Bias in Two Leading Political Science Journals.” Quarterly Journal of Political Science 3(3):313–326.CrossRefGoogle Scholar
Gerber, A. S., Malhotra, N., Dowling, C. M., and Doherty, D.. 2010. “Publication Bias in Two Political Behavior Literatures.” American Politics Research 38(4):591–613.CrossRefGoogle Scholar
Hainmueller, J. 2012. “Entropy Balancing for Causal Effects: A Multivariate Reweighting Method to Produce Balanced Samples in Observational Studies.” Political Analysis 20(1):25–46.CrossRefGoogle Scholar
Ho, D. E., Imai, K., King, G., and Stuart, E. A.. 2007. “Matching as Nonparametric Preprocessing for Reducing Model Dependence in Parametric Causal Inference.” Political Analysis 15(3):199–236.CrossRefGoogle Scholar
Holling, H. 1983. “Suppressor Structures in the General Linear Model.” Educational and Psychological Measurement 43(1):1–9.CrossRefGoogle Scholar
Horst, P. 1941. “The Role of Predictor Variables Which are Independent of the Criterion.” Social Science Research Council 48(4):431–436.Google Scholar
Humphreys, M., de la Sierra, R. S., and van der Windt, P.. 2013. “Fishing, Commitment, and Communication: A Proposal for Comprehensive Nonbinding Research Registration.” Political Analysis 21(1):1–20.CrossRefGoogle Scholar
Imai, K., and Ratkovic, M.. 2014. “Covariate Balancing Propensity Score.” Journal of the Royal Statistical Society: Series B (Statistical Methodology) 76(1):243–263.CrossRefGoogle Scholar
Ioannidis, J. P. A 2005. “Why Most Published Research Findings are False.” PLoS Medicine 2(8):e124.CrossRefGoogle ScholarPubMed
Ioannidis, J. P. A., Stanley, T. D., and Doucouliagos, H.. 2017. “The Power of Bias in Economics Research.” The Economic Journal 127:F236–F265.CrossRefGoogle Scholar
King, G. 1995. “Replication, Replication.” PS: Political Science and Politics 28(3):444–452.Google Scholar
Klein, R. A. et al. 2014. “Investigating Variation in Replicability.” Social Psychology 14(3):142–152.CrossRefGoogle Scholar
Klein, R. A. et al. 2018. “Many Labs 2: Investigating Variation in Replicability Across Samples and Settings.” Advances in Methods and Practices in Psychological Science 1(4):443–490.CrossRefGoogle Scholar
Leamer, E. E. 1983. “Let’s Take the Con Out of Econometrics.” The American Economic Review 73:31–43.Google Scholar
Leamer, E. E. 2016. “S-Values: Conventional Context-Minimal Measures of the Sturdiness of Regression Coefficients.” Journal of Econometrics 193(1):147–161.CrossRefGoogle Scholar
Lenz, G. S., and Sahn, A.. 2020. “Replication Data for: Achieving Statistical Significance with Control Variables and Without Transparency.” https://doi.org/10.7910/DVN/XIEJCR, Harvard Dataverse, V1, UNF:6:X6QRQK+3UeiBgJMBj8re2g== [fileUNF].Google Scholar
Lewis, J. W., and Escobar, L. A.. 1986. “Suppression and Enhancement in Bivariate Regression.” Journal of the Royal Statistical Society: Series D (the Statistician) 35(1):17–26.Google Scholar
MacKinnon, D. P., Krull, J. L., and Lockwood, C. M.. 2000. “Equivalence of the Mediation, Confounding and Suppression Effect.” Prevention Science 1(4):173–181.CrossRefGoogle ScholarPubMed
McShane, B. B., Gal, D., Gelman, A., Robert, C., and Tackett, J. L.. 2019. “Abandon Statistical Significance.” The American Statistician 73(sup1):235–245.CrossRefGoogle Scholar
Middleton, J. A., Scott, M. A., Diakow, R., and Hill, J. L.. 2016. “Bias Amplification and Bias Unmasking.” Political Analysis 24(3):307–323.CrossRefGoogle Scholar
Montgomery, J. M., and Nyhan, B.. 2010. “Bayesian Model Averaging: Theoretical Developments and Practical Applications.” Political Analysis 18(2):245–270.CrossRefGoogle Scholar
Nickerson, C. A., and Brown, N. J. L.. 2019. “Simpson’s Paradox is Suppression, but Lord’s Paradox is Neither: Clarification of and Correction to Tu, Gunnell, and Gilthorpe (2008).” Emerging Themes in Epidemiology 16(1):5–16.CrossRefGoogle Scholar
Ning, Y., Peng, S., and Imai, K.. 2020. “High Dimensional Propensity Score Estimation via Covariate Balancing.” Biometrika 107(3):533–554.Google Scholar
Oster, E. 2016. “Unobservable Selection and Coefficient Stability: Theory and Evidence.” Journal of Business & Economic Statistics 37(2):187–204.CrossRefGoogle Scholar
Pearl, J. 2009. Causality. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
Pearl, J. 2010. “On a Class of Bias-Amplifying Variables that Endanger Effect Estimates.” In Proceedings of the Twenty-Sixth Conference on Uncertainty in Artificial Intelligence, 417–424. Corvallis, OR: AUAI.Google Scholar
Pearl, J. 2011. “Invited Commentary: Understanding Bias Amplification.” American Journal of Epidemiology 174(11):1223–1227.CrossRefGoogle ScholarPubMed
Peters, J., Bühlmann, P., and Meinshausen, N.. 2016. “Causal Inference by Using Invariant Prediction: Identification and Confidence Intervals.” Journal of the Royal Statistical Society: Series B (Statistical Methodology) 78(5):947–1012.CrossRefGoogle Scholar
Sekhon, J. S. 2011. “Multivariate and Propensity Score Matching Software with Automated Balance Optimization: The Matching Package for R.” Journal of Statistical Software 42(7):1–52.CrossRefGoogle Scholar
Simonsohn, U., Simmons, J. P., and Nelson, L. D.. 2020. “Specification Curve: Descriptive and Inferential Statistics on All Reasonable Specifications.” Nature Human Behavior. Google Scholar
Tesler, M., and Sears, D. O.. 2010. Obama’s Race: The 2008 Election and the Dream of a Post-Racial America. Chicago, IL: University of Chicago Press.CrossRefGoogle Scholar
Tzelgov, J., and Henik, A.. 1991. “Suppression Situations in Psychological Research: Definitions, Implications, and Applications.” Psychological Bulletin 109(3):524.CrossRefGoogle Scholar
Wooldridge, J. M. 2016. “Should Instrumental Variables be Used as Matching Variables?” Research in Economics 70(2):232–237.CrossRefGoogle Scholar
Wyss, R., Lunt, M., , M. A. Brookhart, Glynn, R. J., and Stürmer, T.. 2014. “Reducing Bias Amplification in the Presence of Unmeasured Confounding Through Out-of-Sample Estimation Strategies for the Disease Risk Score.” Journal of Causal Inference 2(2):131–146.CrossRefGoogle ScholarPubMed
Supplementary material: PDF

Lenz and Sahn supplementary material

Lenz and Sahn supplementary material

Download Lenz and Sahn supplementary material(PDF)
PDF 293.1 KB
Supplementary material: Link

Lenz and Sahn Dataset

Dataset

https://doi.org/10.7910/DVN/XIEJCR
Link