Skip to main content Accessibility help

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings.

Close cookie message
×
Hostname: page-component-cd9895bd7-jn8rn Total loading time: 0 Render date: 2024-12-22T19:47:39.789Z Has data issue: false hasContentIssue false

6 - Research Design and Issues of Validity

from Part II - Basic Design Considerations to Know, No Matter What Your Research Is About

Published online by Cambridge University Press:  12 December 2024

By
Marilynn B. Brewer  and
William D. Crano
Edited by
Harry T. Reis ,
Tessa West  and
Charles M. Judd
Harry T. Reis
Affiliation:
University of Rochester, New York
Tessa West
Affiliation:
New York University
Charles M. Judd
Affiliation:
University of Colorado Boulder
Get access

Summary

The validity of conclusions drawn from specific research studies must be evaluated in light of the purposes for which the research was undertaken. We distinguish four general types of research: description and point estimation, correlation and prediction, causal inference, and explanation. For causal and explanatory research, internal validity is critical – the extent to which a causal relationship can be inferred from the results of variation in the independent and dependent variables of an experiment. Random assignment is discussed as the key to avoiding threats to internal validity. Internal validity is distinguished from construct validity (the relationship between a theoretical construct and the methods used to operationalize that concept) and external validity (the extent to which the results of a research study can be generalized to other contexts). Construct validity is discussed in terms of multiple operations and discriminant and convergent validity assessment. External validity is discussed in terms of replicability, robustness, and relevance of specific research findings.

Keywords

causationinternal validityexternal validityconstruct validityrandomizationdiscriminant validityconvergent validitymediationmoderation
Type
Chapter
Information
Handbook of Research Methods in Social and Personality Psychology , pp. 115 - 135
Publisher: Cambridge University Press
Print publication year: 2024

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Anderson, C. A., Allen, J. J., Plante, C., Quigley-McBride, A., Lovett, A., and Rokkum, J. N. (2019). The MTurkification of social and personality psychology. Personality and Social Psychology Bulletin, 45, 842–850.CrossRefGoogle ScholarPubMed
Aronson, E., and Mills, J. (1959). The effect of severity of initiation on liking for a group. Journal of Abnormal and Social Psychology, 59, 177–181.CrossRefGoogle Scholar
Aronson, E., Wilson, T., and Brewer, M. B. (1998). Experimentation in social psychology. In Gilbert, D., Fiske, S., and Lindzey, G. (eds) The Handbook of Social Psychology, 4th ed., vol. 1. Boston: McGraw-Hill.Google Scholar
Asch, S. E. (1956). Studies of independence and conformity: A minority of one against a unanimous majority. Psychological Monographs, 70(9), whole no. 416.CrossRefGoogle Scholar
Baron, R. M., and Kenny, D. A. (1986). The moderator–mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations. Journal of Personality and Social Psychology, 51, 1173–1182.CrossRefGoogle ScholarPubMed
Berkowitz, L., and Donnerstein, E. (1982). External validity is more than skin deep: Some answers to criticisms of laboratory experiments. American Psychologist, 37, 245–257.CrossRefGoogle Scholar
Brewer, M. B. (1997). The social psychology of intergroup relations: Can research inform practice? Journal of Social Issues, 53(1), 197–211.CrossRefGoogle Scholar
Brunswik, E. (1956). Perception and the Representative Design of Psychological Experiments, 2nd ed. University of California Press.CrossRefGoogle Scholar
Bullock, J. G., and Green, D. P. (2021). The failings of conventional mediation analysis and a design-based alternative. Advances in Methods and Practices in Psychological Science, 4(4), 1–18.CrossRefGoogle Scholar
Campbell, D. T. (1957). Factors relevant to the validity of experiments in social settings. Psychological Bulletin, 54, 297–312.CrossRefGoogle Scholar
Campbell, D. T., and Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait–multimethod matrix. Psychological Bulletin, 56, 81–105.CrossRefGoogle ScholarPubMed
Campbell, D. T., and Stanley, J. C. (1963). Experimental and quasi-experimental designs for research on teaching. In Gage, N. (ed.) Handbook of Research on Teaching. Rand-McNally.Google Scholar
Campbell, D. T., and Stanley, J. C. (1966). Experimental and Quasi-experimental Designs for Research. Rand-McNally.Google Scholar
Chester, D. S., and Lasko, E. N. (2021). Construct validation of experimental manipulations in social psychology: Current practices and recommendations for the future. Perspectives on Psychological Science, 16, 377–395.CrossRefGoogle ScholarPubMed
Chmielewski, M., and Kucker, S. C. (2020). An MTurk crisis? Shifts in data quality and the impact on study results. Social Psychological and Personality Science, 11, 464–473.CrossRefGoogle Scholar
Collingwood, R. G. (1940). An Essay on Metaphysics. Clarendon.Google Scholar
Cook, T. D., and Campbell, D. T. (1979). Quasi-experimentation: Design and Analysis Issues for Field Settings. Rand-McNally.Google Scholar
Cook, T. D., and Shadish, W. R. (1994). Social experiments: Some developments over the past fifteen years. Annual Review of Psychology, 45, 545–580.CrossRefGoogle Scholar
Crandall, C. S., and Sherman, J. W. (2016). On the scientific superiority of conceptual replications for scientific progress. Journal of Experimental Social Psychology, 66, 93–99.CrossRefGoogle Scholar
Cronbach, L. J. (1982). Designing Evaluations of Educational and Social Programs. Jossey-Bass.Google Scholar
Ejelov, E., and Luke, T. (2020). “Rarely safe to assume”: Evaluating the use and interpretation of manipulations checks in experimental social psychology. Journal of Experimental Social Psychology, 87, Article 103037.CrossRefGoogle Scholar
Fiedler, K., Schott, M., and Meiser, T. (2011). What mediation analysis can (not) do. Journal of Experimental Social Psychology, 47, 1231–1236.CrossRefGoogle Scholar
Flake, J. K., and Fried, E. I. (2020). Measurement schmeasurement: Questionable measurement practices and how to avoid them. Advances in Methods and Practices in Psychological Science, 3, 456–465.CrossRefGoogle Scholar
Flake, J. K., Pek, J., and Hehman, E. (2017). Construct validation in social and personality research: Current practice and recommendations. Social Psychological and Personality Science, 8, 370–378.CrossRefGoogle Scholar
Gasking, D. (1955). Causation and recipes. Mind, 64, 479–487.CrossRefGoogle Scholar
Gerard, H. B., and Mathewson, G. C. (1966). The effects of severity of initiation on liking for a group: A replication. Journal of Experimental Social Psychology, 2, 278–287.CrossRefGoogle Scholar
Greenwald, A. G. (1975). On the inconclusivenes of “crucial” cognitive tests of dissonance versus self-perception theories. Journal of Experimental Social Psychology, 11, 490–499.CrossRefGoogle Scholar
Greenwood, J. D. (1982). On the relation between laboratory experiments and social behaviour: Causal explanation and generalization. Journal for the Theory of Social Behaviour, 12, 225–250.CrossRefGoogle Scholar
Gruijters, S. L. K. (2022). Making inferential leaps: Manipulation checks and the roads towards strong inference. Journal of Experimental Social Psychology, 98, Article 104251.CrossRefGoogle Scholar
Henrich, J., Heine, S. J., and Norenzayan, A. (2010a). Beyond WEIRD: Towards a broad-based behavioral science. Behavioral and Brain Sciences, 33(2–3), 111–135.CrossRefGoogle Scholar
Henrich, J., Heine, S. J., and Norenzayan, A. (2010b). The weirdest people in the world? Behavioral and Brain Sciences, 33(2–3), 61–83.CrossRefGoogle ScholarPubMed
Imai, K., and Jiang, Z. (2020). Identification and sensitivity analysis of contagion effects in randomized placebo‐controlled trials. Journal of the Royal Statistical Society: Series A (Statistics in Society), 183(4), 1637–1657.CrossRefGoogle Scholar
Kenny, D. A. (1995). The multitrait–multimethod matrix: Design, analysis, and conceptual issues. In Shrout, P. E. and Fiske, S. T. (eds.) Personality Research, Methods, and Theory: A Festschrift Honoring Donald W. Fiske. Erlbaum.Google Scholar
Kihlstrom, J. F. (2021). Ecological validity and “ecological validity.” Perspectives on Psychological Science, 16, 466–471.CrossRefGoogle ScholarPubMed
Klein, R. A., et al. (2018). Many labs 2: Investigating variation in replicability across samples and settings. Advances in Methods and Practices in Psychological Science, 1, 443–490.CrossRefGoogle Scholar
Liu, X., and Wang, L. (2021). The impact of measurement error and omitting confounders on statistical inference of mediation effects and tools for sensitivity analysis. Psychological Methods, 26, 327–342.CrossRefGoogle ScholarPubMed
Mackie, J. L. (1974). The Cement of the Universe. Oxford University Press.Google Scholar
Markus, H. R., and Kitayama, S. (1991). Culture and the self: Implications for cognition, emotion, and motivation. Psychological Review, 98, 224–253.CrossRefGoogle Scholar
Marsh, H. W., Hau, K. T., Wen, Z., Nagengast, B., and Morin, A. J. S. (2011). Moderation. In Little, T. D. (ed.) Oxford Handbook of Quantitative Methods. Oxford University Press.Google Scholar
Milgram, S. (1963). Behavioral study of obedience. Journal of Abnormal and Social Psychology, 67, 371–378.CrossRefGoogle ScholarPubMed
Mook, D. G. (1983). In defense of external invalidity. American Psychologist, 38, 379–387.CrossRefGoogle Scholar
Nadler, J., Baumgartner, S., and Washington, M. (2021). MTurk for working samples: Evaluation of data quality 2014–2020. North American Journal of Psychology, 23, 741–752.Google Scholar
Nosek, B. A., et al. (2022). Replicability, robustness, and reproducibility in psychological science. Annual Review of Psychology, 73, 719–748.CrossRefGoogle ScholarPubMed
Orne, M. (1962). On the social psychology of the psychological experiment. American Psychologist, 17, 776–783.CrossRefGoogle Scholar
Orne, M., and Holland, C. (1968). Some conditions of obedience and disobedience to authority. On the ecological validity of laboratory deceptions. International Journal of Psychiatry, 6, 282–293.Google ScholarPubMed
Petty, R. E., and Cacioppo, J. T. (1996). Addressing disturbing and disturbed consumer behavior: Is it necessary to change the way we conduct behavioral science? Journal of Marketing Research, 33, 1–8.CrossRefGoogle Scholar
Qin, X., and Yang, F. (2022). Simulation-based sensitivity analysis for causal mediation studies. PsychologicalMethods, https://doi.org/10.1037/met0000340.CrossRefGoogle Scholar
Rakover, S. S. (1981). Social psychology theory and falsification. Personality and Social Psychology Bulletin, 7, 123–130.CrossRefGoogle Scholar
Reichardt, C. S., and Coleman, S. C. (1995). The criteria for convergent and discriminant validity in a multitrait–multimethod matrix. Multivariate Behavioral Research, 30, 513–538.CrossRefGoogle Scholar
Rohrer, J. M. (2018). Thinking clearly about correlations and causation: Graphical causal models for observational data. Advances in Methods and Practices in Psychological Science, 1, 27–42.CrossRefGoogle Scholar
Rosenberg, M. J. (1969). The conditions and consequences of evaluation apprehension. In Rosenthal, R. and Rosnow, R. (eds) Artifact in Behavioral Research. Academic Press.Google Scholar
Rosenthal, R. (1966). Experimenter Effects in Behavioral Research. Appleton-Century-Crofts.Google Scholar
Sampson, E. E. (1977). Psychology and the American ideal. Journal of Personality and Social Psychology, 35, 767–782.CrossRefGoogle Scholar
Sears, D. O. (1986). College sophomores in the laboratory: Influence of a narrow data base on social psychology’s view of human nature. Journal of Personality and Social Psychology, 51, 515–530.CrossRefGoogle Scholar
Shadish, W. R., Cook, T. D., and Campbell, D. T. (2001). Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton Mifflin.Google Scholar
Sherif, M. (1935). A study of some social factors in perception. Archives of Psychology, 27(187), 1–60.Google Scholar
Sivacek, J., and Crano, W. D. (1982). Vested interest as a moderator of attitude behavior consistency. Journal of Personality and Social Psychology, 43, 210–221.CrossRefGoogle Scholar
Toich, M. J., Schutt, E., and Fisher, D. M. (2022). Do you get what you pay for? Preventing insufficient effort responding in MTurk and student samples. Applied Psychology: An International Review, 71, 640–661.CrossRefGoogle Scholar
Vazire, S., Schiavone, S. R., and Bottesini, J. G. (2022). Credibility beyond replicability: Improving the four validities in psychological science. Current Directions in Psychological Science, 31, 162–168.CrossRefGoogle Scholar
Zanna, M., and Cooper, J. (1974). Dissonance and the pill: An attribution approach to studying the arousal properties of dissonance. Journal of Personality and Social Psychology, 29, 703–709.CrossRefGoogle ScholarPubMed
Zhou, H., and Fishbach, A. (2016). The pitfall of experimenting on the web: How unattended selective attrition leads to surprising (yet false) research conclusions. Journal of Personality and Social Psychology, 111, 493–504.CrossRefGoogle ScholarPubMed
Zillman, D. (1983). Transfer of excitation in emotional behavior. In Cacioppo, J. T. and Petty, R. E. (eds) Social Psychophysiology: A Sourcebook. Guilford Press.Google Scholar

Save book to Kindle

To save this book to your Kindle, first ensure [email protected] is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Available formats
×

Save book to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox.

Available formats
×

Save book to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive.

Available formats
×