Skip to main content Accessibility help

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings.

Close cookie message
×
Hostname: page-component-cd9895bd7-7cvxr Total loading time: 0 Render date: 2025-01-05T11:53:43.144Z Has data issue: false hasContentIssue false

19 - Archival Data

from Part III - Data Collection

Published online by Cambridge University Press:  25 May 2023

By
Jason Miller
Edited by
Austin Lee Nichols  and
John Edlund
Austin Lee Nichols
Affiliation:
Central European University, Vienna
John Edlund
Affiliation:
Rochester Institute of Technology, New York
Get access

Summary

Social and behavioral researchers often draw on archival data – data collected by an entity other than the research team – to conduct scientific inquiry. Researchers typically seek to make claims about measured variables that extend beyond the measures themselves, such as interpreting a measure as representing an unobservable theoretical construct. Though researchers using archival data encounter many issues, this chapter focuses on two that have received less attention. The first concerns how researchers should justify the interpretations and uses they attach to archival measures. The second concerns how to justify generalizing findings. This chapter provides a framework to help researchers address these issues by drawing on contemporary validity theory in education and psychology as well as theory regarding causal mechanisms from philosophy and sociology. These concepts are illustrated using multiple examples from published studies.

Keywords

Archival DataValidityMechanismGeneralizability
Type
Chapter
Information
The Cambridge Handbook of Research Methods and Statistics for the Social and Behavioral Sciences
Volume 1: Building a Program of Research
 , pp. 400 - 419
Publisher: Cambridge University Press
Print publication year: 2023

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Aiken, L. S. & West, S. G. (1991). Multiple Regression: Testing and Interpreting Interactions. SAGE Publications.Google Scholar
Ali, A., Klasa, S., & Yeung, E. (2008). The limitations of industry concentration measures constructed with Compustat data: Implications for finance research. Review of Financial Studies, 22(10), 3839–3871.CrossRefGoogle Scholar
Almon, S. (1965). The distributed lag between capital appropriations and expenditures. Econometrica, 33(1), 178–196.Google Scholar
Angrist, J. D. & Pischke, J. S. (2010). The credibility revolution in empirical economics: How better research design is taking the con out of econometrics. Journal of Economic Perspectives, 24(2), 3–30.CrossRefGoogle Scholar
Asparouhov, T. & Muthén, B. (2014). Multiple-group factor analysis alignment. Structural Equation Modeling: A Multidisciplinary Journal, 21(4), 495–508.CrossRefGoogle Scholar
Asparouhov, T. & Muthén, B. (2020). Comparison of models for the analysis of intensive longitudinal data. Structural Equation Modeling: A Multidisciplinary Journal, 27(2), 275–297.Google Scholar
Asparouhov, T., Hamaker, E. L., & Muthén, B. (2018). Dynamic structural equation models. Structural Equation Modeling: A Multidisciplinary Journal, 25(3), 359–388.CrossRefGoogle Scholar
Astbury, B. & Leeuw, F. L. (2010). Unpacking black boxes: Mechanisms and theory building in evaluation. American Journal of Evaluation, 31(3), 363–381.CrossRefGoogle Scholar
Bai, X. (2018). Forecasting short term trucking rates. Unpublished Master’s Thesis, Massachusetts Institute of Technology. Available at: https://dspace.mit.edu/handle/1721.1/117796.Google Scholar
Baker, S. R., Bloom, N., & Davis, S. J. (2016). Measuring economic policy uncertainty. Quarterly Journal of Economics, 131(4), 1593–1636.Google Scholar
Basu, S. (2019). Are price-cost markups rising in the United States? A discussion of the evidence. Journal of Economic Perspectives, 33(3), 3–22.CrossRefGoogle Scholar
Bauer, D. J. & Hussong, A. M. (2009). Psychometric approaches for developing commensurate measures across independent studies: Traditional and new models. Psychological Methods, 14(2), 101–125.CrossRefGoogle ScholarPubMed
Blinder, A. S. & Watson, M. W. (2016). Presidents and the US economy: An econometric exploration. American Economic Review, 106(4), 1015–45.Google Scholar
Bloom, N., Sadun, R., & Van Reenen, J. (2012). Americans do IT better: US multinationals and the productivity miracle. American Economic Review, 102(1), 167–201.Google Scholar
Bollen, K. A. (1989). Structural Equations with Latent Variables. John Wiley & Sons.Google Scholar
Bollen, K. A. (2002). Latent variables in psychology and the social sciences. Annual Review of Psychology, 53(1), 605–634.Google Scholar
Braguinsky, S., Ohyama, A., Okazaki, T., & Syverson, C. (2015). Acquisitions, productivity, and profitability: Evidence from the Japanese cotton spinning industry. American Economic Review, 105(7), 2086–2119.Google Scholar
Brave, S. A., Butters, R. A., & Fogarty, M. (2021). The perils of working with big data and a SMALL checklist you can use to recognize them. Business Horizons, 65(4), 481–492. https://doi.org/10.1016/j.bushor.2021.06.004CrossRefGoogle Scholar
Browne, M. W. (2001). An overview of analytic rotation in exploratory factor analysis. Multivariate Behavioral Research, 36(1), 111–150.CrossRefGoogle Scholar
Browne, M. W., MacCallum, R. C., Kim, C. T., Andersen, B. L., & Glaser, R. (2002). When fit indices and residuals are incompatible. Psychological Methods, 7(4), 403–421.CrossRefGoogle ScholarPubMed
Bunge, M. (2004). How does it work? The search for explanatory mechanisms. Philosophy of the Social Sciences, 34(2), 182–210.CrossRefGoogle Scholar
Bureau of Economic Analysis (2021). Personal income. Available at: https://fred.stlouisfed.org/series/PI.Google Scholar
Bureau of Labor Statistics (2021a). Handbook of methods. Available at: www.bls.gov/opub/hom/home.htm.Google Scholar
Bureau of Labor Statistics (2021b). Producer price indexes. Available at: www.bls.gov/pPI/.Google Scholar
Bureau of Labor Statistics (2021c). Producer price index by industry: General freight trucking, long-distance truckload (PCU484121484121). Available at: https://fred.stlouisfed.org/series/PCU484121484121.Google Scholar
Casciaro, T. & Piskorski, M. J. (2005). Power imbalance, mutual dependence, and constraint absorption: A closer look at resource dependence theory. Administrative Science Quarterly, 50(2), 167–199.Google Scholar
Census Bureau (2014). American community survey design and methodology (January 2014). Available at: www2.census.gov/programs-surveys/acs/methodology/design_and_methodology/acs_design_methodology_report_2014.pdf.Google Scholar
Census Bureau (2020). 2017 Commodity flow survey methodology. Available at: www2.census.gov/programs-surveys/cfs/technical-documentation/methodology/2017cfsmethodology.pdf#:~:text=%20%20%20Title%20%20%202017%20Commodity,Created%20Date%20%20%201%2F22%2F2021%2012%3A21%3A48%20PM%20.Google Scholar
Census Bureau (2021a). Economic census, technical documentation, methodology, nonsampling error. Available at: www.census.gov/programs-surveys/economic-census/technical-documentation/methodology.html#nonsampling-error.Google Scholar
Census Bureau (2021b). Monthly state retail sales technical documentation. Available at: www.census.gov/retail/mrts/www/statedata/msrs_technical_documentation.pdf.Google Scholar
Cizek, G. J. (2012). Defining and distinguishing validity: Interpretations of score meaning and justifications of test use. Psychological Methods, 17(1), 31–43.Google Scholar
Cook, D. A. & Beckman, T. J. (2006). Current concepts in validity and reliability for psychometric instruments: Theory and application. American Journal of Medicine, 119(2), 166.e7–166.e16.CrossRefGoogle ScholarPubMed
Cudeck, R. (1985). A structural comparison of conventional and adaptive versions of the ASVAB. Multivariate Behavioral Research, 20(3), 305–322.CrossRefGoogle ScholarPubMed
Cudeck, R. & Henly, S. J. (1991). Model selection in covariance structures analysis and the” problem” of sample size: A clarification. Psychological Bulletin, 109(3), 512–519.Google Scholar
DAT Freight & Analytics (2021). National van rates. Available at: www.dat.com/industry-trends/trendlines/van/national-rates.Google Scholar
Downing, S. M. (2003). Validity: On the meaningful interpretation of assessment data. Medical Education, 37(9), 830–837.CrossRefGoogle ScholarPubMed
Emerson, R. M. (1962). Power-dependence relations. American Sociological Review, 27(1), 31–41.CrossRefGoogle Scholar
Enders, W. (2015). Applied Econometric Time Series, 4th ed. John Wiley & Sons.Google Scholar
Espeland, W. N. & Sauder, M. (2007). Rankings and reactivity: How public measures recreate social worlds. American Journal of Sociology, 113(1), 1–40.Google Scholar
Falleti, T. G. & Lynch, J. F. (2009). Context and causal mechanisms in political analysis. Comparative Political Studies, 42(9), 1143–1166.CrossRefGoogle Scholar
Forbes, S. J., Lederman, M., & Tombe, T. (2015). Quality disclosure programs and internal organizational practices: Evidence from airline flight delays. American Economic Journal: Microeconomics, 7(2), 1–26. https://www.aeaweb.org/articles?id=10.1257/mic.20130164.Google Scholar
Foster, S. L. & Cone, J. D. (1995). Validity issues in clinical assessment. Psychological Assessment, 7(3), 248–260.CrossRefGoogle Scholar
Frazier, G. L. (1983). On the measurement of interfirm power in channels of distribution. Journal of Marketing Research, 20(2), 158–166.Google Scholar
General Social Survey (2021). About the GSS. Available at: https://gss.norc.org/About-The-GSS.Google Scholar
Gentzkow, M. & Shapiro, J. M. (2010). What drives media slant? Evidence from US daily newspapers. Econometrica, 78(1), 35–71.Google Scholar
Goldsby, T. J., Michael Knemeyer, A., Miller, J. W., & Wallenburg, C. M. (2013). Measurement and moderation: Finding the boundary conditions in logistics and supply chain research. Journal of Business Logistics, 34(2), 109–116.Google Scholar
Granovetter, M. S. (1973). The strength of weak ties. American Journal of Sociology, 78(6), 1360–1380.Google Scholar
Hamaker, E. L., Asparouhov, T., Brose, A., Schmiedek, F., & Muthén, B. (2018). At the frontiers of modeling intensive longitudinal data: Dynamic structural equation models for the affective measurements from the COGITO study. Multivariate Behavioral Research, 53(6), 820–841.CrossRefGoogle ScholarPubMed
Hedström, P. & Ylikoski, P. (2010). Causal mechanisms in the social sciences. Annual Review of Sociology, 36, 49–67.CrossRefGoogle Scholar
Heide, J. B. & John, G. (1988). The role of dependence balancing in safeguarding transaction-specific assets in conventional channels. Journal of Marketing, 52(1), 20–35.CrossRefGoogle Scholar
Horowitz, K. J. & Planting, M. A. (2009). Concepts and methods of the US input–output accounts. Available at: www.bea.gov/sites/default/files/methodologies/IOmanual_092906.pdf.Google Scholar
Ibanez, M. R. & Toffel, M. W. (2020). How scheduling can bias quality assessment: Evidence from food-safety inspections. Management Science, 66(6), 2396–2416.CrossRefGoogle Scholar
Jin, G. Z. & Leslie, P. (2003). The effect of information on product quality: Evidence from restaurant hygiene grade cards. Quarterly Journal of Economics, 118(2), 409–451.CrossRefGoogle Scholar
Kane, M. T. (1992). An argument-based approach to validity. Psychological Bulletin, 112(3), 527–535.CrossRefGoogle Scholar
Kane, M. T. (2001). Current concerns in validity theory. Journal of Educational Measurement, 38(4), 319–342.CrossRefGoogle Scholar
Kane, M. T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 1–73.Google Scholar
Kane, T. J. & Staiger, D. O. (2002). The promise and pitfalls of using imprecise school accountability measures. Journal of Economic Perspectives, 16(4), 91–114.Google Scholar
Ketchen, D. J., Ireland, R. D., & Baker, L. T. (2013). The use of archival proxies in strategic management studies: Castles made of sand? Organizational Research Methods, 16(1), 32–42.Google Scholar
Lipton, P. (2004). Inference to the Best Explanation, 2nd ed. RoutledgeGoogle Scholar
Little, T. D. (2013). Longitudinal Structural Equation Modeling. Guilford Press.Google Scholar
Macher, J. T., Mayo, J. W., & Nickerson, J. A. (2011). Regulator heterogeneity and endogenous efforts to close the information asymmetry gap. Journal of Law and Economics, 54(1), 25–54.CrossRefGoogle Scholar
Mahoney, J. (2001). Beyond correlational analysis: Recent innovations in theory and method. Sociological Forum 16(3), 575–593.Google Scholar
McKendall, M. A. & Wagner, J. A., III (1997). Motive, opportunity, choice, and corporate illegality. Organization Science, 8(6), 624–647.CrossRefGoogle Scholar
McKone, K. E. & Weiss, E. N. (1998). TPM: Planned and autonomous maintenance – bridging the gap between practice and research. Production and Operations Management, 7(4), 335–351.Google Scholar
Meehl, P. E. (1990). Appraising and amending theories: The strategy of Lakatosian defense and two principles that warrant it. Psychological Inquiry, 1(2), 108–141.Google Scholar
Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741–749.CrossRefGoogle Scholar
Miller, J. & Parast, M. M. (2019). Learning by applying: The case of the Malcolm Baldrige National Quality Award. IEEE Transactions on Engineering Management, 66(3), 337–353.Google Scholar
Miller, J. W. & Saldanha, J. P. (2016). A new look at the longitudinal relationship between motor carrier financial performance and safety. Journal of Business Logistics, 37(3), 284–306.Google Scholar
Miller, J. & Saldanha, J. P. (2018). An exploratory investigation of new entrant motor carriers’ longitudinal safety performance. Transportation Journal, 57(2), 163–192.CrossRefGoogle Scholar
Miller, J. W., Golicic, S. L., & Fugate, B. S. (2018). Reconciling alternative theories for the safety of owner–operators. Journal of Business Logistics, 39(2), 101–122.CrossRefGoogle Scholar
Miller, J. W., Muir, W. A., Bolumole, Y., & Griffis, S. E. (2020). The effect of truckload driver turnover on truckload freight pricing. Journal of Business Logistics, 41(4), 294–309.Google Scholar
Miller, J. W., Bolumole, Y., & Muir, W. A. (2021a). Exploring longitudinal industry‐level large truckload driver turnover. Journal of Business Logistics, 42(4), 428–450. https://doi.org/10.1111/jbl.12235Google Scholar
Miller, J. W., Scott, A., & Williams, B. D. (2021b). Pricing dynamics in the truckload sector: The moderating role of the electronic logging device mandate. Journal of Business Logistics, 42(4), 388–405. https://doi.org/10.1111/jbl.12256CrossRefGoogle Scholar
Miller, J., Davis‐Sramek, B., Fugate, B. S., Pagell, M., & Flynn, B. B. (2021c). Editorial commentary: Addressing confusion in the diffusion of archival data research. Journal of Supply Chain Management, 57(3), 130–146. https://doi.org/10.1111/jscm.12236CrossRefGoogle Scholar
Miller, J., Skowronski, K., & Saldanha, J. (2022) Asset ownership & incentives to undertake non‐contractible actions: The case of trucking. Journal of Supply Chain Management, 58, 65–91. https://doi.org/10.1111/jscm.12263CrossRefGoogle Scholar
Muir, W. A., Miller, J. W., Griffis, S. E., Bolumole, Y. A., & Schwieterman, M. A. (2019). Strategic purity and efficiency in the motor carrier industry: A multiyear panel investigation. Journal of Business Logistics, 40(3), 204–228.Google Scholar
Muthén, B. & Asparouhov, T. (2014). IRT studies of many groups: The alignment method. Frontiers in Psychology, 5, 978.Google ScholarPubMed
Muthén, B. & Asparouhov, T. (2018). Recent methods for the study of measurement invariance with many groups: Alignment and random effects. Sociological Methods & Research, 47(4), 637–664.CrossRefGoogle Scholar
Nyrup, R. (2015). How explanatory reasoning justifies pursuit: A Peircean view of IBE. Philosophy of Science, 82(5), 749–760.Google Scholar
Pawson, R. & Manzano-Santaella, A. (2012). A realist diagnostic workshop. Evaluation, 18(2), 176–191.Google Scholar
Peltzman, S. (2000). Prices rise faster than they fall. Journal of Political Economy, 108(3), 466–502.CrossRefGoogle Scholar
Sauder, M. & Espeland, W. N. (2009). The discipline of rankings: Tight coupling and organizational change. American Sociological Review, 74(1), 63–82.Google Scholar
Schwieterman, M. A., Miller, J., Knemeyer, A. M., & Croxton, K. L. (2020). Do supply chain exemplars have more or less dependent suppliers? Journal of Business Logistics, 41(2), 149–173.Google Scholar
Scott, A. (2015). The value of information sharing for truckload shippers. Transportation Research Part E: Logistics and Transportation Review, 81, 203–214.CrossRefGoogle Scholar
Scott, A. (2018). Carrier bidding behavior in truckload spot auctions. Journal of Business Logistics, 39(4), 267–281.Google Scholar
Scott, A. (2019). Concurrent business and buyer–supplier behavior in B2B auctions: Evidence from truckload transportation. Production and Operations Management, 28(10), 2609–2628.CrossRefGoogle Scholar
Scott, A. & Nyaga, G. N. (2019). The effect of firm size, asset ownership, and market prices on regulatory violations. Journal of Operations Management, 65(7), 685–709.CrossRefGoogle Scholar
Scott, A., Balthrop, A., & Miller, J. W. (2021). Unintended responses to IT‐enabled monitoring: The case of the electronic logging device mandate. Journal of Operations Management, 67(2), 152–181.Google Scholar
Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Houghton Mifflin.Google Scholar
Singer, J. D. and Willett, J. B. (2003). Applied Longitudinal Data Analysis: Modeling Change and Event Occurrence. Oxford University Press.CrossRefGoogle Scholar
Skowronski, K. & BentonJr, W. C. (2018). The influence of intellectual property rights on poaching in manufacturing outsourcing. Production and Operations Management, 27(3), 531–552.Google Scholar
Steel, D. (2004). Social mechanisms and causal inference. Philosophy of the Social Sciences, 34(1), 55–78.Google Scholar
Sutton, R. I. & Staw, B. M. (1995). What theory is not. Administrative Science Quarterly, 40(3), 371–384.Google Scholar
Syverson, C. (2004). Market structure and productivity: A concrete example. Journal of Political Economy, 112(6), 1181–1222.Google Scholar
Vegter, A., Taylor, J. K., & Haider-Markel, D. P. (2020). Old and new data sources and methods for interest group research. Interest Groups & Advocacy, 9(3), 436–450.CrossRefGoogle Scholar
Williamson, O. E. (2005). The economics of governance. American Economic Review, 95(2), 1–18.Google Scholar
Winter, S. G., Szulanski, G., Ringov, D., & Jensen, R. J. (2012). Reproducing knowledge: Inaccurate replication and failure in franchise organizations. Organization Science, 23(3), 672–685.Google Scholar
Wirth, R. J. & Edwards, M. C. (2007). Item factor analysis: Current approaches and future directions. Psychological Methods, 12(1), 58–79.Google Scholar
Zhang, G., Browne, M. W., Ong, A. D., & Chow, S. M. (2014). Analytic standard errors for exploratory process factor analysis. Psychometrika, 79(3), 444–469.Google Scholar

Save book to Kindle

To save this book to your Kindle, first ensure [email protected] is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

  • Archival Data
  • Edited by Austin Lee Nichols, Central European University, Vienna, John Edlund, Rochester Institute of Technology, New York
  • Book: The Cambridge Handbook of Research Methods and Statistics for the Social and Behavioral Sciences
  • Online publication: 25 May 2023
  • Chapter DOI: https://doi.org/10.1017/9781009010054.020
Available formats
×

Save book to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox.

  • Archival Data
  • Edited by Austin Lee Nichols, Central European University, Vienna, John Edlund, Rochester Institute of Technology, New York
  • Book: The Cambridge Handbook of Research Methods and Statistics for the Social and Behavioral Sciences
  • Online publication: 25 May 2023
  • Chapter DOI: https://doi.org/10.1017/9781009010054.020
Available formats
×

Save book to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive.

  • Archival Data
  • Edited by Austin Lee Nichols, Central European University, Vienna, John Edlund, Rochester Institute of Technology, New York
  • Book: The Cambridge Handbook of Research Methods and Statistics for the Social and Behavioral Sciences
  • Online publication: 25 May 2023
  • Chapter DOI: https://doi.org/10.1017/9781009010054.020
Available formats
×