Hostname: page-component-586b7cd67f-dsjbd Total loading time: 0 Render date: 2024-11-28T22:20:03.371Z Has data issue: false hasContentIssue false
  • English
  • Français

What Experiment Did We Just Do? Counterfactual Error Statistics and Uncertainties about the Reference Class

Published online by Cambridge University Press:  01 January 2022

Kent W. Staley
Kent W. Staley*
Affiliation:
Saint Louis University
*
Send requests for reprints to the author, Department of Philosophy, Saint Louis University, 3800 Lindell Blvd., P.O. Box 56907, St. Louis, MO 63156; [email protected].
Get access Rights & Permissions [Opens in a new window]

Abstract

Experimenters sometimes insist that it is unwise to examine data before determining how to analyze them, as it creates the potential for biased results. I explore the rationale behind this methodological guideline from the standpoint of an error statistical theory of evidence, and I discuss a method of evaluating evidence in some contexts when this predesignation rule has been violated. I illustrate the problem of potential bias, and the method by which it may be addressed, with an example from the search for the top quark. A point in favor of the error statistical theory is its ability, demonstrated here, to explicate such methodological problems and suggest solutions, within the framework of an objective theory of evidence.

Type
Research Article
Information
Philosophy of Science , Volume 69 , Issue 2 , June 2002 , pp. 279 - 299
Copyright
Copyright © The Philosophy of Science Association

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

‡

I am grateful to Douglas Allchin, Peter Lewis, Deborah Mayo, Cassandra Pinnick, and two anonymous referees for helpful comments on this paper, and to the participants in Deborah Mayo's 1999 NEH Summer Seminar at Virginia Tech for comments on an embryonic version. Earlier versions were presented at the University of Chicago and at PSA 2000 in Vancouver, B.C. My 1995 interviews with members of the CDF collaboration were assisted by a grant-in-aid from the American Institute of Physics.

References

Abe, F., Amidei, D., et al. [CDF Collaboration] (1992), “Limit on the Top-Quark Mass from Proton-Antiproton Collisions at √s = 1.8 TeV”, Limit on the Top-Quark Mass from Proton-Antiproton Collisions at √s = 1.8 TeV 45:3921–3948.Google Scholar
Abe, F., Amidei, D., et al. [CDF Collaboration] (1994), “Evidence for Top Quark Production in p Collisions at √s = 1.8 TeV”, Evidence for Top Quark Production in p Collisions at √s = 1.8 TeV 50:2966–3026.Google Scholar
Achinstein, Peter (1995), “Explanation v. Prediction: Which Carries More Weight?”, in Hull, David, Forbes, Micky, and Burian, Richard (eds.), PSA 1994, Vol. 2. East Lansing, MI: Philosophy of Science Association, 156–64.Google Scholar
Achinstein, Peter (2000), “Why Philosophical Theories of Evidence Are (and Ought to Be) Ignored by Scientists”, Why Philosophical Theories of Evidence Are (and Ought to Be) Ignored by Scientists 67(Proceedings): S180–92.Google Scholar
Achinstein, Peter (2001), The Book of Evidence. New York: Oxford University Press.CrossRefGoogle Scholar
Berger, James O., and Wolpert, Robert L. (1984), The Likelihood Principle. Hayward, California: Institute of Mathematical Statistics.Google Scholar
Binkley, Morris (1995), Oral History Interview by Kent Staley. Tape Recording. Fermilab, Batavia, Ill., October 19.Google Scholar
Franklin, Allan (1998), “Selectivity and the Production of Experimental Results”, Selectivity and the Production of Experimental Results 53:399–485.Google Scholar
Giere, Ronald N. (1969), “Bayesian Statistics and Biased Procedures”, Bayesian Statistics and Biased Procedures 20:371–87.Google Scholar
Howson, Colin, and Urbach, Peter (1993), Scientific Reasoning: The Bayesian Approach, 2nd ed. Chicago: Open Court.Google Scholar
Kadane, Joseph B., and Seidenfeld, Teddy (1999), “Randomization in a Bayesian Perspective”, in Kadane, Joseph B., Schervish, Mark J., and Seidenfeld, Teddy (eds.), Rethinking the Foundations of Statistics. New York: Cambridge University Press, 293–313.CrossRefGoogle Scholar
Leplin, Jarrett (1997), A Novel Defense of Scientific Realism. New York: Oxford University Press.Google Scholar
Mayo, Deborah (1993), “The Test of Experiment: C. S. Peirce and E. S. Pearson”, in Moore, E. C. (ed.), Charles S. Peirce and the Philosophy of Science. Tuscaloosa: University of Alabama Press, 161–74.Google Scholar
Mayo, Deborah (1996), Error and the Growth of Experimental Knowledge. Chicago: University of Chicago Press.10.7208/chicago/9780226511993.001.0001CrossRefGoogle Scholar
Mayo, Deborah, and Kruse, Michael (2001), “Principles of Inference and Their Consequences”, in Cornfield, D. and Williamson, J. (eds.) Foundations of Bayesianism. Dordrecht: Kluwer Academic Publishers, 381–403.CrossRefGoogle Scholar
Musgrave, Alan (1974), “Logical versus Historical Theories of Confirmation”, Logical versus Historical Theories of Confirmation 25:1–23.Google Scholar
Peirce, Charles Sanders (1931–1958), Collected Papers of Charles Sanders Peirce, 8 vols. Edited by Hartshorne, Charles and Weiss, Paul. Cambridge, MA: Harvard University Press.Google Scholar
Salmon, Wesley (1984), Scientific Explanation and the Causal Structure of the World. Princeton, NJ: Princeton University Press.Google Scholar
Savage, Leonard J. (ed.) (1962), The Foundations of Statistical Inference: A Discussion. London: Methuen.Google Scholar
Snyder, Laura (1994), “Is Evidence Historical?” in Achinstein, Peter and Snyder, Laura (eds.), Scientific Methods: Conceptual and Historical Problems. Malabar, FL: Krieger Publishing Company, 95–117.Google Scholar
Staley, Kent W. (1996), “Novelty, Severity, and History in the Testing of Hypotheses: The Case of the Top Quark.” Philosophy of Science 63 (Proceedings): S248–S255.10.1086/289958CrossRefGoogle Scholar
Staley, Kent W. (2001), “Over the Top: Evidence and the Search for the Top Quark”, Unpublished Manuscript.Google Scholar
Suppes, Patrick (1962), “Models of Data”, in Nagel, Ernest, Suppes, Patrick, and Tarski, Alfred (eds.), Logic, Methodology and Philosophy of Science: Proceedings of the 1960 International Congress. Stanford, CA: Stanford University Press, 252–261.Google Scholar
Tipton, Paul (1995), Oral History Interview by Kent Staley. Tape Recording. Fermilab, Batavia, IL, October 19.Google Scholar
Worrall, John (1985), “Scientific Discovery and Theory-Confirmation”, in Pitt, Joseph (ed.), Change and Progress in Modern Science. Dordrecht: Reidel, 301–32.Google Scholar
Worrall, John (1989), “Fresnel, Poisson and the White Spot: The Role of Successful Predictions in the Acceptance of Scientific Theories”, in Gooding, David, Pinch, Trevor and Schaffer, Simon (eds.), The Uses of Experiment: Studies in the Natural Sciences. New York: Cambridge University Press, 135–57.Google Scholar
Zahar, Elie (1973), “Why Did Einstein's Programme Supercede Lorentz's?”, Why Did Einstein's Programme Supercede Lorentz's? 24:96–123, 223–62.Google Scholar