Skip to main content Accessibility help

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings.

Close cookie message
×
Hostname: page-component-6bf8c574d5-j5c6p Total loading time: 0 Render date: 2025-03-09T15:44:27.326Z Has data issue: false hasContentIssue false

9 - Aggregating Probability Distributions

Published online by Cambridge University Press:  05 June 2012

By
Robert T. Clemen  and
Robert L. Winkler
Edited by
Ward Edwards ,
Ralph F. Miles Jr.  and
Detlof von Winterfeldt
Robert T. Clemen
Affiliation:
Fuqua School of Business, Duke University
Robert L. Winkler
Affiliation:
Fuqua School of Business, Duke University
Ralph F. Miles Jr.
Affiliation:
California Institute of Technology
Detlof von Winterfeldt
Affiliation:
University of Southern California
Get access

Summary

ABSTRACT. This chapter is concerned with the aggregation of probability distributions in decision and risk analysis. Experts often provide valuable information regarding important uncertainties in decision and risk analyses because of the limited availability of hard data to use in those analyses. Multiple experts are often consulted in order to obtain as much information as possible, leading to the problem of how to combine or aggregate their information. Information may also be obtained from other sources such as forecasting techniques or scientific models. Because uncertainties are typically represented in terms of probability distributions, we consider expert and other information in terms of probability distributions. We discuss a variety of models that lead to specific combination methods. The output of these methods is a combined probability distribution, which can be viewed as representing a summary of the current state of information regarding the uncertainty of interest. After presenting the models and methods, we discuss empirical evidence on the performance of the methods. In the conclusion, we highlight important conceptual and practical issues to be considered when designing a combination process for use in practice.

Introduction

Expert judgments can provide useful information for forecasting, making decisions, and assessing risks. Such judgments have been used informally for many years. In recent years, the use of formal methods to combine expert judgments has become increasingly commonplace. Cooke (1991) reviews many of the developments over the years as attempts have been made to use expert judgments in various settings.

Type
Chapter
Information
Advances in Decision Analysis
From Foundations to Applications
 , pp. 154 - 176
Publisher: Cambridge University Press
Print publication year: 2007

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Armstrong, J. S. (2001a). Combining forecasts. In Armstrong, J. S. (Ed.), Principles of forecasting: A handbook for researchers and practitioners. Norwell, MA: Kluwer, pp. 417–439.CrossRefGoogle Scholar
Armstrong, J. S. (Ed.). (2001b). Principles of forecasting: A handbook for researchers and practitioners. Norwell, MA: Kluwer.CrossRefGoogle Scholar
Armstrong, J. S., Denniston, W. B., and Gordon, M. M. (1975). The use of the decomposition principle in making judgments. Organizational Behavior and Human Performance, 14, 257–263.CrossRefGoogle Scholar
Arrow, K. J. (1951). Social choice and individual values. New York: Wiley.Google Scholar
Bacharach, M. (1979). Normal Bayesian dialogues. Journal of the American Statistical Association, 74, 837–846.CrossRefGoogle Scholar
Bates, J. M., and Granger, C. W. J. (1969). The combination of forecasts. Operational Research Quarterly, 20, 451–468.CrossRefGoogle Scholar
Bernreuter, D. L., Savy, J. B., Mensing, R. W., and Chen, J. C. (1989). Seismic hazard characterization of 69 nuclear sites east of the Rocky Mountains. Vol. 1–8, NUREG/CR 5250, UCID-21517.
Bunn, D. W. (1975). A Bayesian approach to the linear combination of forecasts. Operational Research Quarterly, 26, 325–329.CrossRefGoogle Scholar
Bunn, D. W. (1988). Combining forecasts. European Journal of Operational Research, 33, 223–229.CrossRefGoogle Scholar
Bunn, D., and Wright, G. (1991). Interaction of judgmental and statistical forecasting methods: Issues and analysis. Management Science, 37, 501–518.CrossRefGoogle Scholar
Chandrasekharan, R., Moriarty, M. M., and Wright, G. P. (1994). Testing for unreliable estimators and insignificant forecasts in combined forecasts. Journal of Forecasting, 13, 611–624.CrossRefGoogle Scholar
Chhibber, S., and Apostolakis, G. (1993). Some approximations useful to the use of dependent information sources. Reliability Engineering and System Safety, 42, 67–86.CrossRefGoogle Scholar
Clemen, R. T. (1989). Combining forecasts: A review and annotated bibliography. International Journal of Forecasting, 5, 559–583.CrossRefGoogle Scholar
Clemen, R. T., Fischer, G. W., and Winkler, R. L. (2000). Assessing dependence: Some experimental results. Management Science, 46, 1100–1115.CrossRefGoogle Scholar
Clemen, R. T., Jones, S. K., and Winkler, R. L. (1996). Aggregating forecasts: An empirical evaluation of some Bayesian methods. In Berry, D., Chaloner, K., and Geweke, J. (Eds.), Bayesian statistics and econometrics: Essays in honor of Arnold Zellner. New York: Wiley, pp. 3–13.Google Scholar
Clemen, R. T., and Reilly, T. (1999). Correlations and copulas for decision and risk analysis. Management Science, 45, 208–224.CrossRefGoogle Scholar
Clemen, R. T., and Winkler, R. L. (1985). Limits for the precision and value of information from dependent sources. Operations Research, 33, 427–442.CrossRefGoogle Scholar
Clemen, R. T., and Winkler, R. L. (1987). Calibrating and combining precipitation probability forecasts. In Viertl, R. (Ed.), Probability and Bayesian statistics. New York: Plenum, pp. 97–110.CrossRefGoogle Scholar
Clemen, R. T., and Winkler, R. L. (1990). Unanimity and compromise among probability forecasters. Management Science, 36, 767–779.CrossRefGoogle Scholar
Clemen, R. T., and Winkler, R. L. (1993). Aggregating point estimates: A flexible modeling approach. Management Science, 39, 501–515.CrossRefGoogle Scholar
Clemen, R. T., and Winkler, R. L. (1999). Combining probability distributions from experts in risk analysis. Risk Analysis, 19, 187–203.CrossRefGoogle Scholar
Cooke, R. M. (1991). Experts in uncertainty: Opinion and subjective probability in science. New York: Oxford University Press.Google Scholar
Dall'Aglio, G., Kotz, S., and Salinetti, G. (1991). Advances in probability distributions with given marginals: Beyond the copulas. Dordrecht, The Netherlands: Kluwer.CrossRefGoogle Scholar
Davis, J. (1992). Some compelling intuitions about group consensus decisions, theoretical and empirical research, and interpersonal aggregation phenomena: Selected examples, 1950–1990. Organizational Behavior and Human Decision Processes, 52, 3–38.CrossRefGoogle Scholar
Dawes, R. M., Faust, D., and Meehl, P. A. (1989). Clinical versus actuarial judgment. Science, 243, 1668–1673.CrossRefGoogle ScholarPubMed
de Finetti, B. (1937). La prévision: Ses lois logiques, ses sources subjectives. Annales de l'Institut Henri Poincaré, 7, 1–68. Translated in 1980 by H. E. Kyburg, Jr., Foresight. Its logical laws, its subjective sources. In Kyburg, H. E. Jr. and Smokler, H. E. (Eds.), Studies in subjective probability (2nd ed.). Huntington, New York: Robert E. Krieger, pp. 53–118.Google Scholar
Dillon, R., John, R., and Winterfeldt, D. (2002). Assessment of cost uncertainties for large technology projects: A methodology and an application. Interfaces, 32 (Jul/Aug), 52–66.CrossRefGoogle Scholar
EPRI (1986). Seismic hazard methodology for the central and eastern United States. Vol. 1: Methodology. NP-4/26. Palo Alto, CA: Electric Power Research Institute.
Flores, B. E., and White, E. M. (1989). Subjective vs. objective combining of forecasts: An experiment. Journal of Forecasting, 8, 331–341.CrossRefGoogle Scholar
French, S. (1981). Consensus of opinion. European Journal of Operational Research, 7, 332–340.CrossRefGoogle Scholar
French, S. (1985). Group consensus probability distributions: A critical survey. In Bernardo, J. M., DeGroot, M. H., Lindley, D. V., and Smith, A. F. M. (Eds.), Bayesian statistics 2. Amsterdam: North-Holland, pp. 183–197.Google Scholar
French, S., and Ríos Insua, D. (2000). Statistical decision theory. London: Arnold.Google Scholar
Garthwaite, P. H., Kadane, J. B., and O'Hagan, A. (2005). Statistical methods for eliciting prior distributions. Journal of the American Statistical Association, 100, 680–700.CrossRefGoogle Scholar
Gelfand, A. E., Mallick, B. K., and Dey, D. K. (1995). Modeling expert opinion rising as a partial probabilistic specification. Journal of the American Statistical Association, 90, 598–604.CrossRefGoogle Scholar
Genest, C. (1984). Pooling operators with the marginalization property. Canadian Journal of Statistics, 12, 153–163.CrossRefGoogle Scholar
Genest, C., and McConway, K. J. (1990). Allocating the weights in the linear opinion pool. Journal of Forecasting, 9, 53–73.CrossRefGoogle Scholar
Genest, C., and Schervish, M. J. (1985). Modeling expert judgments for Bayesian updating. Annals of Statistics, 13, 1198–1212.CrossRefGoogle Scholar
Genest, C., and Zidek, J. V. (1986). Combining probability distributions: A critique and annotated bibliography. Statistical Science, 1, 114–148.CrossRefGoogle Scholar
Gigone, D., and Hastie, R. (1997). Proper analysis of the accuracy of group judgments. Psychological Bulletin, 121, 149–167.CrossRefGoogle Scholar
Gokhale, D. V., and Press, S. J. (1982). Assessment of a prior distribution for the correlation coefficient in a bivariate normal distribution. Journal of the Royal Statistical Society, Series A, 145, 237–249.CrossRefGoogle Scholar
Goodman, B. (1972). Action selection and likelihood estimation by individuals and groups. Organizational Behavior and Human Performance, 7, 121–141.CrossRefGoogle Scholar
Hammitt, J. K., and Shlyakhter, A. I. (1999). The expected value of information and the probability of surprise. Risk Analysis, 19, 135–152.CrossRefGoogle Scholar
Hastie, R. (1986). Experimental evidence on group accuracy. In Grofman, B. and Owen, G. (Eds.), Information pooling and group decision making. Greenwich, CT: JAI Press, pp. 129–157.Google Scholar
Hill, G. W. (1982). Group vs. individual performance: Are N+1 heads better than one?Psychological Bulletin, 91, 517–539.CrossRefGoogle Scholar
Hogarth, R. M. (1987). Judgment and choice. (2nd ed.). Chichester, England: Wiley.Google Scholar
Hora, S. C. (1992). Acquisition of expert judgment: Examples from risk assessment. Journal of Energy Engineering, 118, 136–148.CrossRefGoogle Scholar
Hora, S. C. (2004). Probability judgments for continuous quantitites: Linear combinations and calibration. Management Science, 50, 597–604.CrossRefGoogle Scholar
Hora, S. C., Dodd, N. G., and Hora, J. A. (1993). The use of decomposition in probability assessments on continuous variables. Journal of Behavioral Decision Making, 6, 133–147.CrossRefGoogle Scholar
Hora, S. C., and Iman, R. L. (1989). Expert opinion in risk analysis: The NUREG-1150 methodology. Nuclear Science and Engineering, 102, 323–331.CrossRefGoogle Scholar
Jouini, M. N., and Clemen, R. T. (1996). Copula models for aggregating expert opinions. Operations Research, 44, 444–457.CrossRefGoogle Scholar
Kahneman, D., and Tversky, A. (1972). Subjective probability: A judgment of representativeness. Cognitive Psychology, 3, 430–454.CrossRefGoogle Scholar
Keeney, R. L., and Winterfeldt, D. (1989). On the uses of expert judgment on complex technical problems. IEEE Transactions on Engineering Management, 36, 83–86.CrossRefGoogle Scholar
Keeney, R. L., and Winterfeldt, D. (1991). Eliciting probabilities from experts in complex technical problems. IEEE Transactions on Engineering Management, 38, 191–201.CrossRefGoogle Scholar
Lacke, C. J., and Clemen, R. T. (2001). Analysis of colorectal cancer screening regimens. Health Care Management Science, 4, 257–267.Google Scholar
Larrick, R. P., and Soll, J. B. (2006). Intuitions about combining opinions: Misappreciation of the averaging principle. Management Science, 52, 111–127.CrossRefGoogle Scholar
Lawrence, M. J., Edmundson, R. H., and O'Connor, M. J. (1986). The accuracy of combining judgmental and statistical forecasts. Management Science, 32, 1521–1532.CrossRefGoogle Scholar
Lichtendahl, K. C. (2005). Bayesian nonparametric combination of multivariate expert forecasts. Ph.D. dissertation, Duke University.Google Scholar
Lindley, D. V. (1983). Reconciliation of probability distributions. Operations Research, 31, 866–880.CrossRefGoogle Scholar
Lindley, D. V. (1985). Reconciliation of discrete probability distributions. In Bernardo, J. M., DeGroot, M. H., Lindley, D. V., and Smith, A. F. M. (Eds.), Bayesian statistics 2. Amsterdam: North-Holland, pp. 375–390.Google Scholar
Lipscomb, J., Parmigiani, G., and Hasselblad, V. (1998). Combining expert judgment by hierarchical modeling: An application to physician staffing. Management Science, 44, 149–161.CrossRefGoogle Scholar
MacGregor, D., Lichtenstein, S., and Slovic, P. (1988). Structuring knowledge retrieval: An analysis of decomposing quantitative judgments. Organizational Behavior and Human Decision Processes, 42, 303–323.CrossRefGoogle Scholar
Merkhofer, M. W. (1987). Quantifying judgmental uncertainty: Methodology, experience, and insights. IEEE Transactions on Systems, Man, and Cybernetics, 17, 741–752.CrossRefGoogle Scholar
Merrick, J. R. W., Dorp, J. R., and Singh, A. (2005). Analysis of correlated expert judgments from extended pairwise comparisons. Decision Analysis, 2, 17–29.CrossRefGoogle Scholar
Morgan, M. G., and Henrion, M. (1990). Uncertainty: A guide to dealing with uncertainty in quantitative risk and policy analysis. Cambridge, MA: Cambridge University Press.CrossRefGoogle Scholar
Morgan, M. G., and Keith, D. W. (1995). Subjective judgments by climate experts. Envirionmental Science and Technology, 29, 468–476.Google ScholarPubMed
Morris, P. A. (1974). Decision analysis expert use. Management Science, 20, 1233–1241.CrossRefGoogle Scholar
Morris, P. A. (1977). Combining expert judgments: A Bayesian approach. Management Science, 23, 679–693.CrossRefGoogle Scholar
Morris, P. A. (1983). An axiomatic approach to expert resolution. Management Science, 29, 24–32.CrossRefGoogle Scholar
Mosleh, A., Bier, V. M., and Apostolakis, G. (1987). A critique of current practice for the use of expert opinions in probabilistic risk assessment. Reliability Engineering and System Safety, 20, 63–85.CrossRefGoogle Scholar
Newbold, P., and Granger, C. W. J. (1974). Experience with forecasting univariate time series and the combination of forecasts. Journal of the Royal Statistical Society, Series A, 137, 131–149.CrossRefGoogle Scholar
Otway, H., and Winterfeldt, D. (1992). Expert judgment in risk analysis and management: Process, context, and pitfalls. Risk Analysis, 12, 83–93.CrossRefGoogle ScholarPubMed
Ravinder, H. V., Kleinmuntz, D. N., and Dyer, J. S. (1988). The reliability of subjective probabilities obtained through decomposition. Management Science, 34, 186–199.CrossRefGoogle Scholar
Reagan-Cirincione, P. (1994). Improving the accuracy of group judgment; A process intervention combining group facilitation, social judgment analysis, and information technology. Organizational Behavior and Human Decision Processes, 58, 246–270.CrossRefGoogle Scholar
Rohrbaugh, J. (1979). Improving the quality of group judgment: Social judgment analysis and the Delphi technique. Organizational Behavior and Human Performance, 24, 73–92.CrossRefGoogle Scholar
Savage, L. J. (1954). The foundations of statistics. New York: Wiley.Google Scholar
Schmittlein, D. C., Kim, J., and Morrison, D. G. (1990). Combining forecasts: Operational adjustments to theoretically optimal rules. Management Science, 36, 1044–1056.CrossRefGoogle Scholar
Seaver, D. A. (1978). Assessing probability with multiple individuals: Group interaction versus mathematical aggregation. Report No. 78–3, Social Science Research Institute, University of Southern California.Google Scholar
Shlyakhter, A. I. (1994). Improved framework for uncertainty analysis: Accounting for unsuspected errors. Risk Analysis, 14, 441–447.CrossRefGoogle Scholar
Shlyakhter, A. I., Kammen, D. M., Brodio, C. L., and Wilson, R. (1994). Quantifying the credibility of energy projections from trends in past data: The U.S. energy sector. Energy Policy, 22, 119–130.CrossRefGoogle Scholar
Sniezek, J. A. (1989). An examination of group process in judgmental forecasting. International Journal of Forecasting, 5, 171–178.CrossRefGoogle Scholar
Spetzler, C. S., and Holstein, Staël C.-A. S. (1975). Probability encoding in decision analysis. Management Science, 22, 340–358.CrossRefGoogle Scholar
Holstein, Staël C.-A. S. (1972). Probabilistic forecasting: An experiment related to the stock market. Organizational Behavior and Human Performance, 8, 139–158.CrossRefGoogle Scholar
Stone, M. (1961). The opinion pool. Annals of Mathematical Statistics, 32, 1339–1342.CrossRefGoogle Scholar
West, M. (1988). Modelling expert opinion. In Bernardo, J. M., DeGroot, M. H., Lindley, D. V., and Smith, A. F. M. (Eds.), Bayesian statistics, 3, Amsterdam: New Holland, pp. 493–508.Google Scholar
West, M. (1992). Modelling agent forecast distributions. Journal of the Royal Statistical Society, Series B, 54, 553–567.Google Scholar
West, M., and Crosse, J. (1992). Modelling probabilistic agent opinion. Journal of the Royal Statistical Society, Series B, 54, 285–299.Google Scholar
Winkler, R. L. (1968). The consensus of subjective probability distributions. Management Science, 15, 361–375.CrossRefGoogle Scholar
Winkler, R. L. (1981). Combining probability distributions from dependent information sources. Management Science, 27, 479–488.CrossRefGoogle Scholar
Winkler, R. L. (1986). Expert resolution. Management Science, 32, 298–303.CrossRefGoogle Scholar
Winkler, R. L., and Clemen, R. T. (1992). Sensitivity of weights in combining forecasts. Operations Research, 40, 609–614.CrossRefGoogle Scholar
Winkler, R. L., and Clemen, R. T. (2004). Multiple experts vs. multiple methods: Combining correlation assessments. Decision Analysis, 1, 167–176.CrossRefGoogle Scholar
Winkler, R. L., and Makridakis, S. (1983). The combination of forecasts. Journal of the Royal Statistical Society, Series A, 146, 150–157.CrossRefGoogle Scholar
Winkler, R. L., and Poses, R. M. (1993). Evaluating and combining physicians' probabilities of survival in an intensive care unit. Management Science, 39, 1526–1543.CrossRefGoogle Scholar
Winkler, R. L., Wallsten, T. S., Whitfield, R. G., Richmond, H. M., Hayes, S. R., and Rosenbaum, A. S. (1995). An assessment of the risk of chronic lung injury attributable to long-term ozone exposure. Operations Research, 43, 19–28.CrossRefGoogle Scholar
Wright, G., and Ayton, P. (Eds). (1987). Judgmental forecasting. Chichester, UK: Wiley.Google Scholar
Wright, G., Saunders, C., and Ayton, P. (1988). The consistency, coherence, and calibration of holistic, decomposed, and recomposed judgmental probability forecasts. Journal of Forecasting, 7, 185–199.CrossRefGoogle Scholar
Wu, J. S., Apostolakis, G., and Okrent, D. (1990). Uncertainties in system analysis: Probabilistic versus nonprobabilistic theories. Reliability Engineering and System Safety, 30, 163–181.CrossRefGoogle Scholar

Save book to Kindle

To save this book to your Kindle, first ensure [email protected] is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Available formats
×

Save book to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox.

Available formats
×

Save book to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive.

Available formats
×