Confidence in Aggregation of Opinions from Multiple Sources

doi:10.1017/CBO9780511614576.014

14 - Confidence in Aggregation of Opinions from Multiple Sources

Published online by Cambridge University Press: 02 February 2010

David V. Budescu

Edited by

Klaus Fiedler and

Peter Juslin

Show author details

David V. Budescu: Affiliation:
University of Illinois
Klaus Fiedler: Affiliation:
Ruprecht-Karls-Universität Heidelberg, Germany
Peter Juslin: Affiliation:
Umeå Universitet, Sweden

Book contents

Get access

Summary

To the memory of my friend and colleague, Janet A. Sniezek

(1951–2003)

This chapter summarizes research related to several interrelated questions regarding the process by which single decision makers (DMs) aggregate probabilistic information regarding a certain event from several, possibly asymmetric, advisors who rely on multiple and, possibly overlapping and correlated, sources of information. In particular I seek to understand and characterize (a) the nature of the aggregation rules used by DMs and (b) the factors that affect the DMs' confidence in the final aggregate.

The chapter starts with a short literature overview whose main goal is to set the stage by placing this work within the broad area of information aggregation. Next I present a descriptive model of confidence in information integration that is based on two principles: (a) People combine multiple sources of information by applying simple averaging rules; and (b) the DM's level of confidence in the aggregate is a monotonically decreasing function of its perceived variance. Some of the model's predictions regarding the structural and natural factors that affect the DM's confidence are discussed and tested with empirical data from four experiments (Budescu & Rantilla, 2000; Budescu et al., 2003). The results document the relation between the DM's confidence and the amount of information underlying the forecasts (number of advisors and cues), the advisors' accuracy, and the distribution of cues over judges with special attention to the level of interjudge overlap in information.

Type: Chapter
Information: Information Sampling and Adaptive Cognition , pp. 327 - 352

DOI: https://doi.org/10.1017/CBO9780511614576.014 [Opens in a new window]

Publisher: Cambridge University Press

Print publication year: 2005

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Book purchase

Temporarily unavailable

References

Anderson, N. H. (1981). Information integration theory. Hillsdale, NJ: Lawrence ErlbaumGoogle Scholar

Ariely, D., Au, W. T., Bender, R. H., Budescu, D. V., Dietz, C. B., Gu, H.Wallsten, T. S., & Zauberman, G. (2000). The effects of averaging subjective probability estimates between and within judges. Journal of Experimental Psychology: Applied, 6, 130–147Google Scholar PubMed

Armstrong, J. S. (2001). Combining forecasts. In Armstrong, J. S. (Ed.), Principles of forecasting: A handbook for researchers and practitioners. Norwell, MA: Kluwer Academic PublishersCrossRef Google Scholar

Arrow, K. (1963). Social choice and individual values (2nd edition). New Haven, CT: Yale University PressGoogle Scholar

Ashton, R. H. (1986). Combining the judgments of advisors: How many and which ones?Organizational Behavior and Human Decision Processes, 38, 405–414CrossRef Google Scholar

Black, D. (1958). The theory of committees and elections. Cambridge, UK: Cambridge University PressGoogle Scholar

Bordley, R. F. (1982). A multiplicative formula for aggregating probability assessments. Management Science, 28, 1137–1148CrossRef Google Scholar

Budescu, D. V. (2001). Aggregation of probabilistic forecasts and opinions. Presidential address at the annual meeting of the Society for Judgment and Decision Making (SJDM), Orlando, FL

Budescu, D. V., Erev, I., & Wallsten, T. S. (1997). On the importance of random error in the study of probability judgment, Part I: New theoretical developments. Journal of Behavioral Decision Making, 10, 157–1723.0.CO;2-#>CrossRef Google Scholar

Budescu, D. V., & Rantilla, A. K. (2000) Confidence in aggregation of expert opinions. Acta Psychologica, 104, 371–398CrossRef Google Scholar PubMed

Budescu, D. V., Rantilla, A. K., Yu, H., & Karelitz, T. M. (2003). The effects of asymmetry among advisors on the aggregation of their opinions. Organizational Behavior and Human Decision Processes, 90, 178–194CrossRef Google Scholar

Chapman, L. J., & Chapman, J. P. (1967). Genesis of popular but erroneous psychodiagnostic observations. Journal of Abnormal Psychology, 72, 193–204CrossRef Google Scholar PubMed

Chapman, L. J., & Chapman, J. P. (1969). Illusory correlation as an obstacle to the use of valid psychodiagnostic signs. Journal of Abnormal Psychology, 74, 271–280CrossRef Google Scholar PubMed

Clemen, R. T. (1989). Combining forecasts: A review and annotated bibliography. International Journal of Forecasting, 5, 559–583CrossRef Google Scholar

Clemen, R. T., & Winkler, R. L. (1993). Aggregating point estimates: A flexible modeling approach. Management Science, 39, 501–515CrossRef Google Scholar

Clemen, R. T., & Winkler, R. L. (1985). Limits for the precision and value of information from dependent sources. Operation Research, 33, 427–442CrossRef Google Scholar

Clemen, R. T., & Winkler, R. L. (1986). Combining economic forecasts. Journal of Business and Economic Statistics, 4, 39–46Google Scholar

Davis, J. H. (1973). Group decision and social interaction: A theory of social decision schemes. Psychological Review, 80, 97–125CrossRef Google Scholar

Davis, J. H. (1996). Group decision making and quantitative judgments: A consensus model. In Witte, E. H. & Davis, J. H. (Eds.), Understanding group behavior (Vol 1): Small group processes and interpersonal relations. Hillsdale, NJ: Lawrence ErlbaumGoogle Scholar

Einhorn, H. J., & Hogarth, R. M. (1978). Confidence in judgment: Persistence of the illusion of validity. Psychological Review, 85, 395–416CrossRef Google Scholar

Erev, I., Wallsten, T. S., & Budescu, D. V. (1994). Simultaneous over- and underconfidence: The role of error in judgment processes. Psychological Review, 101, 519–527CrossRef Google Scholar

Ferrel, W. R. (1985). Combining individual judgments. In Wright, G. (Ed.), Behavioral decision making (pp. 111–145). New York: PlenumCrossRef Google Scholar

Fiedler, K. (1996). Explaining and simulating judgment biases as an aggregation phenomenon in probabilistic multiple-cue environments. Psychological Review, 103, 193–214CrossRef Google Scholar

Fischer, I., & Harvey, N. (1999). Combining forecasts: What information do judges need to outperform the simple average?International Journal of Forecasting, 15, 227–246CrossRef Google Scholar

Flores, B. E., & White, E. M. (1988). A framework for the combination of forecasts. Academy of Marketing Science, 16, 95–103CrossRef Google Scholar

Gigerenzer, G., & Goldstein, D. G. (1999). Betting on one good reason: The take the best heuristic. In Gigerenzer, G., Todd, P. M., & the ABC Research Group (Eds.), Simple heuristics that make us smart (pp. 75–96). New York: Oxford University PressGoogle Scholar

Gigerenzer, G., Hoffrage, U., & Kleinbölting, H. (1991). Probabilistic mental models: A Brunswikian theory of confidence. Psychological Review, 98, 506–528CrossRef Google Scholar

Harvey, N., & Harries, C. (2003). Effects of judges' forecasting on their later combination of forecasts for the same outcomes. International Journal of Forecasting, in press

Hastie, R. (1993). Inside the juror: The psychology of jury decision-making. Cambridge, UK: Cambridge University PressCrossRef Google Scholar

Hogarth, R. M. (1977). Methods for aggregating opinions. In Jungermann, H. & Zeeuw, G. (Eds.), Decision making and change in human affairs (pp. 231–255). Boston: RiedelCrossRef Google Scholar

Hogarth, R. M. (1978). A note on aggregating opinions. Organizational Behavior and Human Performance, 21, 40–46CrossRef Google Scholar

Johnson, T. R., Budescu, D. V., & Wallsten, T. S. (2001). Averaging probability judgments: Monte Carlo analyses of asymptotic diagnostic value. Journal of Behavioral Decision Making, 14, 123–140CrossRef Google Scholar

Juslin, P., Olsson, H., & Bjorkman, M. (1997). Brunswikian and Thurstonian origins of bias in probability assessment: On the interpretation of stochastic components of judgment. Journal of Behavioral Decision Making, 10, 189–2103.0.CO;2-4>CrossRef Google Scholar

Kahneman, D., & Tversky, A. (1973). On the psychology of prediction. Psychological Review, 80, 237–251CrossRef Google Scholar

Kuhn, K. M., & Sniezek, J. A. (1996). Confidence and uncertainty in judgmental forecasting: Differential effects of scenario presentation. Journal of Behavioral Decision Making, 9, 231–2473.0.CO;2-L>CrossRef Google Scholar

Larrick, R. B., & Soll, J. B. (2003). Lay intuitions about combining quantitative judgments. Paper presented at the workshop on Information Aggregation in Decision Making, Silver Spring, MD

Laughlin, P. R., & Ellis, A. L. (1986). Demonstrability and social combination processes on mathematical intellective tasks, Journal of Experimental Social Psychology, 22, 177–189CrossRef Google Scholar

Maines, L. (1996). An experimental examination of subjective forecast combination. International Journal of Forecasting, 12, 223–233CrossRef Google Scholar

Mojzisch, A., & Schulz-Hardt, S. (2004). Information sampling in group decision making: Sampling biases and their consequences. This volume

Morris, P. A. (1983). An axiomatic approach to expert resolution. Management Science, 29, 24–32CrossRef Google Scholar

Morris, P. A. (1986). Comment on Genest and Zideck's “Combining probability distributions: A critique and annotated bibliography.”Statistical Science, 1, 141–144CrossRef Google Scholar

Payne, J. W., Bettman, J. R., & Johnson, E. R. (1993). The adaptive decision maker. New York: Cambridge University PressCrossRef Google Scholar

Price, P. C., & Stone, E. R. (2004). Intuitive evaluation of likelihood judgment producers: Evidence for a confidence heuristic. Journal of Behavioral Decision Making, 17, 39–57CrossRef Google Scholar

Rantilla, A. K., & Budescu, D. V. (1999). Aggregation of advisor opinions. In Proceedings of the 32nd Annual Hawai'i International Conference on Systems Sciences

Salvadori, L., Swol, L. M., & Sniezek, J. A. (2001). Information sampling and confidence within groups and judge advisor systems. Communication Research, 28, 737–771CrossRef Google Scholar

Sniezek, J. A. (1986). The role of variable labels in cue probability learning tasks. Organizational Behavior and Human Decision Processes, 38, 141–161CrossRef Google Scholar

Sniezek, J. A. (1992). Groups under uncertainty: An examination of confidence in group decision making. Organizational Behavior and Human Decision Processes, 52, 124–155CrossRef Google Scholar

Sniezek, J. A., & Buckley, T. (1995). Cueing and cognitive conflict in judge-advisor decision making. Organizational Behavior and Human Decision Processes, 62, 159–174CrossRef Google Scholar

Soll, J. B. (1999). Intuitive theories of information: Beliefs about the value of redundancy. Cognitive Psychology, 38, 317–346CrossRef Google Scholar PubMed

Soll, J. B., Larrick, R. (2000). The 80/20 rule and the revision of judgment in light another's opinion: Why fo we believe ourseleves so much? Paper presented at the annual meeting of the Society for Judgment and Decision Making (SJDM), Los Angeles

Trafimow, D., & Sniezek, J. A. (1994). Perceived expertise and its effect on confidence. Organizational Behavior and Human Decision Processes, 57, 290–302CrossRef Google Scholar

van Swol, L. M., & Sniezek, J. A. (2002). Trust me, I'm an expert: Trust and confidence and acceptance of expert advice. Paper presented at the 8th Conference on Behavioral Decision Research in Management (BDRM), Chicago

Wallsten, T. S., Budescu, D. V., Erev, I., & Diederich, A. (1997). Evaluating and combining subjective probability estimates. Journal of Behavioral Decision Making, 10, 243–2683.0.CO;2-M>CrossRef Google Scholar

Wallsten, T. S., & Diederich, A. (2001). Understanding pooled subjective probability estimates. Mathematical Social Sciences, 18, 1–18CrossRef Google Scholar

Walz, D. T., & Walz, D. D. (1989). Combining forecast: Multiple regression versus a Bayesian approach. Decision Sciences, 20, 77–89CrossRef Google Scholar

Winkler, R. L. (1971). Probabilistic prediction: Some experimental results. Journal of the American Statistical Association, 66, 675–685CrossRef Google Scholar

Winkler, R. L., & Makridakis, S. (1983). The combination of forecasts. Journal of the Royal Statistical Society: Series A, 146, 150–157CrossRef Google Scholar

Winkler, R. L., & Poses, R. M. (1993). Evaluating and combining physician's probabilities of survival in an intensive care unit. Management Science, 39, 1526–1543CrossRef Google Scholar

Wittenbaum, G., & Stasser, G. (1996). Management of information in small groups. In Nye, J. L. & Brower, A. M. (Eds.), What's social about social cognition? Research on socially shared cognition in small groups. (pp. 3–28). Thousand Oaks, CA: SageCrossRef Google Scholar

Yaniv, I. (1997). Weighting and trimming: Heuristics for aggregating judgments under uncertainty. Organizational Behavior and Human Decision Processes, 69, 237–249CrossRef Google Scholar

Yaniv, I. (2004). Receiving other people's advice: Influence and benefit. Organizational Behavior and Human Decision Processes, 93, 1–13CrossRef Google Scholar

Yaniv, I., & Kleinberger, E. (2000). Advice taking in decision-making: Egocentric discounting and reputation formation. Organizational Behavior and Human Decision Processes, 83, 260–281CrossRef Google Scholar PubMed