Book contents
- Frontmatter
- Contents
- List of Contributors
- PART I INTRODUCTION
- PART II THE PSYCHOLOGICAL LAW OF LARGE NUMBERS
- PART III BIASED AND UNBIASED JUDGMENTS FROM BIASED SAMPLES
- PART IV WHAT INFORMATION CONTENTS ARE SAMPLED?
- 11 What's in a Sample? A Manual for Building Cognitive Theories
- 12 Assessing Evidential Support in Uncertain Environments
- 13 Information Sampling in Group Decision Making: Sampling Biases and Their Consequences
- 14 Confidence in Aggregation of Opinions from Multiple Sources
- 15 Self as Sample
- PART V VICISSITUDES OF SAMPLING IN THE RESEARCHER'S MIND AND METHOD
- Index
- References
14 - Confidence in Aggregation of Opinions from Multiple Sources
Published online by Cambridge University Press: 02 February 2010
Book contents
- Frontmatter
- Contents
- List of Contributors
- PART I INTRODUCTION
- PART II THE PSYCHOLOGICAL LAW OF LARGE NUMBERS
- PART III BIASED AND UNBIASED JUDGMENTS FROM BIASED SAMPLES
- PART IV WHAT INFORMATION CONTENTS ARE SAMPLED?
- 11 What's in a Sample? A Manual for Building Cognitive Theories
- 12 Assessing Evidential Support in Uncertain Environments
- 13 Information Sampling in Group Decision Making: Sampling Biases and Their Consequences
- 14 Confidence in Aggregation of Opinions from Multiple Sources
- 15 Self as Sample
- PART V VICISSITUDES OF SAMPLING IN THE RESEARCHER'S MIND AND METHOD
- Index
- References
Summary
To the memory of my friend and colleague, Janet A. Sniezek
(1951–2003)This chapter summarizes research related to several interrelated questions regarding the process by which single decision makers (DMs) aggregate probabilistic information regarding a certain event from several, possibly asymmetric, advisors who rely on multiple and, possibly overlapping and correlated, sources of information. In particular I seek to understand and characterize (a) the nature of the aggregation rules used by DMs and (b) the factors that affect the DMs' confidence in the final aggregate.
The chapter starts with a short literature overview whose main goal is to set the stage by placing this work within the broad area of information aggregation. Next I present a descriptive model of confidence in information integration that is based on two principles: (a) People combine multiple sources of information by applying simple averaging rules; and (b) the DM's level of confidence in the aggregate is a monotonically decreasing function of its perceived variance. Some of the model's predictions regarding the structural and natural factors that affect the DM's confidence are discussed and tested with empirical data from four experiments (Budescu & Rantilla, 2000; Budescu et al., 2003). The results document the relation between the DM's confidence and the amount of information underlying the forecasts (number of advisors and cues), the advisors' accuracy, and the distribution of cues over judges with special attention to the level of interjudge overlap in information.
- Type
- Chapter
- Information
- Information Sampling and Adaptive Cognition , pp. 327 - 352Publisher: Cambridge University PressPrint publication year: 2005
References
, , , , , , & (2000). The effects of averaging subjective probability estimates between and within judges. Journal of Experimental Psychology: Applied, 6, 130–147Google ScholarPubMed
Armstrong, J. S. (2001). Combining forecasts. In (Ed.), Principles of forecasting: A handbook for researchers and practitioners. Norwell, MA: Kluwer Academic PublishersCrossRefGoogle Scholar
(1963). Social choice and individual values (2nd edition). New Haven, CT: Yale University PressGoogle Scholar
(1986). Combining the judgments of advisors: How many and which ones?Organizational Behavior and Human Decision Processes, 38, 405–414CrossRefGoogle Scholar
(1958). The theory of committees and elections. Cambridge, UK: Cambridge University PressGoogle Scholar
(1982). A multiplicative formula for aggregating probability assessments. Management Science, 28, 1137–1148CrossRefGoogle Scholar
Budescu, D. V. (2001). Aggregation of probabilistic forecasts and opinions. Presidential address at the annual meeting of the Society for Judgment and Decision Making (SJDM), Orlando, FL
, , & (1997). On the importance of random error in the study of probability judgment, Part I: New theoretical developments. Journal of Behavioral Decision Making, 10, 157–1723.0.CO;2-#>CrossRefGoogle Scholar
, & (2000) Confidence in aggregation of expert opinions. Acta Psychologica, 104, 371–398CrossRefGoogle ScholarPubMed
, , , & (2003). The effects of asymmetry among advisors on the aggregation of their opinions. Organizational Behavior and Human Decision Processes, 90, 178–194CrossRefGoogle Scholar
, & (1967). Genesis of popular but erroneous psychodiagnostic observations. Journal of Abnormal Psychology, 72, 193–204CrossRefGoogle ScholarPubMed
, & (1969). Illusory correlation as an obstacle to the use of valid psychodiagnostic signs. Journal of Abnormal Psychology, 74, 271–280CrossRefGoogle ScholarPubMed
(1989). Combining forecasts: A review and annotated bibliography. International Journal of Forecasting, 5, 559–583CrossRefGoogle Scholar
, & (1993). Aggregating point estimates: A flexible modeling approach. Management Science, 39, 501–515CrossRefGoogle Scholar
, & (1985). Limits for the precision and value of information from dependent sources. Operation Research, 33, 427–442CrossRefGoogle Scholar
, & (1986). Combining economic forecasts. Journal of Business and Economic Statistics, 4, 39–46Google Scholar
(1973). Group decision and social interaction: A theory of social decision schemes. Psychological Review, 80, 97–125CrossRefGoogle Scholar
Davis, J. H. (1996). Group decision making and quantitative judgments: A consensus model. In & (Eds.), Understanding group behavior (Vol 1): Small group processes and interpersonal relations. Hillsdale, NJ: Lawrence ErlbaumGoogle Scholar
, & (1978). Confidence in judgment: Persistence of the illusion of validity. Psychological Review, 85, 395–416CrossRefGoogle Scholar
, , & (1994). Simultaneous over- and underconfidence: The role of error in judgment processes. Psychological Review, 101, 519–527CrossRefGoogle Scholar
Ferrel, W. R. (1985). Combining individual judgments. In (Ed.), Behavioral decision making (pp. 111–145). New York: PlenumCrossRefGoogle Scholar
(1996). Explaining and simulating judgment biases as an aggregation phenomenon in probabilistic multiple-cue environments. Psychological Review, 103, 193–214CrossRefGoogle Scholar
, & (1999). Combining forecasts: What information do judges need to outperform the simple average?International Journal of Forecasting, 15, 227–246CrossRefGoogle Scholar
, & (1988). A framework for the combination of forecasts. Academy of Marketing Science, 16, 95–103CrossRefGoogle Scholar
Gigerenzer, G., & Goldstein, D. G. (1999). Betting on one good reason: The take the best heuristic. In , , & the ABC Research Group (Eds.), Simple heuristics that make us smart (pp. 75–96). New York: Oxford University PressGoogle Scholar
, , & (1991). Probabilistic mental models: A Brunswikian theory of confidence. Psychological Review, 98, 506–528CrossRefGoogle Scholar
Harvey, N., & Harries, C. (2003). Effects of judges' forecasting on their later combination of forecasts for the same outcomes. International Journal of Forecasting, in press
(1993). Inside the juror: The psychology of jury decision-making. Cambridge, UK: Cambridge University PressCrossRefGoogle Scholar
Hogarth, R. M. (1977). Methods for aggregating opinions. In & (Eds.), Decision making and change in human affairs (pp. 231–255). Boston: RiedelCrossRefGoogle Scholar
(1978). A note on aggregating opinions. Organizational Behavior and Human Performance, 21, 40–46CrossRefGoogle Scholar
, , & (2001). Averaging probability judgments: Monte Carlo analyses of asymptotic diagnostic value. Journal of Behavioral Decision Making, 14, 123–140CrossRefGoogle Scholar
, , & (1997). Brunswikian and Thurstonian origins of bias in probability assessment: On the interpretation of stochastic components of judgment. Journal of Behavioral Decision Making, 10, 189–2103.0.CO;2-4>CrossRefGoogle Scholar
, & (1973). On the psychology of prediction. Psychological Review, 80, 237–251CrossRefGoogle Scholar
, & (1996). Confidence and uncertainty in judgmental forecasting: Differential effects of scenario presentation. Journal of Behavioral Decision Making, 9, 231–2473.0.CO;2-L>CrossRefGoogle Scholar
Larrick, R. B., & Soll, J. B. (2003). Lay intuitions about combining quantitative judgments. Paper presented at the workshop on Information Aggregation in Decision Making, Silver Spring, MD
, & (1986). Demonstrability and social combination processes on mathematical intellective tasks, Journal of Experimental Social Psychology, 22, 177–189CrossRefGoogle Scholar
(1996). An experimental examination of subjective forecast combination. International Journal of Forecasting, 12, 223–233CrossRefGoogle Scholar
Mojzisch, A., & Schulz-Hardt, S. (2004). Information sampling in group decision making: Sampling biases and their consequences. This volume
(1983). An axiomatic approach to expert resolution. Management Science, 29, 24–32CrossRefGoogle Scholar
(1986). Comment on Genest and Zideck's “Combining probability distributions: A critique and annotated bibliography.”Statistical Science, 1, 141–144CrossRefGoogle Scholar
, , & (1993). The adaptive decision maker. New York: Cambridge University PressCrossRefGoogle Scholar
, & (2004). Intuitive evaluation of likelihood judgment producers: Evidence for a confidence heuristic. Journal of Behavioral Decision Making, 17, 39–57CrossRefGoogle Scholar
Rantilla, A. K., & Budescu, D. V. (1999). Aggregation of advisor opinions. In Proceedings of the 32nd Annual Hawai'i International Conference on Systems Sciences
, , & (2001). Information sampling and confidence within groups and judge advisor systems. Communication Research, 28, 737–771CrossRefGoogle Scholar
(1986). The role of variable labels in cue probability learning tasks. Organizational Behavior and Human Decision Processes, 38, 141–161CrossRefGoogle Scholar
(1992). Groups under uncertainty: An examination of confidence in group decision making. Organizational Behavior and Human Decision Processes, 52, 124–155CrossRefGoogle Scholar
, & (1995). Cueing and cognitive conflict in judge-advisor decision making. Organizational Behavior and Human Decision Processes, 62, 159–174CrossRefGoogle Scholar
(1999). Intuitive theories of information: Beliefs about the value of redundancy. Cognitive Psychology, 38, 317–346CrossRefGoogle ScholarPubMed
Soll, J. B., Larrick, R. (2000). The 80/20 rule and the revision of judgment in light another's opinion: Why fo we believe ourseleves so much? Paper presented at the annual meeting of the Society for Judgment and Decision Making (SJDM), Los Angeles
, & (1994). Perceived expertise and its effect on confidence. Organizational Behavior and Human Decision Processes, 57, 290–302CrossRefGoogle Scholar
van Swol, L. M., & Sniezek, J. A. (2002). Trust me, I'm an expert: Trust and confidence and acceptance of expert advice. Paper presented at the 8th Conference on Behavioral Decision Research in Management (BDRM), Chicago
, , , & (1997). Evaluating and combining subjective probability estimates. Journal of Behavioral Decision Making, 10, 243–2683.0.CO;2-M>CrossRefGoogle Scholar
, & (2001). Understanding pooled subjective probability estimates. Mathematical Social Sciences, 18, 1–18CrossRefGoogle Scholar
, & (1989). Combining forecast: Multiple regression versus a Bayesian approach. Decision Sciences, 20, 77–89CrossRefGoogle Scholar
(1971). Probabilistic prediction: Some experimental results. Journal of the American Statistical Association, 66, 675–685CrossRefGoogle Scholar
, & (1983). The combination of forecasts. Journal of the Royal Statistical Society: Series A, 146, 150–157CrossRefGoogle Scholar
, & (1993). Evaluating and combining physician's probabilities of survival in an intensive care unit. Management Science, 39, 1526–1543CrossRefGoogle Scholar
Wittenbaum, G., & Stasser, G. (1996). Management of information in small groups. In & (Eds.), What's social about social cognition? Research on socially shared cognition in small groups. (pp. 3–28). Thousand Oaks, CA: SageCrossRefGoogle Scholar
(1997). Weighting and trimming: Heuristics for aggregating judgments under uncertainty. Organizational Behavior and Human Decision Processes, 69, 237–249CrossRefGoogle Scholar
(2004). Receiving other people's advice: Influence and benefit. Organizational Behavior and Human Decision Processes, 93, 1–13CrossRefGoogle Scholar
, & (2000). Advice taking in decision-making: Egocentric discounting and reputation formation. Organizational Behavior and Human Decision Processes, 83, 260–281CrossRefGoogle ScholarPubMed
- 12
- Cited by