Book contents
- Frontmatter
- Contents
- Acknowledgments
- 1 Introduction and Overview of Security Games
- PART I SECURITY EXPERTS' PERSPECTIVES
- PART II DEPLOYED APPLICATIONS
- PART III EFFICIENT ALGORITHMS FOR MASSIVE SECURITY GAMES
- PART IV FUTURE RESEARCH
- 10 Effective Solutions for Real-World Stackelberg Games: When Agents Must Deal with Human Uncertainties
- 11 Approximation Methods for Infinite Bayesian Stackelberg Games: Modeling Distributional Payoff Uncertainty
- 12 Stackelberg versus Nash in Security Games: Interchangeability, Equivalence, and Uniqueness
- 13 Evaluating Deployed Decision-Support Systems for Security: Challenges, Analysis, and Approaches
- PART V SHORT BIOS
- References
- Index
- References
13 - Evaluating Deployed Decision-Support Systems for Security: Challenges, Analysis, and Approaches
from PART IV - FUTURE RESEARCH
Published online by Cambridge University Press: 05 January 2012
- Frontmatter
- Contents
- Acknowledgments
- 1 Introduction and Overview of Security Games
- PART I SECURITY EXPERTS' PERSPECTIVES
- PART II DEPLOYED APPLICATIONS
- PART III EFFICIENT ALGORITHMS FOR MASSIVE SECURITY GAMES
- PART IV FUTURE RESEARCH
- 10 Effective Solutions for Real-World Stackelberg Games: When Agents Must Deal with Human Uncertainties
- 11 Approximation Methods for Infinite Bayesian Stackelberg Games: Modeling Distributional Payoff Uncertainty
- 12 Stackelberg versus Nash in Security Games: Interchangeability, Equivalence, and Uniqueness
- 13 Evaluating Deployed Decision-Support Systems for Security: Challenges, Analysis, and Approaches
- PART V SHORT BIOS
- References
- Index
- References
Summary
Introduction
As discussed in other chapters of this book, an increasing number of technically sophisticated tools are available to support decisionmaking for security resource allocation in many different domains. In this chapter we discuss the question of evaluating these deployed security systems, using examples from our own research to illustrate some of the key challenges in doing evaluation for security systems. Two of the primary difficulties are (1) that we cannot rely on adversaries to cooperate in evaluation, which makes it difficult to validate models, and (2) there is (thankfully) very little data available about real-world terrorist attacks.
Despite the difficulties of comprehensive evaluation in security domains, it is only by asking the question, how well does a system work? that policy makers can decide how to allocate finite resources to to different security measures. We discuss the goals of security systems, the elements that comprise these systems, and different approaches for evaluation. Every approach has drawbacks, so in lieu of an ideal test, we advocate a comprehensive style of evaluation that uses diverse metrics and data to perform cost-benefit analysis for the complete system. We also emphasize that the focus of the evaluation is not, is system X the perfect security system? which is an impossible standard. Rather, the relevant question is which of the available alternatives should be used? Providing strong evidence that one alternative is superior to other approaches is often feasible, even when providing exact quantitative measures of value is not.
- Type
- Chapter
- Information
- Security and Game TheoryAlgorithms, Deployed Systems, Lessons Learned, pp. 254 - 284Publisher: Cambridge University PressPrint publication year: 2011
References
- 2
- Cited by