Hostname: page-component-78c5997874-mlc7c Total loading time: 0 Render date: 2024-11-04T19:18:24.937Z Has data issue: false hasContentIssue false

Analyses for elucidating current question answering technology

Published online by Cambridge University Press:  14 February 2002

MARC LIGHT
Affiliation:
The MITRE Corporation, 202 Burlington Road, Bedford, MA 01730, USA e-mail: [email protected]
GIDEON S. MANN
Affiliation:
Department of Computer Science, Johns Hopkins University, Baltimore, MD 21218, USA e-mail: [email protected]
ELLEN RILOFF
Affiliation:
School of Computing, University of Utah, Salt Lake City, UT 84112, USA e-mail: [email protected]
ERIC BRECK
Affiliation:
Department of Computer Science, Cornell University, 4161 Upson Hall, Ithaca, NY 14853, USA e-mail: [email protected]

Abstract

In this paper, we take a detailed look at the performance of components of an idealized question answering system on two different tasks: the TREC Question Answering task and a set of reading comprehension exams. We carry out three types of analysis: inherent properties of the data, feature analysis, and performance bounds. Based on these analyses we explain some of the performance results of the current generation of Q/A systems and make predictions on future work. In particular, we present four findings: (1) Q/A system performance is correlated with answer repetition; (2) relative overlap scores are more effective than absolute overlap scores; (3) equivalence classes on scoring functions can be used to quantify performance bounds; and (4) perfect answer typing still leaves a great deal of ambiguity for a Q/A system because sentences often contain several items of the same type.

Type
Research Article
Copyright
© 2001 Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)