Hostname: page-component-78c5997874-4rdpn Total loading time: 0 Render date: 2024-11-07T07:10:44.909Z Has data issue: false hasContentIssue false
  • English
  • Français

Query-focused multi-document summarization: automatic data annotations and supervised learning approaches

Published online by Cambridge University Press:  07 April 2011

YLLIAS CHALI  and
SADID A. HASAN
YLLIAS CHALI
Affiliation:
University of Lethbridge, Lethbridge, Alberta T1K 3M4, Canada e-mail: [email protected], [email protected]
SADID A. HASAN
Affiliation:
University of Lethbridge, Lethbridge, Alberta T1K 3M4, Canada e-mail: [email protected], [email protected]
Get access Rights & Permissions [Opens in a new window]

Abstract

In this paper, we apply different supervised learning techniques to build query-focused multi-document summarization systems, where the task is to produce automatic summaries in response to a given query or specific information request stated by the user. A huge amount of labeled data is a prerequisite for supervised training. It is expensive and time-consuming when humans perform the labeling task manually. Automatic labeling can be a good remedy to this problem. We employ five different automatic annotation techniques to build extracts from human abstracts using ROUGE, Basic Element overlap, syntactic similarity measure, semantic similarity measure, and Extended String Subsequence Kernel. The supervised methods we use are Support Vector Machines, Conditional Random Fields, Hidden Markov Models, Maximum Entropy, and two ensemble-based approaches. During different experiments, we analyze the impact of automatic labeling methods on the performance of the applied supervised methods. To our knowledge, no other study has deeply investigated and compared the effects of using different automatic annotation techniques on different supervised learning approaches in the domain of query-focused multi-document summarization.

Type
Articles
Information
Natural Language Engineering , Volume 18 , Issue 1 , January 2012 , pp. 109 - 145
Copyright
Copyright © Cambridge University Press 2011

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Banko, M., Mittal, V., Kantrowitz, M. and Goldstein, J. 1999. Generating extraction-based summaries from hand-written summaries by aligning text spans. In Proceedings of the 4th Meeting of the Pacific Association for Computational Linguistics, PACLING, Waterloo, Canada.Google Scholar
Barzilay, R. and Elhadad, N. 2003. Sentence alignment for monolingual comparable corpora. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 25–32, Sapporo, Japan.ACL.Google Scholar
Berger, A. L., Pietra, V. J. D and Pietra, S. A. D. 1996. A maximum entropy approach to natural language processing. Computational Linguistics 22 (1): 39–71 (Cambridge, MA: MIT Press).Google Scholar
Cancedda, N., Gaussier, E., Goutte, C., and Renders, J. M. 2003. Word sequence kernels. Journal of Machine Learning Research 3: 1059–1082 (Cambridge, MA: MIT Press).Google Scholar
Carbonell, J., Geng, Y., and Goldstein, J. 1997. Automated query-relevant summarization and diversity-based reranking. In Proceedings of the 15th International Joint Conference on Artificial Intelligence, Workshop: AI in Digital Libraries, pp. 12–19, Nagoya, Japan. IJCAI.Google Scholar
Chali, Y., Hasan, S. A., and Joty, S. R. 2009a. A SVM-based ensemble approach to multi-document summarization. In Proceedings of the 22nd Canadian Conference on Artificial Intelligence, pp. 199–202, Kelowna, Canada. Berlin, Germany: Springer-Verlag.Google Scholar
Chali, Y., Hasan, S. A., and Joty, S. R. 2009b. Do automatic annotation techniques have any impact on supervised complex question answering? In Proceedings of the Joint conference of the 47th Annual Meeting of the Association for Computational Linguistics and the 4th International Joint Conference on Natural Language Processing, pp. 329–332, Suntec, Singapore. ACL.Google Scholar
Chali, Y., and Joty, S. R. 2007. Word sense disambiguation using lexical cohesion. In Proceedings of the 4th International Conference on Semantic Evaluations, pp. 476–479, Prague, Czech Republic. ACL.Google Scholar
Chali, Y., and Joty, S. R. 2008. Selecting sentences for answering complex questions. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 304–313, Hawaii. ACL.CrossRefGoogle Scholar
Chali, Y., Joty, S. R., and Hasan, S. A. 2009. Complex question answering: unsupervised learning approaches and experiments. Journal of Artificial Intelligence Research 35 (1): 1–47 (El Segundo, CA, USA: AI Access Foundation).CrossRefGoogle Scholar
Charniak, E. 2000. A maximum-entropy-inspired parser. In Proceedings of the 1st North American Chapter of the Association for Computational Linguistics Conference, pp. 132–139, Seattle, Washington. Massachusetts, USA: Morgan Kaufmann.Google Scholar
Collins, M., and Duffy, N. 2001. Convolution kernels for natural language. In Proceedings of Advances in Neural Information Processing Systems 14, pp. 625–632, Vancouver, Canada. Cambridge, MA, USA: MIT Press.Google Scholar
Conroy, J. M. and O'Leary, D. P. 2001. Text summarization via hidden Markov models. In Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp, 406–407, New Orleans, USA. New York, NY, USA: ACM.CrossRefGoogle Scholar
Conroy, J. M., Schlesinger, J. D. and O'Leary, D. P. 2006. Topic-focused multi-document summarization using an approximate Oracle score. In Proceedings of the COLING/ACL on Main Conference Poster Sessions, pp. 152–159, Sydney, Australia. ACL.CrossRefGoogle Scholar
Cortes, C., and Vapnik, V. 1995. Support vector networks. Machine Learning 20 (3): 273–297 (Hingham, USA: Kluwer).CrossRefGoogle Scholar
Daumé, H. III, and Marcu, D. 2006. Bayesian query-focused summarization. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th Annual Meeting of the Association for Computational Linguistics, pp. 305–312, Sydney, Australia. ACL.Google Scholar
Dietterich, T. G. 2000. Ensemble methods in machine learning. In Proceedings of the First International Workshop on Multiple Classifier Systems, pp. 1–15, London, UK. Berlin, Germany: Springer-Verlag.Google Scholar
Edmundson, H. P. 1969. New methods in automatic extracting. Journal of the Association for Computing Machinery (ACM) 16 (2): 264–285 (New York, NY, USA: ACM).CrossRefGoogle Scholar
Efron, B., and Tibshirani, R. J. 1994. An Introduction to the Bootstrap. Boca Raton, FL, USA: CRC Press.CrossRefGoogle Scholar
Fellbaum, C. 1998. WordNet-An Electronic Lexical Database. Cambridge, MA, USA: MIT Press.CrossRefGoogle Scholar
Ferrier, L. 2001. A Maximum Entropy Approach to Text Summarization. MSc thesis, School of Artificial Intelligence, Division of Informatics, University of Edinburgh.Google Scholar
Hacioglu, K., Pradhan, S., Ward, W., Martin, J. H., and Jurafsky, D. 2003. Shallow semantic parsing using support vector machines. Technical Report TR-CSLR-2003-1, Center for Spoken Language Research, Boulder, CO, USA.Google Scholar
Harnly, A., Nenkova, A., Passonneau, R., and Rambow, O. 2005. Automation of summary evaluation by the pyramid method. In Proceedings of the Conference of Recent Advances in Natural Language Processing, pp. 226–232, Borovets, Bulgeria. RANLP.Google Scholar
Hirao, T., Isozaki, H., Maeda, E., and Matsumoto, Y. 2002a. Extracting important sentences with support vector machines. In Proceedings of the 19th International Conference on Computational Linguistics – Vol. 1, pp. 1–7, Taipei, Taiwan. ACL.Google Scholar
Hirao, T., Sasaki, Y., Isozaki, H., and Maeda, E. 2002b. NTT's text summarization system for DUC-2002. In Proceedings of the Document Understanding Conference, pp. 104–107, Philadelphia, PA, USA. Gaithersburg, MD, USA: NIST.Google Scholar
Hirao, T., Suzuki, J., Isozaki, H., and Maeda, E. 2003. NTT's multiple document summarization system for DUC-2003. In Proceedings of the Document Understanding Conference, Edmonton, Canada. Gaithersburg, MD, USA: NIST.Google Scholar
Hirao, T., Suzuki, J., Isozaki, H., and Maeda, E. 2004. Dependency-based sentence alignment for multiple document summarization. In Proceedings of the 20th International Conference on Computational Linguistics, pp. 446–452, Geneva, Switzerland. ACL.Google Scholar
Hovy, E., Lin, C. Y., and Zhou, L. 2005. A BE-based multi-document summarizer with query interpretation. In Proceedings of the Document Understanding Conference, Vancouver, BC, Canada. Gaithersburg, MD, USA: NIST.Google Scholar
Hovy, E., Lin, C. Y., Zhou, L., and Fukumoto, J. 2006. Automated summarization evaluation with basic elements. In Proceedings of the 5th Conference on Language Resources and Evaluation, Genoa, Italy. LREC.Google Scholar
Hsu, C., Chang, C., and Lin, C. 2008. A Practical Guide to Support Vector Classification, Taipei, Taiwan: National Taiwan University. http://www.csie.ntu.edu.tw/cjlin.Google Scholar
Jing, H., and McKeown, K. R. 1999. The decomposition of human-written summary sentences. In Proceedings of the 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 129–136, Berkeley, CA, USA. New York, NY, USA: ACM.CrossRefGoogle Scholar
Joachims, T. 1998. Text categorization with support vector machines: learning with many relevant features. In Proceedings of the 10th European Conference on Machine Learning, pp. 137–142, Chemnitz, Germany. Berlin, Germany: Springer-Verlag.Google Scholar
Joachims, T. 1999. Making Large-Scale Support Vector Machine Learning Practical, pp. 169–184. Cambridge, MA, USA: MIT Press.Google Scholar
Jurafsky, D., and Martin, J. H. 2009. Speech and Language Processing: An Introduction to Natural Language Processing, Speech Recognition, and Computational Linguistics, 2nd ed. Boston, MA, USA: Prentice-Hall.Google Scholar
Kingsbury, P., and Palmer, M. 2002. From Treebank to PropBank. In Proceedings of the International Conference on Language Resources and Evaluation, pp. 1989–1993, Las Palmas, Spain. LREC.Google Scholar
Kudo, T., and Matsumoto, Y. 2001. Chunking with support vector machine. In Proceedings of the 2nd Meeting of the North American Chapter of the Association for Computational Linguistics on Language Technologies, pp. 192–199, Pittsburgh, USA. ACL.Google Scholar
Kupiec, J., Pedersen, J., and Chen, F. 1995. A trainable document summarizer. In Proceedings of the 18th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 68–73, Seattle, USA. New York, NY, USA: ACM.Google Scholar
Lafferty, J. D., McCallum, A., and Pereira, F. C. N. 2001. Conditional random fields: probabilistic models for segmenting and labeling sequence data. In Proceedings of the 18th International Conference on Machine Learning, pp. 282–289, Williamstown, MA, USA. Massachusetts, USA: Morgan Kaufmann.Google Scholar
Lin, C. Y. 2004. ROUGE: a package for automatic evaluation of summaries. In Proceedings of Workshop on Text Summarization Branches Out, Post-Conference Workshop of Association for Computational Linguistics, pp. 74–81, Barcelona, Spain. ACL.Google Scholar
Lodhi, H., Saunders, C., Shawe-Taylor, J., Cristianini, N., and Watkins, C. 2002. Text classification using string kernels. Journal of Machine Learning Research 2: 419–444 (Cambridge, MA, USA: MIT Press).Google Scholar
Mani, I. 2001. Automatic Summarization. Natural Language Processing. Philadelphia, PA, USA: John Benjamins.CrossRefGoogle Scholar
Mani, I., and Maybury, M. T. 1999. Advances in Automatic Text Summarization. Cambridge, MA, USA: MIT Press.Google Scholar
Marcu, D. 1999. The automatic construction of large-scale corpora for summarization research. In Proceedings of the 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 137–144, Berkeley, CA, USA. New York, NY, USA: ACM.CrossRefGoogle Scholar
McCallum, A. K. 2002. MALLET: a machine learning for language toolkit. http://mallet.cs.umass.edu.Google Scholar
Moschitti, A., and Basili, R. 2006. A tree kernel approach to question and answer classification in question answering systems. In Proceedings of the 5th International Conference on Language Resources and Evaluation, pp. 1510–1513, Genoa, Italy. LREC.Google Scholar
Moschitti, A., Quarteroni, S., Basili, R., and Manandhar, S. 2007. Exploiting syntactic and shallow semantic kernels for question/answer classificaion. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pp. 776–783, Prague, Czech Republic. ACL.Google Scholar
Nastase, V. 2008. Topic-driven multi-document summarization with encyclopedic knowledge and spreading activation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 763–772, Honolulu, Hawaii, USA. ACL.Google Scholar
Nguyen, M. L., Shimazu, A., Phan, X. H., Ho, T. B., and Horiguchi, S. 2005. Sentence extraction with support vector machine ensemble. In First World Congress of the International Federation for Systems Research (IFSR'05), Symposium on Data/Text Mining from Large Databases, Kobe, Japan. Komatsu, Japan: JAIST Press.Google Scholar
Parmanto, B., Munro, P. W., and Doyle, H. R. 1996. Improving committee diagnosis with resampling techniques. In Advances in Neural Information Processing Systems, vol. 8, pp. 882–888, Denver, CO, USA. NIPS.Google Scholar
Pasca, M., and Harabagiu, S. M. 2001. Answer mining from on-line documents. In Proceedings of the 39th Annual Meeting of the Association for Computational Linguistics and 10th Conference of the European Chapter Workshop on Open-Domain Question Answering, pp.38–45, Toulouse, France. ACL.Google Scholar
Qi, H., and Huang, M. 2007. Research on SVM ensemble and its application to remote sensing classification. In Proceedings of the International Conference on Intelligent Systems and Knowledge Engineering (ISKE), Chengdu, China. Amsterdam, Netherlands: Atlantis Press.Google Scholar
Rooney, N., Patterson, D., Tsymbal, A., and Anand, S. 2004. Random subspacing for regression ensembles. In Proceedings of the 17th International Florida Artificial Intelligence Research Society Conference (FLAIRS), Miami Beach, FL, USA. California, USA: AAAI Press.Google Scholar
Sekine, S. 2002. Proteus project-OAK system (English sentence analyzer). http://nlp.nyu.edu/oak.Google Scholar
Sekine, S., and Nobata, C. A. 2001. Sentence extraction with information extraction technique. In Proceedings of the Document Understanding Conference, New Orleans, LA, USA. Gaithersburg, MD, USA: NIST.Google Scholar
Shen, D., Sun, J., Li, H., Yang, Q., and Chen, Z. 2007. Document summarization using conditional random fields. In Proceedings of the 20th International Joint Conference on Artificial Intelligence, pp. 2862–2867, Hyderabad, India. Massachusetts, USA: Morgan Kaufmann.Google Scholar
Toutanova, K., Brockett, C., Gamon, M., Jagarlamudi, J., Suzuki, H., and Vanderwende, L. 2007. The PYTHY summarization system: Microsoft research at DUC 2007. In Proceedings of the Document Understanding Conference, Rochester, NY, USA. Gaithersburg, MD, USA: NIST.Google Scholar
Wallach, H. 2002. Efficient Training of Conditional Random Fields. MSc thesis, Division of Informatics, University of Edinburgh.Google Scholar
Wan, X., and Xiao, J. 2009. Graph-based multi-modality learning for topic-focused multi-document summarization. In Proceedings of the 21st International Joint Conference on Artifical Intelligence, pp. 1586–1591, Pasadena, CA, USA. Massachusetts, USA: Morgan Kaufmann.Google Scholar
Wan, X., Yang, J., and Xiao, J. 2007. Manifold-ranking based topic-focused multi-document summarization. In Proceedings of the 20th International Joint Conference on Artifical Intelligence, pp. 2903–2908, Hyderabad, India. Massachusetts, USA: Morgan Kaufmann.Google Scholar
Wong, K., Wu, M., and Li, W. 2008. Extractive summarization using supervised and semi-supervised learning. In Proceedings of the 22nd International Conference on Computational Linguistics – Vol. 1, pp. 985–992, Manchester, UK. ACL.Google Scholar
Zhang, D., and Lee, W. S. 2003. Question classification using support vector machines. In Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Informaion Retrieval, pp. 26–32, Toronto, Canada. New York, NY, USA: ACM.CrossRefGoogle Scholar