Book contents
- Frontmatter
- Contents
- Contributors
- Preface
- Editorial
- Guest Editorial: Meeting the challenge
- 1 Fifty years of UK research in information science
- 2 Smoother pebbles and the shoulders of giants: the developing foundations of information science
- 3 The last 50 years of knowledge organization: a journey through my personal archives
- 4 On the history of evaluation in IR
- 5 The information user: past, present and future
- 6 The sociological turn in information science
- 7 From chemical documentation to chemoinformatics: 50 years of chemical information science
- 8 Health informatics: current issues and challenges
- 9 Social informatics and sociotechnical research – a view from the UK
- 10 The evolution of visual information retrieval
- 11 Information policies: yesterday, today, tomorrow
- 12 The disparity in professional qualifications and progress in information handling: a European perspective
- 13 Electronic scholarly publishing and Open Access
- 14 Social software: fun and games, or business tools?
- 15 Bibliometrics to webometrics
- 16 How I learned to love the Brits
- Index
4 - On the history of evaluation in IR
Published online by Cambridge University Press: 08 June 2018
- Frontmatter
- Contents
- Contributors
- Preface
- Editorial
- Guest Editorial: Meeting the challenge
- 1 Fifty years of UK research in information science
- 2 Smoother pebbles and the shoulders of giants: the developing foundations of information science
- 3 The last 50 years of knowledge organization: a journey through my personal archives
- 4 On the history of evaluation in IR
- 5 The information user: past, present and future
- 6 The sociological turn in information science
- 7 From chemical documentation to chemoinformatics: 50 years of chemical information science
- 8 Health informatics: current issues and challenges
- 9 Social informatics and sociotechnical research – a view from the UK
- 10 The evolution of visual information retrieval
- 11 Information policies: yesterday, today, tomorrow
- 12 The disparity in professional qualifications and progress in information handling: a European perspective
- 13 Electronic scholarly publishing and Open Access
- 14 Social software: fun and games, or business tools?
- 15 Bibliometrics to webometrics
- 16 How I learned to love the Brits
- Index
Summary
Abstract
This paper is a personal take on the history of evaluation experiments in information retrieval. It describes some of the early experiments that were formative in our understanding, and goes on to discuss the current dominance of TREC (the Text REtrieval Conference) and to assess its impact.
Introduction
The foundation of the Institute of Information Scientists in the UK in 1958 coincides closely with the beginning of the notion of experimental evaluation of information retrieval systems. Although there had been some earlier attempts, we usually mark the start of the tradition as the Cranfield experiments, which ran from 1958 to 1966. Information retrieval is commonly regarded as a core component of information science, and systematic empirical evaluation of IR systems probably represents the strongest claim that information science can make to being a science in any traditional sense. There is a nice irony here: the founder of the empirical tradition in IR, the Cranfield librarian Cyril Cleverdon, was not at all a supporter of the Institute. But more of this anon.
As for the present, and despite the concerns of the founders of the Institute, academic information science is now quite closely associated with the former library schools, many of which have adopted titles which include the word ‘information’. However, a lot of current work in IR, theoretical and experimental, takes place elsewhere, mainly in computer science departments, though several other academic domains are represented. It probably comes as a considerable surprise to a current PhD student, working on (say) a machine learning optimization technique applied to search engine ranking, that he or she is in thrall to an experimental tradition founded by a librarian, working with card indexes, a half-century ago.
Thus the history that is the subject of this paper is not too readily defined in terms of institutional or academic boundaries – or national ones. Despite this, it can be seen as a remarkably coherent development of a set of principles and methods. Like all academic subjects it generates argument and disagreement and heated disputation, but there remains a relatively stable common core, which has, despite its limitations (I will argue), served us well over the last 50 years. Furthermore, while its present international status developed out of a US dominance for a large part of that period, the strength of the UK contribution has been remarkable.
- Type
- Chapter
- Information
- Information Science in Transition , pp. 63 - 94Publisher: FacetPrint publication year: 2009