Introduction

From the perspective of the user of an information retrieval system, information retrieval involves a number of compromises, many of which have been discussed elsewhere in this book. The user's information need may be poorly con - ceptualized, and may be poorly expressed in search or query terms. While recognizing the problems inherent in producing a query that is representative of the user's information need, in this chapter we will take the query as the starting point, and look at the means by which the subsequent processing is carried out – at what goes on ‘under the hood’ once the user releases a query to the system.

For many users of information systems, the process by which their query is compared to millions of documents, resulting in a list in some ranked order, is a mystery. The search engine is a black box, and the processes that occur within it are not at all transparent to the user. This is especially true in the web environment, where the techniques used by search engines are proprietary, and very little published information on retrieval methods is available. This lack of transparency arises in part because of the commercial advantage attached to providing a better search engine (and attracting more users and more advertising revenues), and in part because search engine companies wish to protect themselves against unscrupulous information providers who may use detailed knowledge about the retrieval algorithms to design web pages to receive a high ranking from the search engine, whether or not they are actually relevant to a particular query. A discussion of the problems of information retrieval on the web is provided in Chapter 10, ‘Web Retrieval, Ranking and Personalization’.

However, despite the lack of transparency about the details of specific search engines, the general principles underlying the techniques used by retrieval engines are well known. These techniques are publicized in the research literature as well as in many textbooks (see, for example, Manning, Raghavan and Schutze, 2008; Croft, Metzler and Strohman, 2009).