WHY IS WEB SEARCH SO HARD... TO EVALUATE?
Keywords:
Web search, evaluation, user satisfactionAbstract
Web search has several important characteristics that distinguish it from traditional information retrieval: the often adversarial relationship between content creators and search engine designers, the nature of the corpus, and the multiplicity of user goals. In addition to making the search task itself difficult, these characteristics make it particularly hard to evaluate search effectiveness. In this paper, we examine these characteristics and then consider the problems with several different standard evaluation techniques.
Downloads
References
Bates, M.J. The Design of Browsing and Berrypicking Techniques for the Online Search
Interface. Online Review 13, October 1989, 407-424.
Broder, A. A Taxonomy of Web Search. SIGIR Forum 36(2), 2002.
Cho J. and Garcia-Molina H. The Evolution of the Web and Implications for an Incremental
Crawler. Proceedings of the 26th International Conference on Very Large Databases, Sep. 2000.
Cuadra, C.A. and Katter, R.V. Opening the Black Box of ‘Relevance.’ Journal of Documentation
:291-303, 1967.
Fetterly, D., Manasse, M., Najork, M., and Wiener, J.L. A Large-Scale Study of the Evolution of
Web Pages. Proceedings of WWW 2003.
Flynn, L. Desperately Seeking Surfers: Web Programmers Try to Alter Search Engines’ Results.
New York Times, November 11, 1996, p. C5.
Furnas, G.W., Landauer, T.K., Gomez, L.M., and Dumais, S.T. The Vocabulary Problem in
Human-System Communication. Communications of the ACM, 30(11):964-971, November 1987.
Harmon, D. Overview of the First Text REtrieval Conference. Proceedings of the First Text
REtrieval Conference (TREC-1), 1992.
Joachims, T. Optimizing Search Engines Using Clickthrough Data. Proceedings of the ACM
Conference on Knowledge Discovery and Data Mining (KDD), 2002.
Kobayashi, M. and Takeda, K. Information Retrieval on the Web. ACM Computing Surveys,
(2), June 2000.
Koehler, W. A Longitudinal Study of Web Pages Continued: A Report After Six Years.
Information Research, 9(2) paper 174.
Liberatore, K. Getting to the Source: Is it Real or Spam, Ma’am? MacWorld, May 1997.
Mizzaro, S. Relevance, the Whole History. Journal of the American Society for Information
Science, 48(9):810-832, 1997.
Nielsen, J. Usability Engineering. Morgan Kaufmann, 1994.
Ntoulos, A., Cho, J., and Olston, C. What’s New on the Web? The Evolution of the Web from a
Search Engine Perspective. Proceedings of WWW 2004.
Rose, D.E. and Levinson, D. Understanding User Goals in Web Search. Proceedings of WWW
Rubin, J. Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests.
Wiley, 1994.
