TOWARD ENABLING USERS TO VISUALLY EVALUATE THE EFFECTIVENESS OF DIFFERENT SEARCH METHODS
Keywords:
Web search, evaluation, user satisfactionAbstract
This paper explores how information visualization can provide insights into the effectiveness of different query formulations or the same query submitted to multiple search engines. Different queries or search methods are deemed more effective if the fusion of their results leads to a new result set that contains an increased number of relevant documents. The MetaCrystal toolset can be used to visualize the degree of overlap and similarity between the results returned by different queries or engines. The work presented is guided by two working hypotheses: 1) documents found by multiple methods are more likely to be relevant; 2) high degrees of overlap and/or systematic relationships between the ranked lists being compared will not lead to fusion results that contain more relevant documents. MetaCrystal visually identifies documents found by multiple queries or engines. The number and distribution patterns of documents found by multiple methods can be used as a visual measure of the fusion effectiveness. Examples, using Internet and TREC data, are presented that support both in a qualitative and quantitative way the working hypotheses.
Downloads
References
Banks D. , Over P. & Zhang N. (1999) Blind Men and Elephants: Six Approaches to TREC data.
Information Retrieval 1, 7–34,
Belkin, N. & Croft, B. (1992). Information Filtering and Information Retrieval: Two Sides of the Same
Coin. Comm. of the ACM, Dec., 1992.
Chen H., Fan H., Chau M. and Zeng D. MetaSpider: (2001). Meta-Searching and Categorization on the
Web. JASIS, Volume 52 (13), 1134 - 1147.
Eastman, C. and Jansen, B. J. (2003) Coverage, relevance, and ranking: the impact of query operators on
Web search engine results. ACM Transactions on Information Systems. 21(4), 383 - 411.
Foltz, P. and Dumais, St. (1992) Personalized information delivery: An analysis of information-filtering
methods. Comm. of the ACM, 35 (12):51-60.
Gordon, M., & Pathak, P. (1999). Finding information on the World Wide Web: The retrieval effectiveness
of search engines. Information Processing and Management, 35 (2), 141–180.
Grokker – www.groxis.com
Havre, S., Hetzler, E., Perrine K., Jurrus E., and Miller N. (2001). Interactive Visualization of Multiple
Query Results. Proc. IEEE Information Visualization Symp. 2001.
Hearst M. (1999). User interfaces and visualization. Modern Information Retrieval. R. Baeza-Yates and B.
Ribeiro-Neto (eds.). Addison-Wesley, 257-323.
Kartoo – www.kartoo.com
Lawrence, S., & Giles, C.L. (1999). Accessibility of information on the Web. Nature, 400, 107–109.
Olsen, K. A., Korfhage, R. R., Sochats, K. M., Spring, M. B., & Williams, J. G. (1993). “Visualization of a
Document Collection: the VIBE System”, Information Processing & Management, 29(1), 69-81.
Saracevic, T. and Kantor, P. (1988). A study of information seeking and retrieving. III. Searchers, searches
and overlap. JASIS. 39, 3, 197-216.
Saracevic, T. (1995). Evaluation of evaluation in information retrieval. Proceedings of ACM SIGIR ‘ 95.
Spink, A. (2002). A user centered approach to the evaluation of Web search engines: An exploratory study.
Information Processing and Management, 38(3), 401-426.
Spink, A., Wilson, T. D. (1999). Toward a theoretical framework for information retrieval (IR) evaluation
in an information seeking context. Proceedings of MIRA 99:
Spink, A., Wolfram, D., Jansen, B. J., and Saracevic, T. (2001). Searching of the Web: the public and their
queries. JASIS, 52 (3) (2001), 226 - 234 .
Spoerri, A. (1995). InfoCrystal: A Visual Tool for Information Retrieval. Interdepartmental Ph.D. Thesis.
Massachusetts Institute of Technology. February, 1995.
Spoerri, A. (1999). InfoCrystal: A Visual Tool for Information Retrieval. In Card S., Mackinlay J. and B.
Shneiderman (Eds.), Readings in Information Visualization: Using Vision to Think (pp. 140 – 147). San
Francisco: Morgan Kaufmann.
Spoerri, A. (2004). MetaCrystal: A Visual Interface for Meta Searching. Proceedings of ACM CHI 2004.
Spoerri, A. (2004). Coordinated Views and Tight Coupling to Support Meta Searching. Proceedings of
IEEE CMV 2004.
Spoerri, A. (2004). Visual Search Editor for Composing Meta Searches. Proceedings of ASIS&T 2004.
Su. L. T., Chen, H. L., & Dong, X. Y. (1998). Evaluation of Web-based search engines from an end-user's
perspective: A pilot study. Proceedings of ASIS&T 1998.
Vivisimo – www.vivisimo.com.
Voorhees E. & Harman D (2000). Overview of the eighth Text REtrieval Conference (TREC-8). In
E.M.Voorhees and D.K. Harman, editors, Proceedings of the Eighth Text REtrieval Conference (TREC-8),
NIST Special Publication 500-246.
