• MAXIMILIAN SPEICHER Department of Computer Science, Technische Universitat Chemnitz 09111 Chemnitz, Germany
  • SEBASTIAN NUCK Agile Knowledge Engineering and Semantic Web Group, Leipzig University 04109 Leipzig, Germany
  • LARS WESEMANN Research & Development, Unister GmbH 04109 Leipzig, Germany
  • ANDREAS BOTH Research & Development, Unister GmbH 04109 Leipzig, Germany
  • MARTIN GAEDKE Department of Computer Science, Technische Universitat Chemnitz 09111 Chemnitz, Germany


Case Study, Crowdsourcing, Industry, Interaction Tracking, Learning to Rank, Real-Time, Relevance Prediction, Streaming


The prime aspect of quality for search-driven web applications is to provide users with the best possible results for a given query. Thus, it is necessary to predict the relevance of re- sults a priori. Current solutions mostly engage clicks on results for respective predictions, but research has shown that it is highly benecial to also consider additional features of user interaction. Nowadays, such interactions are produced in steadily growing amounts by internet users. Processing these amounts calls for streaming-based approaches and incrementally updatable relevance models. We present StreamMyRelevance!|a novel streaming-based system for ensuring quality of ranking in search engines. Our approach provides a complete pipeline from collecting interactions in real-time to processing them incrementally on the server side. We conducted a large-scale evaluation with real-world data from the hotel search domain. Results show that our system yields predictions as good as those of competing state-of-the-art systems, but by design of the underlying framework at higher eciency, robustness, and scalability. Additionally, our system has been transferred into a real-world industry context. A modied solution called Turtle has been integrated into a new search engine for general web search. To obtain high-quality judgments for learning relevance models, it has been augmented with a novel crowdsourcing tool.



Download data is not yet available.


N. Craswell, O. Zoeter, M. Tylor and B. Ramsey (2008), An Experimental Comparison of Click

Position-Bias Models, Proc. WSDM, pp. 87{94.

Q. Guo and E. Agichtein (2012), Beyond Dwell Time: Estimating Document Relevance from Cur-

sor Movements and other Post-click Searcher Behavior, Proc. WWW, pp. 569{578.

J. Huang, R.W. White and S. Dumais (2011), No Clicks, No Problem: Using Cursor Movements

to Understand and Improve Search, Proc. CHI, pp. 1225{1234.

V. Navalpakkam and E.F. Churchill (2012), Mouse Tracking: Measuring and Predicting Users'

Experience of Web-based Content, Proc. CHI, pp. 2963{2972.

J. Huang (2011), On the Value of Page-Level Interactions in Web Search, Proc. HCIR Workshop.

M. Speicher, A. Both and M. Gaedke (2013), TellMyRelevance! Predicting the Relevance of Web

Search Results from Cursor Interactions, Proc. CIKM, pp. 1281{1290.

J. Huang, R.W. White, G. Buscher and K. Wang (2012), Improving Searcher Models Using Mouse

Cursor Activity, Proc. SIGIR, pp. 195{204.

C. Liu, F. Guo and C. Faloutsos (2009), BBM: Bayesian Browsing Model from Petabyte-scale

Data, Proc. KDD, pp. 537{546.

A. Both, A.-C. Ngonga Ngomo, R. Usbeck, D. Lukovnikov, C. Lemke and M. Speicher (2014),

A Service-oriented Search Framework for Full Text, Geospatial and Semantic Search, Proc. SE-

MANTiCS, pp. 65{72.

N. Marz, Storm Wiki,, retrieved October 15, 2014.

M. Zaharia, T. Das, H. Li, T. Hunter, S. Shenker and I. Stoica (2012), Discretized streams: A

fault-tolerant model for scalable stream processing, Technical Report, UC Berkeley.

T. Joachims (2002), Optimizing Search Engines using Clickthrough Data, Proc. KDD, pp. 133{142.

F. Guo, C. Liu and Y.M. Wang (2009), Ecient Multiple-Click Models in Web Search, Proc.

WSDM, pp. 124{131.

O. Chapelle and Y. Zhang (2009), A Dynamic Bayesian Network Click Model for Web Search,

Proc. WWW, pp. 1{10.

G.E. Dupret and B. Piwowarski (2008), A User Browsing Model to Predict Search Engine Click

Data from Past Observations, Proc. SIGIR, pp. 331{338.

M. Hall, E. Frank, G. Holmes, B. Pfahringer, P. Reutemann and I.H. Witten (2009), The WEKA

Data Mining Software: An Update, SIGKDD Explor. Newsl., 11(1), pp. 10{18.

P. Baldi, S. Brunak, Y. Chauvin, C.A. Andersen and H. Nielsen (2000), Assessing the accuracy of

prediction algorithms for classi cation: an overview, Bioinformatics, 16(5), pp. 412{424.

P. Domingos and G. Hulten (2000), Mining High-Speed Data Streams, Proc. KDD, pp. 71{80.

J. Dean, S. Ghemawat (2008), MapReduce: Simpli ed Data Processing on Large Clusters, CACM,

(1), pp. 107{113.

F. Radlinski (2007), Addressing Malicious Noise in Clickthrough Data, Proc. LR4IR@SIGIR.

J. Bian, Y. Liu, E. Agichtein and H. Zha (2008), A Few Bad Votes Too Many? Towards Robust

Ranking in Social Media, Proc. AIRWeb, pp. 53{60.

A. Tsymbal (2004), The problem of concept drift: de nitions and related work, Technical Report,

Trinity College Dublin.

G. Hulten, L. Spencer and P. Domingos (2001), Mining Time-Changing Data Streams, Proc. KDD,

pp. 97{106.

M. Speicher, S. Nuck, A. Both and M. Gaedke (2014), StreamMyRelevance! Prediction of Result

Relevance from Real-Time Interactions and its Application to Hotel Search, Proc. ICWE, pp.