• JURGEN CITO s.e.a.l. – software evolution & architecture lab, University of Zurich, Switzerland
  • DEVAN GOTOWKA Catchpoint Systems, Inc., New York, USA
  • PHILIPP LEITNER s.e.a.l. – software evolution & architecture lab, University of Zurich, Switzerland
  • RYAN PELETTE Catchpoint Systems, Inc., New York, USA
  • DRITAN SULJOTI Catchpoint Systems, Inc., New York, USA
  • SCHAHRAM DUSTDAR Distributed Systems Group, Vienna University of Technology, Austria


Web services, performance engineering, application performance monitoring, changepoint analysis


The large scale of the Internet has offered unique economic opportunities, that in turn introduce overwhelming challenges for development and operations to provide reliable and fast services in order to meet the high demands on the performance of online services. In this paper, we investigate how performance engineers can identify three different classes of externally-visible performance problems (global delays, partial delays, periodic delays) from concrete traces. We develop a simulation model based on a taxonomy of root causes in server performance degradation. Within an experimental setup, we obtain results through synthetic monitoring of a targetWeb service, and observe changes inWeb performance over time through exploratory visual analysis and changepoint detection. We extend our analysis and apply our methods to real-user monitoring (RUM) data. In a use case study, we discuss how our underlying model can be applied to real performance data gathered from a multinational, high-traffic website in the financial sector. Finally, we interpret our findings and discuss various challenges and pitfalls.



Download data is not yet available.


Marcos K Aguilera, Jeffrey C Mogul, Janet L Wiener, Patrick Reynolds, and Athicha Muthitacharoen.

Performance debugging for distributed systems of black boxes. In ACM SIGOPS

Operating Systems Review, volume 37, pages 74–89. ACM, 2003.

Leszek Borzemski. The experimental design for data mining to discover web performance issues

in a wide area network. Cybernetics and Systems, 41(1):31–45, 2010.

Leszek Borzemski and Maciej Drwal. Time series forecasting of web performance data monitored

by mwing multiagent distributed system. In ICCCI (1), pages 20–29, 2010.

Leszek Borzemski and Anna Kaminska-Chuchmala. Knowledge engineering relating to spatial web

performance forecasting with sequential gaussian simulation method. In KES, 1439-1448, 2012.

Leszek Borzemski and Anna Kaminska-Chuchmala. Knowledge discovery about web performance

with geostatistical turning bands method. In KES (2), pages 581–590, 2011.

Leszek Borzemski, Marta Kliber, and Ziemowit Nowak. Using data mining algorithms in web

performance prediction. Cybernetics and Systems, 40(2):176–187, 2009.

Anna Bouch, Allan Kuchinsky, and Nina Bhatti. Quality is in the eye of the beholder: meeting

users’ requirements for internet quality of service. In Proceedings of the SIGCHI conference on

Human factors in computing systems, pages 297–304. ACM, 2000.

Jie Chen and Arjun Gupta. Parametric statistical change point analysis with applications to

genetics, medicine, and finance. Boston, 2012.

Mike Y Chen, Emre Kiciman, Eugene Fratkin, Armando Fox, and Eric Brewer. Pinpoint: Problem

determination in large, dynamic internet services. In Dependable Systems and Networks, 2002.

DSN 2002. Proceedings. International Conference on, pages 595–604. IEEE, 2002.

Yingying Chen, Ratul Mahajan, Baskar Sridharan, and Zhi-Li Zhang. A provider-side view of

web search response time. SIGCOMM Comput. Commun. Rev., 43(4):243–254, August 2013.

Ludmila Cherkasova, Kivanc Ozonat, Ningfang Mi, Julie Symons, and Evgenia Smirni. Automated

anomaly detection and performance modeling of enterprise applications. ACM Transactions on

Computer Systems (TOCS), 27(3):6, 2009.

Thiam Kian Chiew. Web page performance analysis. PhD thesis, University of Glasgow, 2009.

J¨urgen Cito, Dritan Suljoti, Philipp Leitner, and Schahram Dustdar. Identifying root causes of

web performance degradation using changepoint analysis. In Web Engineering, 14th International

Conference, ICWE 2014, Toulouse, France, July 1-4, 2014. Proceedings, pages 181–199, 2014.

Ira Cohen, Steve Zhang, Moises Goldszmidt, Julie Symons, Terence Kelly, and Armando Fox. Capturing,

indexing, clustering, and retrieving system history. In ACM SIGOPS Operating Systems

Review, volume 39, pages 105–118. ACM, 2005.

Yehia Elkhatib, Gareth Tyson, and Michael Welzl. The effect of network and infrastructural

variables on spdy’s performance, 2014.

R. Fielding, J. Gettys, J. Mogul, H. Frystyk, L. Masinter, P. Leach, and T. Berners-Lee. RFC

: Hypertext Transfer Protocol – HTTP/1.1. The Internet Society, June 1999.

David Gourley. HTTP : the definitive guide. O’Reilly, Beijing Sebastopol, CA, 2002.

Russell G Heikes, Douglas C Montgomery, and Ronald L Rardin. Using common random numbers

in simulation experiments - an approach to statistical analysis. Simulation, 27(3):81–85, 1976.

Paul Jaccard. The distribution of flora in alpine zone. 1. New phytologist, 11(2):37–50, 1912.

Rebecca Killick and Idris A. Eckley. changepoint: an R package for changepoint analysis. R

package version 0.5, 2011.

Andrew King. Speed up your site : Web site optimization. New Riders, Indianapolis, Ind, 2003.

Philipp Leitner, Johannes Ferner, Waldemar Hummer, and Schahram Dustdar. Data-Driven and

Automated Prediction of Service Level Agreement Violations in Service Compositions. Distributed

and Parallel Databases, 31(3):447–470, 2013.

Zhen Liu, Nicolas Niclausse, Cesar Jalpa-Villanueva, and Sylvain Barbier. Traffic Model and

Performance Evaluation of Web Servers. Technical Report RR-3840, INRIA, December 1999.

Joao Paulo Magalhaes and Luis Moura Silva. Anomaly detection techniques for web-based applications:

An experimental study. In Network Computing and Applications (NCA), 2012 11th IEEE

International Symposium on, pages 181–190. IEEE, 2012.

Makoto Matsumoto and Takuji Nishimura. Mersenne twister: a 623-dimensionally equidistributed

uniform pseudo-random number generator. ACM Transactions on Modeling and Computer Sim-

ulation (TOMACS), 8(1):3–30, 1998.

Paul. Mockapetris.

Thanh HD Nguyen, Bram Adams, Zhen Ming Jiang, Ahmed E Hassan, Mohamed Nasser, and

Parminder Flora. Automated detection of performance regressions using statistical process control

techniques. In Proceedings of the third joint WOSP/SIPEW international conference on Perfor-

mance Engineering, pages 299–310. ACM, 2012.

R Core Team. R: A Language and Environment for Statistical Computing. R Foundation for

Statistical Computing, Vienna, Austria, 2013.

Forrester research. Ecommerce web site performance today: An updated look at consumer reaction

to a poor online shopping experience, August 2009.

Behrooz A. Shirazi, Krishna M. Kavi, and Ali R. Hurson, editors. Scheduling and Load Balancing

in Parallel and Distributed Systems. IEEE Computer Society Press, Los Alamitos,, 1995.

Amanda Strong. A review of anomaly detection with focus on changepoint detection, 2012.

Clifford H Wagner. Simpson’s paradox in real life. The American Statistician, 36(1):46–48, 1982.