IDENTIFYING WEB PERFORMANCE DEGRADATIONS THROUGH SYNTHETIC AND REAL-USER MONITORING
Keywords:Web services, performance engineering, application performance monitoring, changepoint analysis
The large scale of the Internet has offered unique economic opportunities, that in turn introduce overwhelming challenges for development and operations to provide reliable and fast services in order to meet the high demands on the performance of online services. In this paper, we investigate how performance engineers can identify three different classes of externally-visible performance problems (global delays, partial delays, periodic delays) from concrete traces. We develop a simulation model based on a taxonomy of root causes in server performance degradation. Within an experimental setup, we obtain results through synthetic monitoring of a targetWeb service, and observe changes inWeb performance over time through exploratory visual analysis and changepoint detection. We extend our analysis and apply our methods to real-user monitoring (RUM) data. In a use case study, we discuss how our underlying model can be applied to real performance data gathered from a multinational, high-traffic website in the financial sector. Finally, we interpret our findings and discuss various challenges and pitfalls.
Marcos K Aguilera, Jeffrey C Mogul, Janet L Wiener, Patrick Reynolds, and Athicha Muthitacharoen.
Performance debugging for distributed systems of black boxes. In ACM SIGOPS
Operating Systems Review, volume 37, pages 74–89. ACM, 2003.
Leszek Borzemski. The experimental design for data mining to discover web performance issues
in a wide area network. Cybernetics and Systems, 41(1):31–45, 2010.
Leszek Borzemski and Maciej Drwal. Time series forecasting of web performance data monitored
by mwing multiagent distributed system. In ICCCI (1), pages 20–29, 2010.
Leszek Borzemski and Anna Kaminska-Chuchmala. Knowledge engineering relating to spatial web
performance forecasting with sequential gaussian simulation method. In KES, 1439-1448, 2012.
Leszek Borzemski and Anna Kaminska-Chuchmala. Knowledge discovery about web performance
with geostatistical turning bands method. In KES (2), pages 581–590, 2011.
Leszek Borzemski, Marta Kliber, and Ziemowit Nowak. Using data mining algorithms in web
performance prediction. Cybernetics and Systems, 40(2):176–187, 2009.
Anna Bouch, Allan Kuchinsky, and Nina Bhatti. Quality is in the eye of the beholder: meeting
users’ requirements for internet quality of service. In Proceedings of the SIGCHI conference on
Human factors in computing systems, pages 297–304. ACM, 2000.
Jie Chen and Arjun Gupta. Parametric statistical change point analysis with applications to
genetics, medicine, and finance. Boston, 2012.
Mike Y Chen, Emre Kiciman, Eugene Fratkin, Armando Fox, and Eric Brewer. Pinpoint: Problem
determination in large, dynamic internet services. In Dependable Systems and Networks, 2002.
DSN 2002. Proceedings. International Conference on, pages 595–604. IEEE, 2002.
Yingying Chen, Ratul Mahajan, Baskar Sridharan, and Zhi-Li Zhang. A provider-side view of
web search response time. SIGCOMM Comput. Commun. Rev., 43(4):243–254, August 2013.
Ludmila Cherkasova, Kivanc Ozonat, Ningfang Mi, Julie Symons, and Evgenia Smirni. Automated
anomaly detection and performance modeling of enterprise applications. ACM Transactions on
Computer Systems (TOCS), 27(3):6, 2009.
Thiam Kian Chiew. Web page performance analysis. PhD thesis, University of Glasgow, 2009.
J¨urgen Cito, Dritan Suljoti, Philipp Leitner, and Schahram Dustdar. Identifying root causes of
web performance degradation using changepoint analysis. In Web Engineering, 14th International
Conference, ICWE 2014, Toulouse, France, July 1-4, 2014. Proceedings, pages 181–199, 2014.
Ira Cohen, Steve Zhang, Moises Goldszmidt, Julie Symons, Terence Kelly, and Armando Fox. Capturing,
indexing, clustering, and retrieving system history. In ACM SIGOPS Operating Systems
Review, volume 39, pages 105–118. ACM, 2005.
Yehia Elkhatib, Gareth Tyson, and Michael Welzl. The effect of network and infrastructural
variables on spdy’s performance, 2014.
R. Fielding, J. Gettys, J. Mogul, H. Frystyk, L. Masinter, P. Leach, and T. Berners-Lee. RFC
: Hypertext Transfer Protocol – HTTP/1.1. The Internet Society, June 1999.
David Gourley. HTTP : the definitive guide. O’Reilly, Beijing Sebastopol, CA, 2002.
Russell G Heikes, Douglas C Montgomery, and Ronald L Rardin. Using common random numbers
in simulation experiments - an approach to statistical analysis. Simulation, 27(3):81–85, 1976.
Paul Jaccard. The distribution of flora in alpine zone. 1. New phytologist, 11(2):37–50, 1912.
Rebecca Killick and Idris A. Eckley. changepoint: an R package for changepoint analysis. R
package version 0.5, 2011.
Andrew King. Speed up your site : Web site optimization. New Riders, Indianapolis, Ind, 2003.
Philipp Leitner, Johannes Ferner, Waldemar Hummer, and Schahram Dustdar. Data-Driven and
Automated Prediction of Service Level Agreement Violations in Service Compositions. Distributed
and Parallel Databases, 31(3):447–470, 2013.
Zhen Liu, Nicolas Niclausse, Cesar Jalpa-Villanueva, and Sylvain Barbier. Traffic Model and
Performance Evaluation of Web Servers. Technical Report RR-3840, INRIA, December 1999.
Joao Paulo Magalhaes and Luis Moura Silva. Anomaly detection techniques for web-based applications:
An experimental study. In Network Computing and Applications (NCA), 2012 11th IEEE
International Symposium on, pages 181–190. IEEE, 2012.
Makoto Matsumoto and Takuji Nishimura. Mersenne twister: a 623-dimensionally equidistributed
uniform pseudo-random number generator. ACM Transactions on Modeling and Computer Sim-
ulation (TOMACS), 8(1):3–30, 1998.
Thanh HD Nguyen, Bram Adams, Zhen Ming Jiang, Ahmed E Hassan, Mohamed Nasser, and
Parminder Flora. Automated detection of performance regressions using statistical process control
techniques. In Proceedings of the third joint WOSP/SIPEW international conference on Perfor-
mance Engineering, pages 299–310. ACM, 2012.
R Core Team. R: A Language and Environment for Statistical Computing. R Foundation for
Statistical Computing, Vienna, Austria, 2013.
Forrester research. Ecommerce web site performance today: An updated look at consumer reaction
to a poor online shopping experience, August 2009.
Behrooz A. Shirazi, Krishna M. Kavi, and Ali R. Hurson, editors. Scheduling and Load Balancing
in Parallel and Distributed Systems. IEEE Computer Society Press, Los Alamitos,, 1995.
Amanda Strong. A review of anomaly detection with focus on changepoint detection, 2012.
Clifford H Wagner. Simpson’s paradox in real life. The American Statistician, 36(1):46–48, 1982.