Auto-Extraction and Integration of Metrics for Web User Interfaces
Keywords:Automated metrics, HCI Vision, web design mining, visual complexity
Metric-based assessment of web user interface (WUI) quality attributes is shifting from code (HTML/CSS) analysis to mining webpages’visual representations based on image recognition techniques. In our paper, we describe a visual analysis tool which takes a WUI screenshot and produces structured and machine-readable representation (JSON) of the interface elements’ spatial allocation. The implementation is based on OpenCV (image recognition functions), dlib (trained detector for the elements’ classification), and Tesseract (label and content text recognition). The JSON representation is used to automatically calculate several metrics related to visual complexity, which is known to have major effect on user experience with UIs. We further describe a WUI measurement platform that allows integration of the currently dispersed sets of metrics from different providers and demonstrate the platform’s use with several remote services. We perform statistical analysis of the collected metrics in relation to complexity-related subjective evaluations obtained from 63 human subjects of various nationalities. Finally, we build predictive models for visual complexity and show that their accuracy can be improved by integrating the metrics from different sets. Regressions with the single index of visual complexity metric that we proposed had R2=0.460, while the best joint model with 4 metrics had R2=0.647.
J. Grigera, et al., ‘Automatic detection of usability smells in web applications’, International Journal of Human-Computer Studies, 97, 2017, pp. 129–148.
A. Miniukovich, A. De Angeli, ‘Computation of interface aesthetics’, In Proc. of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pp. 1163–1172. ACM, 2015.
T. Schmidt, C. Wolff, ‘The Influence of User Interface Attributes on Aesthetics’, i-com 17, no. 1, 2018, pp. 41–55.
R. Rosenholtz, Y. Li, L. Nakano, ‘Measuring visual clutter’, Journal of Vision, 7, 2, 2007, pp. 1–22.
A. Oulasvirta, ‘Can computers design interaction?’, In Proc. of the 8th ACM SIGCHI Symposium on Engineering Interactive Computing Systems, pp. 1–2. ACM, 2016.
M. Zen, J. Vanderdonckt, ‘Towards an evaluation of graphical user interfaces aesthetics based on metrics’, In IEEE Int Conf on Research Challenges in Information Science (RCIS), pp. 1–12, 2014.
A. Dingli, J. Mifsud, ‘Useful: A framework to mainstream web site usability through automated evaluation’, International Journal of Human Computer Interaction (IJHCI), 2(1), 10, 2011.
D. Cai, et al., ‘Extracting content structure for web pages based on visual representation’, In Asia-Pacific Web Conference, pp. 406–417, 2003.
R. Kumar, et al., ‘Bricolage: example-based retargeting for web design’, In Proc. of the ACM SIGCHI Conference on Human Factors in Computing Systems, pp. 2197–2206, 2011.
A. Sanoja, B. Gançarski, ‘Block-o-matic: A web page segmentation framework’, In Proc. IEEE Int. Conf. Multimedia Computing and Systems (ICMCS), pp. 595–600, 2014.
J. Pu, J. Liu, J. Wang, ‘A Vision-Based Approach for Deep Web Form Extraction’, In Advanced Multimedia and Ubiquitous Engineering, pp. 696–702, Springer, 2017.
J. Kong, et al., ‘Web Interface Interpretation Using Graph Grammars’, IEEE Trans. Syst. Man, Cybern. Part C (Applications Rev.), 42 (4), pp. 590–602, 2012.
K. Reinecke, et al., ‘Predicting users’ first impressions of website aesthetics with a quantification of perceived visual complexity and colorfulness’, In Proc. of the ACM SIGCHI Conference on Human Factors in Computing Systems, pp. 2049–2058, 2013.
P. Machado, et al., ‘Computerized measures of visual complexity’, Acta psychologica, 160, pp. 43–57, 2015.
E. Michailidou, S. Harper, S. Bechhofer, ‘Visual complexity and aesthetic perception of web pages’, In Proc. of the 26th annual ACM international conference on Design of communication, pp. 215–224, ACM, 2008.
S.E.S. Taba, et al., ‘An exploratory study on the relation between user interface complexity and the perceived quality’, In International Conference on Web Engineering (ICWE), pp. 370–379, Springer, Cham, 2014.
V. Chikhman, et al., ‘Complexity of images: Experimental and computational estimates compared,’ Perception, 41(6), pp. 631–647, 2012.
A. Miniukovich, A. De Angeli, ‘Quantification of interface visual complexity. In Proc. of the 2014 international working conference on advanced visual interfaces, pp. 153–160, ACM, 2014.
M. Bakaev, O. Razumnikova, ‘Opredeleine slozhnosti zadach dlya zritelno-prostranstvennoi pamyati i propustkoi spospobnosti cheloveka-operatora’, Upravlenie bol’shimi sistemami = Large-Scale Systems Control, 70, pp. 25–57, 2017. – In Russian.
A. Carballal, et al., ‘Distinguishing paintings from photographs by complexity estimates’, Neural Computing and Applications, 30, no. 6, pp. 1957–1969, 2018.
A. Oulasvirta, et al., ‘Aalto Interface Metrics (AIM): A Service and Codebase for Computational GUI Evaluation’, In The 31st Annual ACM Symposium on User Interface Software and Technology Adjunct Proceedings, pp. 16–19. ACM, 2018.
O. Wu, W. Hu, L. Shi, ‘Measuring the visual complexities of web pages’, ACM Transactions on the Web (TWEB), 7(1), p.1, 2013.
K. Alemerien, K. Magel, ‘GUIEvaluator: A Metric-tool for Evaluating the Complexity of Graphical User Interfaces’, In SEKE, pp. 13–18, 2014.
C. Stickel, M. Ebner, A. Holzinger, ‘The XAOS metric– understanding visual complexity as measure of usability’, In Symposium of the Austrian HCI and Usability Engineering Group, pp. 278-290, Springer, Berlin, Heidelberg, 2010.
M. Bakaev, S. Heil, V. Khvorostov, M. Gaedke, ‘HCI Vision for Automated Analysis and Mining of Web User Interfaces’, In International Conference on Web Engineering (ICWE), pp. 136–144. Springer, Cham, 2018.
S.T. Bow, Pattern recognition and image preprocessing. New York: Marcel Dekker, 2002.
M. Bakaev, M. Gaedke, S. Heil, ‘Kansei Engineering Experimental Research with University Websites’, TU Chemnitz Technical Report, CSR-16-01, 2016.
M. Bakaev, T. Mamysheva, M. Gaedke, ‘Current trends in automating usability evaluation of websites: Can you manage what you can’t measure?’ In Proc. IEEE 11th International Forum on Strategic Technology (IFOST), pp. 510–514, 2016.