Refresh Rate and Graphical Benchmarks for Mobile VR Application Development
Keywords:Virtual Reality, Usability, Smart Mobile Devices, Lag
Virtual reality (VR) technology is quickly becoming more accessible to the general public due to the availability and capabilities of modern smartphone devices. However, such mobile devices are not as powerful as high-end desktop systems where VR is mostly established. Running demanding VR apps leads to performance issues such as lag, excessive heat, and fast battery drainage. To avoid these problems, software factors must be optimised. The user evaluation (N=51)(N=51) involved presenting multiple VR scenes (with varying frame rates), requiring participants to judge which scenes felt smooth; the results indicate that anything below 50 FPS was tolerable at best and nauseating at worst. To also measure the performance impact of various software settings, benchmarks were conducted on different smartphones. The results highlight the effects when varying the number of displayed on-screen objects, as well as outlining which settings should be avoid when specifically targeting mobile VR platforms. Heat and battery life were found to be non-issues at recommended performance levels. The proposed work established valuable guidelines which can be helpful for real time applications in development time reduction and complexity simplification from graphical and refresh rate optimization perspectives.
A. Nayyar, B. Mahapatra, D. Le, and G. Suseendran, “Virtual reality (VR) & augmented reality (AR) technologies for tourism and hospitality industry,” International Journal of Engineering & Technology, vol. 7, no. 2.21, pp. 156–160, 2018.
G. Burdea and P. Coiffet, “Virtual reality technology. John Wiley & Sons,” Inc-1993, 1994.
S. LaValle, Virtual Reality/University of Illinois. Cambridge University Press., 2017.
R. A. Kramer, A. Y. Taikar, W. Paweenbampen, H. Alorayj, and S. T. Godambe, “The challenges and advances in mixed reality video technology,”
M. Alcañiz, J. A. Lozano, and B. Rey, “Technological background of vr,” Cybertherapy, no. 10.3233/978-1-60750-943-1-199, pp. 199–214, 2004.
M. El Beheiry, S. Doutreligne, C. Caporal, C. Ostertag, M. Dahan, and J.-B. Masson, “Virtual reality: Beyond visualization,” Journal of Molecular Biology, vol. 431, no. 10.1016/j.jmb.2019.01.033, pp. 1315–1321, 2019.
Z. Tan, Y. Li, Q. Li, Z. Zhang, Z. Li, and S. Lu, “Supporting mobile vr in lte networks: How close are we?,” Proc. ACM Meas. Anal. Comput. Syst., vol. 2, apr 2018.
G. Arrighi, Z. S. See, and D. Jones, “Victoria theatre virtual reality: A digital heritage case study and user experience design,” Digital Applications in Archaeology and Cultural Heritage, vol. 21, no. 10.1016/j.daach.2021.e00176, p. 176, 2021.
M. Nagappan and E. Shihab, “Future trends in software engineering research for mobile apps,” in 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER), vol. 5, pp. 21–32, March 2016.
R. Alturki and V. Gay, “Augmented and virtual reality in mobile fitness applications: a survey,” Applications of Intelligent Technologies in Healthcare, pp. 67–75, 2019.
Y. Hu, J. Zhu, W. Li, Y. Zhang, Q. Zhu, H. Qi, H. Zhang, Z. Cao, W. Yang, and P. Zhang, “Construction and optimization of three-dimensional disaster scenes within mobile virtual reality,” ISPRS International Journal of Geo-Information, vol. 7, no. 6, 2018.
L. Gugerty and G. Olson, “Debugging by skilled and novice programmers,” in Proceedings of the SIGCHI conference on human factors in computing systems, pp. 171–174, 1986.
W. L. Johnson, “Understanding and debugging novice programs,” Artificial intelligence, vol. 42, no. 1, pp. 51–97, 1990.
J. Martín-Gutiérrez, C. E. Mora, B. Añorbe-Díaz, and A. González-Marrero, “Virtual technologies trends in education,” EURASIA Journal of Mathematics Science and Technology Education, vol. 13, no. 2, pp. 469–486, 2017.
D. Cearley and B. Burke, “Top 10 strategic technology trends for 2019.” https://www.gartner.com/doc/3891569. Accessed: March 2020.
“Buy PS VR.” https://www.playstation.com/en-nz/explore/playstation-vr/buy-now. Accessed: March 2020.
“Oculus Rift.” https://www.oculus.com/rift. Accessed: March 2020.
“Buy VIVE hardware.” https://www.vive.com/nz/product. Accessed: March 2020.
“HoloLens 2 pricing and options.” https://www.microsoft.com/hololens/buy. Accessed: March 2020.
L. Avila and M. Bailey, “Virtual reality for the masses.,” IEEE computer graphics and applications, vol. 34, no. 5, pp. 103–104, 2014.
J. M. Zelle and C. Figura, “Simple, low-cost stereographics: VR for everyone,” in ACM SIGCSE Bulletin, vol. 36, pp. 348–352, ACM, 2004.
A. Amer and P. Peralez, “Affordable altered perspectives: Making augmented and virtual reality technology accessible,” in IEEE Global Humanitarian Technology Conference (GHTC 2014), pp. 603–608, IEEE, 2014.
W. Powell, V. Powell, P. Brown, M. Cook, and J. Uddin, “Getting around in Google Cardboard–exploring navigation preferences with low-cost mobile VR,” in 2016 IEEE 2nd Workshop on Everyday Virtual Reality, pp. 5–8, 2016.
B. S. Santos, P. Dias, A. Pimentel, J.-W. Baggerman, C. Ferreira, S. Silva, and J. Madeira, “Head-mounted display versus desktop for 3D navigation in virtual reality: a user study,” Multimedia Tools and Applications, vol. 41, no. 1, p. 161, 2009.
J. R. Smith, B. Snapp, S. Madar, J. R. Brown, J. Fowler, E. Andersen, C. D. Porter, and C. Orban, “A smartphone-based virtual reality plotting system for stem education,” PRIMUS, no. just-accepted, p. 1, 2021.
A. Agrawal, “Putting VR/AR to work,” IEEE Computer Graphics and Applications, vol. 38, no. 1, pp. 115–118, 2018.
T. Wasserman, “Software engineering issues for mobile application development,” FoSER 2010, 2010.
X. Ma, Z. Deng, M. Dong, and L. Zhong, “Characterizing the performance and power consumption of 3D mobile games,” Computer, vol. 46, no. 4, pp. 76–82, 2012.
Z. Luo, H. Cho, X. Luo, and K.-i. Cho, “System thermal analysis for mobile phone,” Applied Thermal Engineering, vol. 28, no. 14-15, pp. 1889–1895, 2008.
A. Prakash, H. Amrouch, M. Shafique, T. Mitra, and J. Henkel, “Improving mobile gaming performance through cooperative CPU-GPU thermal management,” in Proceedings of the 53rd Annual Design Automation Conference, p. 47, ACM, 2016.
Z. Lai, Y. C. Hu, Y. Cui, L. Sun, N. Dai, and H. Lee, “Furion: Engineering high-quality immersive virtual reality on today’s mobile devices,” IEEE Transactions on Mobile Computing, pp. 1–1, 2019.
S. He, Y. Liu, and H. Zhou, “Optimizing smartphone power consumption through dynamic resolution scaling,” in Proceedings of the 21st Annual International Conference on Mobile Computing and Networking, MobiCom ’15, (New York, NY, USA), pp. 27–39, Association for Computing Machinery, 2015.
X. Chen, T. Tan, G. Cao, and T. F. L. Porta, “Context-aware and energy-aware video streaming on smartphones,” IEEE Transactions on Mobile Computing, vol. 21, no. 3, pp. 862–877, 2022.
T. A. Funkhouser and C. H. Séquin, “Adaptive display algorithm for interactive frame rates during visualization of complex virtual environments,” in Siggraph, vol. 93, pp. 247–254, 1993.
M. M. Wloka, “Lag in multiprocessor virtual reality,” Presence: Teleoperators & Virtual Environments, vol. 4, no. 1, pp. 50–63, 1995.
S. Sharples, S. Cobb, A. Moody, and J. R. Wilson, “Virtual reality induced symptoms and effects (VRISE): Comparison of head mounted display (HMD), desktop and projection display systems,” Displays, vol. 29, no. 2, pp. 58–69, 2008.
H. Khalid, E. Shihab, M. Nagappan, and A. E. Hassan, “What do mobile app users complain about?,” IEEE Software, vol. 32, no. 3, pp. 70–77, 2015.
B. Fu, J. Lin, L. Li, C. Faloutsos, J. Hong, and N. Sadeh, “Why people hate your app: Making sense of user feedback in a mobile app store,” in Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 1276–1284, ACM, 2013.
“Optimisation for VR in Unity.” https://unity3d.com/learn/tutorials/topics/virtual-reality/optimisation-vr-unity. Accessed: March 2020.
“VR performance best practices.” https://developers.google.com/vr/develop/best-practices/perf-best-practices. Accessed: March 2020.
“CPU usage profiler.” https://docs.unity3d.com/Manual/ProfilerCPU.html. Accessed: March 2020.
“CPU profiling.” https://docs.unrealengine.com/en-US/Engine/Performance/CPU. Accessed: March 2020.
J. Ruan and D. Xie, “Networked vr: State of the art, solutions, and challenges,” Electronics, vol. 10, no. 2, 2021.
K. Foo, “Understanding the VR potential of mobile devices: Literature review and statement of research intent,” 2019.
Oculus, “Guidelines for vr performance optimization,” 2019.
M. S. Elbamby, C. Perfecto, M. Bennis, and K. Doppler, “Toward low-latency and ultra-reliable virtual reality,” IEEE Network, vol. 32, no. 2, pp. 78–84, 2018.
J. Zhao, R. S. Allison, M. Vinnikov, and S. Jennings, “Estimating the motion-to-photon latency in head mounted displays,” in 2017 IEEE Virtual Reality, pp. 313–314, 2017.
S.-W. Choi, S. Lee, M.-W. Seo, and S.-J. Kang, “Time sequential motion-to-photon latency measurement system for virtual reality head-mounted displays,” Electronics, vol. 7, no. 9, p. 171, 2018.
M. Meehan, S. Razzaque, M. C. Whitton, and F. P. Brooks, “Effect of latency on presence in stressful virtual environments,” in IEEE Virtual Reality, pp. 141–148, 2003.
B. Conner and L. Holden, “Providing a low latency user experience in a high latency application.,” in SI3D, pp. 45–48, 1997.
I. Tošić, D. Hoffman, and N. Balram, “Effect of latency on simulator sickness in smartphone virtual reality,” Journal of the Society for Information Display, vol. 29, no. 7, pp. 561–572, 2021.
L. Rebenitsch and C. Owen, “Review on cybersickness in applications and visual displays,” Virtual Reality, vol. 20, 06 2016.
G. Reina, H. Childs, K. Matković, K. Bühler, M. Waldner, D. Pugmire, B. Kozlíková, T. Ropinski, P. Ljung, T. Itoh, E. Gröller, and M. Krone, “The moving target of visualization software for an increasingly complex world,” Computers and Graphics, vol. 87, pp. 12–29, 2020.
F. Salimi and F. Salimi, “Chapter 5 – modeling and simulation: The essential tools to manage the complexities,” in A Systems Approach to Managing the Complexities of Process Industries (F. Salimi and F. Salimi, eds.), pp. 279–407, Elsevier, 2018.
D. Weidlich, L. Cser, T. Polzin, D. Cristiano, and H. Zickner, “Virtual reality approaches for immersive design,” CIRP Annals, vol. 56, no. 1, pp. 139–142, 2007.
K. Debattista, K. Bugeja, S. Spina, T. Bashford-Rogers, and V. Hulusic, “Frame rate vs resolution: A subjective evaluation of spatiotemporal perceived quality under varying computational budgets,” Computer Graphics Forum, vol. 37, 09 2017.
A. Mackin, F. Zhang, and D. R. Bull, “A study of high frame rate video formats,” IEEE Transactions on Multimedia, vol. 21, no. 6, pp. 1499–1512, 2019.
K. Sekar, “Power and thermal challenges in mobile devices,” in Proceedings of the 19th annual international conference on Mobile computing & networking, pp. 363–368, 2013.
Q. Xie, J. Kim, Y. Wang, D. Shin, N. Chang, and M. Pedram, “Dynamic thermal management in mobile devices considering the thermal coupling between battery and application processor,” in 2013 IEEE/ACM International Conference on Computer-Aided Design (ICCAD), pp. 242–247, IEEE, 2013.
M. Pedram and S. Nazarian, “Thermal modeling, analysis, and management in VLSI circuits: Principles and methods,” Proceedings of the IEEE, vol. 94, no. 8, pp. 1487–1501, 2006.
S. Wen, “Cell balancing buys extra run time and battery life,” Analog Applications Journal, vol. 1, 2009.
W. Wang, V. Reddy, A. T. Krishnan, R. Vattikonda, S. Krishnan, and Y. Cao, “Compact modeling and simulation of circuit reliability for 65-nm CMOS technology,” IEEE Transactions on Device and Materials Reliability, vol. 7, no. 4, pp. 509–517, 2007.
K. Y. Wong, “Cell phones as mobile computing devices,” IT professional, vol. 12, no. 3, pp. 40–45, 2010.
Y. G. Kim, J. Kong, and S. W. Chung, “A survey on recent os-level energy management techniques for mobile processing units,” IEEE Transactions on Parallel and Distributed Systems, vol. 29, no. 10, pp. 2388–2401, 2018.
M. Xia, W. He, X. Liu, and J. Liu, “Why application errors drain battery easily?: a study of memory leaks in smartphone apps,” in Proceedings of the Workshop on Power-Aware Computing and Systems, p. 2, ACM, 2013.
R. Mittal, A. Kansal, and R. Chandra, “Empowering developers to estimate app energy consumption,” in Proceedings of the 18th annual international conference on Mobile computing and networking, pp. 317–328, ACM, 2012.
A. Banerjee, L. K. Chong, S. Chattopadhyay, and A. Roychoudhury, “Detecting energy bugs and hotspots in mobile apps,” in International Symposium on Foundations of Software Engineering, pp. 588–598, 2014.
K.-T. Cheng and Y.-C. Wang, “Using mobile GPU for general-purpose computing–a case study of face recognition on smartphones,” in International Symposium on VLSI Design, Automation and Test, pp. 1–4, 2011.
R. S. Pressman, Software engineering: a practitioner’s approach. No. 007301933X, Palgrave macmillan, 2005.
Y. Hu, J. Zhu, W. Li, Y. Zhang, Q. Zhu, H. Qi, H. Zhang, Z. Cao, W. Yang, and P. Zhang, “Construction and optimization of three-dimensional disaster scenes within mobile virtual reality,” ISPRS International Journal of Geo-Information, vol. 7, no. 6, p. 215, 2018.
P. M. Isaacs, “Interactively reducing polygon count in three-dimensional graphic objects,” Apr. 13 1999. US Patent 5,894,308.
G. Schaufler and W. Stürzlinger, “A three dimensional image cache for virtual reality,” in Computer Graphics Forum, vol. 15, pp. 227–235, Wiley Online Library, 1996.
R. W. Latham, “If VR is so great, why are VR entertainment systems so poor?,” ACM SIGGRAPH Computer Graphics, vol. 28, no. 2, pp. 113–114, 1994.
Y.-J. Tsai, Y.-X. Wang, and M. Ouhyoung, “Affordable system for measuring motion-to-photon latency of virtual reality in mobile devices,” in SIGGRAPH Asia 2017 Posters, pp. 1–2, 2017.
R. Munro and J. Morgan, “Delay analyser.” https://csfieldguide.org.nz/en/interactives/delay-analyser. Accessed: March 2020.
M. Cinelli, A. Patla, and B. Stuart, “Involvement of the head and trunk during gaze reorientation during standing and treadmill walking,” Exp Brain Res, vol. 181, no. 1, pp. 183–91, 2007.
M. H. Draper, E. S. Viirre, T. A. Furness, and V. J. Gawron, “Effects of image scale and system time delay on simulator sickness within head-coupled virtual environments,” Human factors, vol. 43, no. 1, pp. 129–146, 2001.
J. Treleaven, J. Battershill, D. Cole, C. Fadelli, S. Freestone, K. Lang, and H. Sarig-Bahat, “Simulator sickness incidence and susceptibility during neck motion-controlled virtual reality tasks,” Virtual Reality, vol. 19, no. 3-4, pp. 267–275, 2015.