Adaptive Sampling for Real-time Neural View Synthesis on the Web with Reinforcement Learning
DOI:
https://doi.org/10.13052/jwe1540-9589.2521Keywords:
Neural Radiance Field, Multi Resolution Hash Encoding, Reinforcement Learning, proximal policy optimizationAbstract
The proliferation of immersive 3D web applications, from e-commerce product viewers to virtual real estate tours, has created a critical need for high-quality, real-time rendering directly within the browser. Neural radiance fields (NeRF) offer unprecedented photorealism but are hamstrung by immense computational demands, making their deployment on resource-constrained web platforms a significant web engineering challenge. The core bottleneck is NeRF’s reliance on dense point sampling for volume rendering. This paper introduces a novel framework that directly tackles this challenge through a pioneering adaptive sampling technique powered by reinforcement learning.
We name this framework PPO-NeRF. It integrates the rapid training capabilities of Instant-NGP’s hash encoding with an agent trained via proximal policy optimization (PPO). This agent learns to adaptively predict the minimal set of crucial sample points along each camera ray, dynamically pruning computationally redundant samples to optimize rendering specifically for web-based, real-time scenarios. Experimental results demonstrate that PPO-NeRF significantly lowers the barrier to web deployment. Compared to the original NeRF, it reduces training time by approximately 73.63%, enabling faster content iteration for web developers. More critically, our adaptive sampling slashes rendering time by approximately 44.7% and VRAM usage by approximately 29.9%, while maintaining comparable visual fidelity. These gains directly translate to faster load times, smoother user interaction, and broader device compatibility.
In conclusion, PPO-NeRF provides a practical solution to NeRF’s long-standing performance bottlenecks, establishing a viable pathway for deploying high-fidelity, interactive 3D experiences at scale across the modern web.
Downloads
References
Brebin, R. A., et al. “Volume Rendering.” Seminal Graphics: Pioneering Efforts that Shaped the Field (1998): 363–372.
Tewari, Ashish, et al. “Advances in Neural Rendering.” Computer Graphics Forum 41.2 (2022): 703–735.
Mildenhall, Ben, et al. “Nerf: Representing Scenes as Neural Radiance Fields for View Synthesis.” Communications of the ACM 65.1 (2021): 99–106.
Müller, Thomas, et al. “Instant Neural Graphics Primitives with a Multiresolution Hash Encoding.” ACM Transactions on Graphics (TOG) 41.4 (2022): 1–15.
Bae, O., and C. P. Hong. “An Effective Scheme to Accelerate NeRF for Web Applications Using Hash-based Caching and Precomputed Features.” Journal of Web Engineering 23.7 (2024): 1041–1056.
Schulman, John, et al. “Proximal Policy Optimization Algorithms.” arXiv preprint arXiv:1707.06347 (2017).
Tsai, R. “A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses”. IEEE Journal on Robotics and Automation, 3(4), (2003): 323–344.
Fridovich-Keil, Sara, et al. “Plenoxels: Radiance Fields Without Neural Networks.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2022): 5501–5510.
Sloan, Peter-Pike, et al. “Precomputed Radiance Transfer for Real-Time Rendering in Dynamic, Low-Frequency Lighting Environments.” Seminal Graphics Papers: Pushing the Boundaries, Volume 2 (2023): 339–348.
Yu, Alex, et al. “Plenoctrees for Real-Time Rendering of Neural Radiance Fields.” Proceedings of the IEEE/CVF International Conference on Computer Vision (2021): 5752–5761.
Garbin, Stephan J., et al. “FastNeRF: High-Fidelity Neural Rendering at 200fps.” Proceedings of the IEEE/CVF International Conference on Computer Vision (2021): 14346–14355.
Piala, Matthew, and Roman Clark. “TermiNeRF: Ray Termination Prediction for Efficient Neural Rendering.” 2021 International Conference on 3D Vision (3DV) (2021): 1106–1114.
Lin, Cheng, et al. “Modeling 3D Shapes by Reinforcement Learning.” European Conference on Computer Vision (2020): 545–561.
Van Hasselt, Hado, et al. “Deep Reinforcement Learning with Double Q-Learning.” Proceedings of the AAAI Conference on Artificial Intelligence 30.1 (2016).
Ross, Stéphane, et al. “A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning.” Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics (2011): 627–635.
Hester, Todd, et al. “Deep Q-Learning from Demonstrations.” Proceedings of the AAAI Conference on Artificial Intelligence 32.1 (2018).
