Enhancement of 3D Point Cloud Contents Using 2D Image Super Resolution Network
Keywords:Point cloud, Super resolution, Deep learning network, 3D data
Media technology has been developed to give users a sense of immersion. Recent media using 3D spatial data, such as augmented reality and virtual reality, has attracted attention. A point cloud is a data format that consists of a number of points, and thus can express 3D media using coordinates and color information for each point. Since a point cloud has a larger capacity than 2D images, a technology to compress the point cloud is required, i.e., standardized in the international standard organization MPEG as a video-based point cloud compression (V-PCC). V-PCC decomposes 3D point cloud data into 2D patches along orthogonal directions, and those patches are placed into a 2D image sequence, and then compressed using existing 2D video codecs. However, data loss may occur while converting a 3D point cloud into a 2D image sequence and encoding this sequence using a legacy video codec. This data loss can cause deterioration in the quality of a reconstructed point cloud. This paper proposed a method of enhancing a reconstructed point cloud by applying a super resolution network to the 2D patch image sequence of a 3D point cloud.
“Ubiquitous AR to dominate focused VR by 2022,” https://techcrunch.com/2018/01/25/ubiquitous-ar-to-dominatefocused-vr-by-2022/, accessed: 2019.
“Creating 3D Content with Reality Composer,” https://developer.apple.com/documentation/realitykit/creating_3d_content_with_reality_composer, accessed: 2019.
Emin Zerman, Cagri Ozcinar, Pan Gao, and Aljosa Smolic, “Textured Mesh vs Coloured Point Cloud: A Subjective Study for Volumetric Video Compression”, 2020 Twelfth International Conference on Quality of Multimedia Experience (QoMEX).
“Text of ISO/IEC CD 23090-5 Video-based Point Cloud Compression,” ISO/IEC JTC1/SC29/WG11 MPEG2019/N18670, Gothenburg, Sweden, Oct. 2019.
H. Hoppe, T. DeRose, T. Duchamp, J. A. McDonald, W. Stuetzle, “Surface reconstruction from unorganized points,” Proc. SIGGRAPH, pp. 71–78, 1992.
Schwarz Sebastian, Miska M. Hannuksela, FakourSevom Vida, Sheikhi-Pour Nahid, “2d video coding of volumetric video data”, Picture Coding Symposium (PCS) 2018 – Proceedings, pp. 61–65, 2018.
S. Schwarz, et al., “Emerging mpeg standards for point cloud compression,” in IEEE Journal on Emerging and Selected Topics in Circuits and Systems, vol. 9, no. 1, pp. 133–148, Mar. 2019.
“High efficiency video coding test model, HM16.18+SCM8.7,” https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/tags/HM-16.18+SCM-8.7/, accessed: 2019.
“Image Super-Resolution Using Deep Convolutional Networks”, Chao Dong, Chen Change Loy, Kaiming He, Xiaoou Tang, ECCV, 2014.
“Deep Residual Learning for Image Recognition”, Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770–778.
“Accurate Image Super-Resolution Using Very Deep Convolutional Networks”, Jiwon Kim, Jung Kwon Lee and Kyoung Mu Lee, CVPR, 2016.
E. S. Jang et al., “Video-Based Point-Cloud-Compression Standard in MPEG: From Evidence Collection to Committee Draft [Standards in a Nutshell],” in IEEE Signal Processing Magazine, vol. 36, no. 3, pp. 118–123, May 2019.
“Super-resolution from a single image”, Daniel Glasner; Shai Bagon; Michal Irani, IEEE 12th International Conference on Computer Vision, 2009.
“Example-based super-resolution”, W.T. Freeman; T.R. Jones; E.C. Pasztor, IEEE Computer Graphics and Applications, Vol. 22, Issue: 2, Mar/Apr 2002.
“Image super-resolution as sparse representation of raw image patches”, Jianchao Yang; John Wright; Thomas Huang; Yi Ma, IEEE Conference on Computer Vision and Pattern Recognition, 2008.
S. A. Nene and S. K. Nayar, “A simple algorithm for nearest neighbor search in high dimensions,” in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 19, no. 9, pp. 989–1003, Sept. 1997.
H. Samet, “K-Nearest Neighbor Finding Using MaxNearestDist,” in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 30, no. 2, pp. 243–252, Feb. 2008.
K. C. K. Lee, B. Zheng and W. Lee, “Ranked Reverse Nearest Neighbor Search,” in IEEE Transactions on Knowledge and Data Engineering, vol. 20, no. 7, pp. 894–910, July 2008.
E. Agrell, T. Eriksson, A. Vardy and K. Zeger, “Closest point search in lattices,” in IEEE Transactions on Information Theory, vol. 48, no. 8, pp. 2201–2214, Aug. 2002.
“Deep Networks for Image Super-Resolution With Sparse Prior”, Zhaowen Wang, Ding Liu, Jianchao Yang, Wei Han, Thomas Huang, Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2015, pp. 370–378.
A Convolutional Neural Network Approach for Objective Video Quality Assessment, Patrick Le Callet, Member, IEEE, Christian Viard-Gaudin, and Dominique Barba, IEEE Transactions on Neural Networks, vol. 17, No. 5, September 2006
Deep Learning of Human Visual Sensitivity in Image Quality Assessment Framework, Jongyoo Kim, Sanghoon Lee, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 1676–1684.