Enhancement of 3D Point Cloud Contents Using 2D Image Super Resolution Network

Authors

DOI:

https://doi.org/10.13052/jwe1540-9589.21213

Keywords:

Point cloud, Super resolution, Deep learning network, 3D data

Abstract

Media technology has been developed to give users a sense of immersion. Recent media using 3D spatial data, such as augmented reality and virtual reality, has attracted attention. A point cloud is a data format that consists of a number of points, and thus can express 3D media using coordinates and color information for each point. Since a point cloud has a larger capacity than 2D images, a technology to compress the point cloud is required, i.e., standardized in the international standard organization MPEG as a video-based point cloud compression (V-PCC). V-PCC decomposes 3D point cloud data into 2D patches along orthogonal directions, and those patches are placed into a 2D image sequence, and then compressed using existing 2D video codecs. However, data loss may occur while converting a 3D point cloud into a 2D image sequence and encoding this sequence using a legacy video codec. This data loss can cause deterioration in the quality of a reconstructed point cloud. This paper proposed a method of enhancing a reconstructed point cloud by applying a super resolution network to the 2D patch image sequence of a 3D point cloud.

Downloads

Download data is not yet available.

Author Biographies

Seonghwan Park, Kyung Hee University, Korea

Seonghwan Park received B.S. in digital information engineering from Hankuk University of Foreign Studies, Yongin, South Korea, in 2013 and received M.S. in electronics engineering from Kyung Hee University, Yongin, South Korea, in 2015, where he is currently pursuing the Ph.D. degree in electronics engineering. His current research interests include MPEG systems, point cloud, super resolution and image processing

Junsik Kim, Kyung Hee University, Korea

Junsik Kim received the B.S. and M.S. degrees in electronics engineering from Kyung Hee University, Yongin, South Korea, in 2017 and 2019, respectively, where he is currently pursuing the Ph.D. degree in electronics engineering. His current research interests include point cloud compression, MPEG systems, digital broadcasting technologies, and image processing.

Yonghae Hwang, Kyung Hee University, Korea

Yonghae Hwang received B.S. in astronomy & space science and electronics engineering from Kyung Hee University, Yongin, South Korea, in 2020, where he is currently pursuing M.S. in electronics and information convergence engineering. His current research interests include MPEG 3DG, V-PCC, point cloud and video codec.

Doug Young Suh, Kyung Hee University, Korea

Doug Young Suh (Member, IEEE) received the B.S. degree in nuclear engineering from Seoul National University, Seoul, South Korea, in 1980, and the Ph.D. degree in electrical and computer engineering from the Georgia Institute of Technology, Atlanta, GA, USA, in 1990. In 1990, he joined the Korea Academy of Industry and Technology, and conducted research on HDTV until 1992. Since 1992, he has been a Professor with the College of Electronics and Information, Kyung Hee University, Seoul, South Korea. He has been a Korean Delegate for the ISO/IEC MPEG Forum since 1996. His research interests include networked video and computer games.

Kyuheon Kim, Kyung Hee University, Korea

Kyuheon Kim received the B.S. degree in electronic engineering from Hanyang University, Seoul, South Korea, in 1989, and the M.Phil. and Ph.D. degrees in electrical and electronic engineering from The University of Newcastle, Newcastle upon Tyne, U.K., in 1996. From 1996 to 1997, he was with Sheffield University, U.K., as a Research Fellow. From 1997 to 2006, he was with the Electronics and Telecommunications Research Institute, South Korea, as the Head of the Interactive Media Research Team, where he standardized and developed T-DMB specification, and conducted the Head of Korean delegates for MPEG standard body, from 2001 to 2005. Since 2006, he has conducted research at Kyung Hee University, Seoul. He has published numerous technical articles. His current research interests include interactive media processing, digital signal processing, and digital broadcasting technologies. He was a recipient of the Ministry Award from the Ministry of Information and Communication, in 2003 and the Prime Minister Award, in 2005.

References

“Ubiquitous AR to dominate focused VR by 2022,” https://techcrunch.com/2018/01/25/ubiquitous-ar-to-dominatefocused-vr-by-2022/, accessed: 2019.

“Creating 3D Content with Reality Composer,” https://developer.apple.com/documentation/realitykit/creating_3d_content_with_reality_composer, accessed: 2019.

Emin Zerman, Cagri Ozcinar, Pan Gao, and Aljosa Smolic, “Textured Mesh vs Coloured Point Cloud: A Subjective Study for Volumetric Video Compression”, 2020 Twelfth International Conference on Quality of Multimedia Experience (QoMEX).

“Text of ISO/IEC CD 23090-5 Video-based Point Cloud Compression,” ISO/IEC JTC1/SC29/WG11 MPEG2019/N18670, Gothenburg, Sweden, Oct. 2019.

H. Hoppe, T. DeRose, T. Duchamp, J. A. McDonald, W. Stuetzle, “Surface reconstruction from unorganized points,” Proc. SIGGRAPH, pp. 71–78, 1992.

Schwarz Sebastian, Miska M. Hannuksela, FakourSevom Vida, Sheikhi-Pour Nahid, “2d video coding of volumetric video data”, Picture Coding Symposium (PCS) 2018 – Proceedings, pp. 61–65, 2018.

S. Schwarz, et al., “Emerging mpeg standards for point cloud compression,” in IEEE Journal on Emerging and Selected Topics in Circuits and Systems, vol. 9, no. 1, pp. 133–148, Mar. 2019.

“High efficiency video coding test model, HM16.18+SCM8.7,” https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/tags/HM-16.18+SCM-8.7/, accessed: 2019.

“Image Super-Resolution Using Deep Convolutional Networks”, Chao Dong, Chen Change Loy, Kaiming He, Xiaoou Tang, ECCV, 2014.

“Deep Residual Learning for Image Recognition”, Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770–778.

“Accurate Image Super-Resolution Using Very Deep Convolutional Networks”, Jiwon Kim, Jung Kwon Lee and Kyoung Mu Lee, CVPR, 2016.

E. S. Jang et al., “Video-Based Point-Cloud-Compression Standard in MPEG: From Evidence Collection to Committee Draft [Standards in a Nutshell],” in IEEE Signal Processing Magazine, vol. 36, no. 3, pp. 118–123, May 2019.

“Super-resolution from a single image”, Daniel Glasner; Shai Bagon; Michal Irani, IEEE 12th International Conference on Computer Vision, 2009.

“Example-based super-resolution”, W.T. Freeman; T.R. Jones; E.C. Pasztor, IEEE Computer Graphics and Applications, Vol. 22, Issue: 2, Mar/Apr 2002.

“Image super-resolution as sparse representation of raw image patches”, Jianchao Yang; John Wright; Thomas Huang; Yi Ma, IEEE Conference on Computer Vision and Pattern Recognition, 2008.

S. A. Nene and S. K. Nayar, “A simple algorithm for nearest neighbor search in high dimensions,” in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 19, no. 9, pp. 989–1003, Sept. 1997.

H. Samet, “K-Nearest Neighbor Finding Using MaxNearestDist,” in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 30, no. 2, pp. 243–252, Feb. 2008.

K. C. K. Lee, B. Zheng and W. Lee, “Ranked Reverse Nearest Neighbor Search,” in IEEE Transactions on Knowledge and Data Engineering, vol. 20, no. 7, pp. 894–910, July 2008.

E. Agrell, T. Eriksson, A. Vardy and K. Zeger, “Closest point search in lattices,” in IEEE Transactions on Information Theory, vol. 48, no. 8, pp. 2201–2214, Aug. 2002.

“Deep Networks for Image Super-Resolution With Sparse Prior”, Zhaowen Wang, Ding Liu, Jianchao Yang, Wei Han, Thomas Huang, Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2015, pp. 370–378.

A Convolutional Neural Network Approach for Objective Video Quality Assessment, Patrick Le Callet, Member, IEEE, Christian Viard-Gaudin, and Dominique Barba, IEEE Transactions on Neural Networks, vol. 17, No. 5, September 2006

Deep Learning of Human Visual Sensitivity in Image Quality Assessment Framework, Jongyoo Kim, Sanghoon Lee, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 1676–1684.

Downloads

Published

2022-01-17

How to Cite

Park, S. ., Kim, J. ., Hwang, Y. ., Suh, D. Y. ., & Kim, K. . (2022). Enhancement of 3D Point Cloud Contents Using 2D Image Super Resolution Network. Journal of Web Engineering, 21(02), 425–442. https://doi.org/10.13052/jwe1540-9589.21213

Issue

2022: Vol 21 Iss 2

Section

SPECIAL ISSUE ON Future Multimedia Contents and Technology on Web in the 5G Era