An Enhanced Video Inpainting Technique with Grey Wolf Optimization for Object Removal Application
DOI:
https://doi.org/10.13052/jmm1550-4646.1835Keywords:
Inpainting, curvature, structure tensor, Grey Wolf Optimization, Sum of Absolute Difference, PSNR, SSIMAbstract
Video inpainting is the most trending research topic from the last decade. Video inpainting is the process of restoring the damaged parts of the vintage video or the filling of the regions by removing the unwanted objects with sophisticated techniques. The video inpainting is achieved by dividing the video into frames and the motion of the moving objects in the frames are tracked by applying the motion tracking method. The existing inpainting method proposed by the Criminisi, neglected the local similarities in the images so it suffered from dropping effect in the priority computation. This paper proposed a new priority computation method by introducing gradient operation with the addition of curvature in the data term and local structure measurement function with structure tensor theory as an additional term. Later, the patch matching is achieved with the Sum of Absolute Difference (SAD) distance method. Further, the optimal patch is selected by applying the Grey Wolf Optimization (GWO) algorithm. The efficiency of the proposed video inpainting technique is evaluated with the performance metrics, viz., Peak Signal to Noise Ratio (PSNR), Structural Similarity (SSIM), and Edge Similarity (ESIM) executed in MATLAB. The PSNR and SSIM of the proposed method for Fontaine_chatelet video is improved by 18.9% and 4.19% than existing method. The proposed method is compared with other existing methods also and it outperformed the existing methods.
Downloads
References
G. Sridevi, S.S. Kumar, “Image inpainting and enhancement using fractional order variational model”, Defence Sci. J. 67(3), 308–315 (2017).
G. Sridevi, S.S. Kumar, “Image inpainting based on fractional-order nonlinear diffusion for image reconstruction”, Circuits, Systems, and Signal Processing (2019): 1–16.
G. Sridevi, and S. Srinivas Kumar. “p-Laplace Variational Image Inpainting Model Using Riesz Fractional Differential Filter.” International Journal of Electrical and Computer Engineering 7.2 (2017): 850.
Janardhana Rao, B., Y. Chakrapani, and S. Srinivas Kumar. “Image Inpainting Method with Improved Patch Priority and Patch Selection.” IETE Journal of Education 59.1 (2018): 26–34.
A. Criminisi, P. Perez, K. Toyama, “Region filling and object removal by exemplar-based image inpainting”, IEEE Trans. Image Process. 13(9), 1200–1212 (2004).
Lee, Jino, Dong-Kyu Lee, and Rae-Hong Park. “Robust exemplar-based inpainting algorithm using region segmentation.” IEEE Transactions on Consumer Electronics 58, no. 2 (2012): 553–561.
Yu, Jiahui, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S. Huang. “Generative image inpainting with contextual attention.” In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5505–5514. 2018.
Y. Matsushita, E. Ofek, W. Ge, X. Tang, and H.-Y. Shum, “Full-frame video stabilization with motion inpainting,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 28, no. 7, pp. 1150–1163, Jul. 2006.
K. A. Patwardhan, G. Sapiro, and M. Bertalmio, “Video inpainting under constrained camera motion,” IEEE Trans. Image Process., vol. 16, no. 2, pp. 545–553, Feb. 2007.
T. K. Shih, N. C. Tang, and J.-N. Hwang, “Exemplar-based video inpainting without ghost shadow artifacts by maintaining temporal continuity,” IEEE Trans. Circuits Syst. Video Technol., vol. 19, no. 3, pp. 347–360, Mar. 2009.
T. K. Shih, N. C. Tan, J. C. Tsai, and H.-Y. Zhong, “Video falsifying by motion interpolation and inpainting,” in Proc. IEEE Conf. Comput.Vis. Pattern Recognit., Jun. 2008, pp. 1–8.
M. Granados, J. Tompkin, K. I. Kim, J. Kautz, and C. Theobalt, “Background inpainting for videos with dynamic objects and a free moving camera,” in Proc. Eur. Conf. Comput. Vis., 2012, pp. 682–695.
O. Whyte, J. Sivic, and A. Zisserman, “Get out of my picture! Internet based inpainting,” in Proc. Brit. Mach. Vis. Conf., 2009, pp. 1–11.
X. Chen, Y. Shen, and Y. H. Yang, “Background estimation using graph cuts and inpainting,” in Proc. Graph. Inter., 2010, pp. 97–103.
A. Newson, A. Almansa, M. Fradet, Y. Gousseau, and P. Pérez, “Video inpainting of complex scenes,” SIAM J. Imag. Sci., vol. 7, no. 4, pp. 1993–2019, 2014.
C. Barnes, E. Shechtman, A. Finkelstein, and D. B. Goldman, “Patch Match: A randomized correspondence algorithm for structural image editing,” ACM Trans. Graph., vol. 28, no. 3, pp. 24:1–24:11, Jul. 2009.
M. Ebdelli, O. Le Meur, and C. Guillemot, “Video inpainting with short term windows: application to object removal and error concealment,” IEEE Trans. Image Processing, vol. 24, no. 10, pp. 3034–3047, May 2015.
Huang, J.B., Kang, S.B., Ahuja, N. and Kopf, J., 2016. Temporally coherent completion of dynamic video. ACM Transactions on Graphics (TOG), 35(6), pp. 1–11.
Le, Thuc Trinh, et al. “Object removal from complex videos using a few annotations”, Computational Visual Media 5.3 (2019): 267–291.
M. Okabe, K. Noda, Y. Dobashi and K. Anjyo, “Interactive video completion,” IEEE computer graphics and applications, 40(1), pp. 127–139, 2019.
C. Wang, X. Chen, S. Min, Z. J. Zha, and J. Wang, “Structure-Guided Deep Video Inpainting,” IEEE Transactions on Circuits and Systems for Video Technology, 2020.
B Janardhana Rao, Y Chakrapani, S Srinivas Kumar, Hybridized Cuckoo Search with Multi-Verse Optimization-Based Patch Matching and Deep Learning Concept for Enhancing Video Inpainting, The Computer Journal, 2021; bxab067, https://doi.org/10.1093/comjnl/bxab067.
J. Wang, K. Lu, D. Pan, N He, and B. Bao, “Robust object removal with an exemplar-based image inpainting approach,” Neurocomputing, pp. 150–155, 2014.
Y. Liu, C. J. Liu, H. L. Zou, S.S Zhou, Q. Shen, and T.T. Chen, “A novel exemplar-based image inpainting algorithm,” In 2015 International Conference on Intelligent Networking and Collaborative Systems, IEEE September 2015, pp. 86–90.
Xiang, C., Cao, Y., Duan, P., & Shi, L. (2014, August). An improved exemplar-based image inpainting algorithm. In 2014 9th International Conference on Computer Science & Education (pp. 770–775). IEEE.
J. Patel and T. K. Sarode, “Exemplar based image inpainting with reduced search region,” International Journal of Computer Applications, 92(12), 2014.
L. J. Deng, T. Z. Huang and X. L. Zhao, “Exemplar-based image inpainting using a modified priority definition,” PloS one, 10(10), e0141199, 2015.
S. Mirjalili, S. M. Mirjalili, and A. Lewis, “Grey Wolf Optimizer,” Adv Eng Softw, 69, pp. 46–61, 2014.
Khwaja Muinuddin Chisti Mohammed, Srinivas Kumar S., Prasad Gandikota, “Segmenting the Defective Leather Texture Image Using Modified Region Growing With GWO Technique,” Journal of Adv Research in Dynamical & Control Systems, Vol. 10, no. 04, Oct. 4, 2018.
N. Xu, L. Yang, Y. Fan, J. Yang, D. Yue, Y. Liang, and T. Huang, “Youtube-vos: Sequence-to-sequence video object segmentation,” In Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 585–601.
Perazzi, Federico, Jordi Pont-Tuset, Brian McWilliams, Luc Van Gool, Markus Gross, and Alexander Sorkine-Hornung. “A benchmark dataset and evaluation methodology for video object segmentation.” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 724–732. 2016.
Odugu VK, C VN, K SP. An efficient VLSI architecture of 2-D finite impulse response filter using enhanced approximate compressor circuits. Int J Circ Theor Appl. 2021;1–16. doi:10.1002/cta.3114
