Semantic Segmentation on FDFD-generated Wideband Radar Images of Potential Shooters

Authors

  • Mahshid Asri Department of Electrical Engineering Northeastern University, Boston, MA 02115, USA
  • Ann Morgenthaler Department of Electrical Engineering Northeastern University, Boston, MA 02115, USA
  • Carey M. Rappaport Department of Electrical Engineering Northeastern University, Boston, MA 02115, USA

DOI:

https://doi.org/10.13052/2025.ACES.J.400101

Keywords:

Concealed object, deep learning, millimeter-wave radar, object detection, semantic segmentation, U-Net

Abstract

This paper presents a deep learning model for fast and accurate radar detection and pixel-level localization of large concealed metallic weapons on pedestrians walking along a sidewalk. The considered radar is stationary, with a multi-beam antenna operating at 30 GHz with 6 GHz bandwidth. A large modeled data set has been generated by running 2155 2D-FDFD simulations of torso cross sections of persons walking toward the radar in various scenarios.

Downloads

Download data is not yet available.

Author Biographies

Mahshid Asri, Department of Electrical Engineering Northeastern University, Boston, MA 02115, USA

Mahshid Asri received the B.S. in electrical engineering from Iran University of Science and Technology, Tehran, Iran, in 2017 and the MS and Ph.D. in electrical engineering from Northeastern University, Boston, MA, USA, in 2020 and 2023, respectively. She is currently working as a postdoctoral research associate at Northeastern University, Boston, MA, USA. Her work is focused on radar image processing and object characterization, and applying deep learning techniques to radar images for anomaly detection. She completed an internship at Metalenz in 2022.

Ann Morgenthaler, Department of Electrical Engineering Northeastern University, Boston, MA 02115, USA

Ann Morgenthaler received the B.A. in physics from Harvard University, Cambridge, MA, and the E.E. degree in electrical engineering and computer science from Massachusetts Institute of Technology (MIT), Cambridge. She is an author of the textbook “Electromagnetic Waves” with David Staelin and Jin Au Kong. Her research interests include fast forward modeling techniques for ground-penetrating radar scattering from realistic objects in lossy media under rough ground surfaces.

Carey M. Rappaport, Department of Electrical Engineering Northeastern University, Boston, MA 02115, USA

Carey M. Rappaport(Fellow, IEEE) received the SB degree in mathematics and the SB, SM, and EE degrees in electrical engineering from the Massachusetts Institute of Technology (MIT), Cambridge, MA, USA, in 1982, and the Ph.D. degree in electrical engineering from MIT, in June 1987.

He worked as a Teaching and Research Assistant with MIT from 1981 to 1987, and during the sum- mers at COMSAT Labs, Clarksburg, MD, USA, and The Aerospace Corporation, El Segundo, CA, USA. He joined the Faculty of Northeastern University, Boston, MA, USA, in 1987. He has been a Professor of electrical and computer engineering since July 2000. In 2011, he was appointed as the College of Engineering Distinguished Professor. During fall 1995, he was a Visiting Professor of electrical engineering with the Electromagnetics Institute of the Technical University of Denmark, Lyngby, Denmark, as a part of the W. Fulbright International Scholar Program. During the second half of 2005, he was a Visiting Research Scientist with the Commonwealth Scientific Industrial and Research Organization (CSIRO) in Epping, Australia. He has consulted for CACI, Alion Science and Technology, Inc., Geo-Centers, Inc., PPG, Inc., and several municipalities on wave propagation and modeling, and microwave heating and safety. He was PI of an ARO-sponsored Multi- disciplinary University Research Initiative on Humanitarian Demining, Co-PI of the NSF-sponsored Engineering Research Center for Subsurface Sensing and Imaging Systems (CenSSIS), and Co-Principal Investigator and Deputy Director of the DHS-Sponsored Awareness and Localization of Explosive Related Threats (ALERT) Center of Excellence. He has authored over 430 technical journal and conference paper in the areas of microwave antenna design, electromagnetic wave propagation and scattering computation, and bioelectromagnetics. He holds two reflector antenna patents, two biomedical device patents, and four subsurface sensing device patents.

References

Z. Guo, Y. Huang, X. Hu, H. Wei, and B. Zhao, “A survey on deep learning based approaches for scene understanding in autonomous driving,” Electronics, vol. 10, no. 4, p. 471, 2021.

K. Muhammad, T. Hussain, H. Ullah, J. Del Ser, M. Rezaei, N. Kumar, M. Hijji, P. Bellavista, and V. H. C. de Albuquerque, “Vision-based semantic segmentation in scene understanding for autonomous driving: Recent achievements, challenges, and outlooks,” IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 12, pp. 22694-22715, 2022.

Q. M. Rahman, P. Corke, and F. Dayoub, “Run-time monitoring of machine learning for robotic perception: A survey of emerging trends,” IEEE Access, vol. 9, pp. 20067-20075, 2021.

X. Cao, S. Gao, L. Chen, and Y. Wang, “Ship recognition method combined with image segmentation and deep learning feature extraction in video surveillance,” Multimedia Tools and Applications, vol. 79, pp. 9177-9192, 2020.

Z. Wang, L. Wei, L. Wang, Y. Gao, W. Chen, and D. Shen, “Hierarchical vertex regression-based segmentation of head and neck CT images for radiotherapy planning,” IEEE Transactions on Image Processing, vol. 27, no. 2, pp. 923-937, 2017.

X. Zhu, H.-I. Suk, S.-W. Lee, and D. Shen, “Subspace regularized sparse multitask learning for multiclass neurodegenerative disease identification,” IEEE Transactions on Biomedical Engineering, vol. 63, no. 3, pp. 607-618, 2015.

L. Tanzi, P. Piazzolla, F. Porpiglia, and E. Vezzetti, “Real-time deep learning semantic segmentation during intra-operative surgery for 3D augmented reality assistance,” International Journal of Computer Assisted Radiology and Surgery, vol. 16, no. 9, pp. 1435-1445, 2021.

M. Akbari, J. Liang, and J. Han, “DSSLIC: Deep semantic segmentation-based layered image compression,” in ICASSP 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2042-2046, 2019.

N. Sebe, M. S. Lew, and T. S. Huang, “The state-of-the-art in human-computer interaction,” in Computer Vision in Human-Computer Interaction: ECCV 2004 Workshop on HCI, Prague, Czech Republic, May 16, 2004. Proceedings, pp. 1-6,2004.

M. Lew, E. M. Bakker, N. Sebe, and T. S. Huang, “Human-computer intelligent interaction: A survey,” in Human-Computer Interaction: IEEE International Workshop, HCI 2007 Rio de Janeiro, Brazil, October 20, 2007 Proceedings 4, pp. 1-5, 2007.

P. Yu, A. K. Qin, and D. A. Clausi, “Unsupervised polarimetric SAR image segmentation and classification using region growing with edge penalty,” IEEE Transactions on Geoscience and Remote Sensing, vol. 50, no. 4, pp. 1302-1317, 2011.

B. Li, S. Liu, W. Xu, and W. Qiu, “Real-time object detection and semantic segmentation for autonomous driving,” in MIPPR 2017: Automatic Target Recognition and Navigation, vol. 10608, pp. 167-174, 2018.

Y.-H. Tseng and S.-S. Jan, ‘‘Combination of computer vision detection and segmentation for autonomous driving,” in 2018 IEEE/ION Position, Location and Navigation Symposium (PLANS), pp. 1047-1052, 2018.

F. Flohr and D. Gavrila, “PedCut: An iterative framework for pedestrian segmentation combining shape models and multiple data cues,” in BMVC, 2013.

G. Brazil, X. Yin, and X. Liu, “Illuminating pedestrians via simultaneous detection & segmentation,” in Proceedings of the IEEE International Conference on Computer Vision, pp. 4950-4959, 2017.

S. Minaee, Y. Boykov, F. Porikli, A. Plaza, N. Kehtarnavaz, and D. Terzopoulos, “Image segmentation using deep learning: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 7, pp. 3523-3542, 2021.

M. Asri, R. Chowdhury, A. Care, D. F. Lamptey, A. Morgenthaler, O. Camps, and C. M. Rappaport, “Pixel-wise localization of concealed objects on millimeter-wave radar images using deep learning,” IEEE Transactions on Radar Systems, vol. 2, pp. 1027-1035, 2024.

J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431-3440, 2015.

L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “Semantic image segmentation with deep convolutional nets and fully connected crfs,” arXiv preprint arXiv:1412.7062, 2014.

G. Lin, C. Shen, A. Van Den Hengel, and I. Reid, “Efficient piecewise training of deep structured models for semantic segmentation,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3194-3203, 2016.

Z. Liu, X. Li, P. Luo, C.-C. Loy, and X. Tang, “Semantic image segmentation via deep parsing network,” in Proceedings of the IEEE International Conference on Computer Vision, pp. 1377-1385, 2015.

H. Noh, S. Hong, and B. Han, “Learning deconvolution network for semantic segmentation,” in Proceedings of the IEEE International Conference on Computer Vision, pp. 1520-1528, 2015.

V. Badrinarayanan, A. Kendall, and R. Cipolla, “Segnet: A deep convolutional encoder-decoder architecture for image segmentation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, no. 12, pp. 2481-2495, 2017.

Y. Yuan, X. Chen, X. Chen, and J. Wang, “Segmentation transformer: Object-contextual representations for semantic segmentation,” arXiv preprint arXiv:1909.11065, 2019.

X. Xia and B. Kulis, “W-net: A deep model for fully unsupervised image segmentation,” arXiv preprint arXiv:1711.08506, 2017.

T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie, “Feature pyramid networks for object detection,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2117-2125, 2017.

S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” Advances in Neural Information Processing Systems, vol. 28, 2015.

K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask r-cnn,” in Proceedings of the IEEE International Conference on Computer Vision, pp. 2961-2969, 2017.

L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 40, no. 4, pp. 834-848, 2017.

F. Yu and V. Koltun, “Multi-scale context aggregation by dilated convolutions,” arXiv preprint arXiv:1511.07122, 2015.

P. Wang, P. Chen, Y. Yuan, D. Liu, Z. Huang, X. Hou, and G. Cottrell, “Understanding convolution for semantic segmentation,” in 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1451-1460, 2018.

A. Paszke, A. Chaurasia, S. Kim, and E. Culurciello, “Enet: A deep neural network architecture for real-time semantic segmentation,” arXiv preprint arXiv:1606.02147, 2016.

F. Visin, M. Ciccone, A. Romero, K. Kastner, K. Cho, Y. Bengio, M. Matteucci, and A. Courville, “Reseg: A recurrent neural network-based model for semantic segmentation,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 41-48,2016.

W. Byeon, T. M. Breuel, F. Raue, and M. Liwicki, “Scene labeling with lstm recurrent neural networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3547-3555, 2015.

X. Liang, X. Shen, J. Feng, L. Lin, and S. Yan, “Semantic object parsing with graph lstm,” in Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part I 14, pp. 125-143,2016.

Y. Xiang and D. Fox, “DA-RNN: Semantic mapping with data associated recurrent neural networks,” arXiv preprint arXiv:1703.03098, 2017.

R. Hu, M. Rohrbach, and T. Darrell, “Segmentation from natural language expressions,” in Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part I 14, pp. 108-124, 2016.

L.-C. Chen, Y. Yang, J. Wang, W. Xu, and A. L. Yuille, “Attention to scale: Scale-aware semantic image segmentation,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3640-3649, 2016.

Q. Huang, C. Xia, C. Wu, S. Li, Y. Wang, Y. Song, and C.-C. J. Kuo, “Semantic segmentation with reverse attention,” arXiv preprint arXiv:1707.06426, 2017.

H. Li, P. Xiong, J. An, and L. Wang, “Pyramid attention network for semantic segmentation,” arXiv preprint arXiv:1805.10180, 2018.

P. Luc, C. Couprie, S. Chintala, and J. Verbeek, “Semantic segmentation using adversarial networks,” arXiv preprint arXiv:1611.08408, 2016.

W.-C. Hung, Y.-H. Tsai, Y.-T. Liou, Y.-Y. Lin, and M.-H. Yang, “Adversarial learning for semi-supervised semantic segmentation,” arXiv preprint arXiv:1802.07934, 2018.

N. Souly, C. Spampinato, and M. Shah, “Semi supervised semantic segmentation using generative adversarial network,” in Proceedings of the IEEE International Conference on Computer Vision, pp. 5688-5696, 2017.

P. Marquez-Neila, L. Baumela, and L. Alvarez, “A morphological approach to curvature-based evolution of curves and surfaces,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 36, no. 1, pp. 2-17, 2013.

T. H. N. Le, K. G. Quach, K. Luu, C. N. Duong, and M. Savvides, “Reformulating level sets as deep recurrent neural network approach to semantic segmentation,” IEEE Transactions on Image Processing, vol. 27, no. 5, pp. 2393-2407,2018.

C. Rupprecht, E. Huaroc, M. Baust, and N. Navab, “Deep active contours,” arXiv preprint arXiv:1607.05074, 2016.

O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pp. 234-241, Springer, 2015.

K. Li, Y. Zhuang, J. Lai, and Y. Zeng, “PFYOLOv4: An improved small object pedestrian detection algorithm,” IEEE Access, vol. 11, pp. 17197-17206, 2023.

S. Kim, S. Kwak, and B. C. Ko, “Fast pedestrian detection in surveillance video based on soft target training of shallow random forest,” IEEE Access, vol. 7, pp. 12415-12426, 2019.

D. Guan, Y. Cao, J. Yang, Y. Cao, and M. Y. Yang, “Fusion of multispectral data through illumination-aware deep neural networks for pedestrian detection,” Information Fusion, vol. 50, pp. 148-157, 2019.

J. V. B. Severino, A. Zimmer, T. Brandmeier, and R. Z. Freire, “Pedestrian recognition using micro Doppler effects of radar signals based on machine learning and multi-objective optimization,” Expert Systems with Applications, vol. 136, pp. 304-315, 2019.

J. Castanheira, F. Curado, A. Tomé, and E. Gonçalves, “Machine learning methods for radar-based people detection and tracking,” in Progress in Artificial Intelligence: 19th EPIA Conference on Artificial Intelligence, EPIA 2019, Vila Real, Portugal, September 3-6, 2019, Proceedings, Part I 19, pp. 412-423, 2019.

R. Nabati and H. Qi, “Rrpn: Radar region proposal network for object detection in autonomous vehicles,” in 2019 IEEE International Conference on Image Processing (ICIP), pp. 3093-3097,2019.

H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom, “NuScenes: A multimodal dataset for autonomous driving,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11621-11631, 2020.

H. Li, R. Liu, S. Wang, W. Jiang, and C. X. Lu, ‘‘Pedestrian liveness detection based on mmwave radar and camera fusion,” in 2022 19th Annual IEEE International Conference on Sensing, Communication, and Networking (SECON), pp. 262-270, 2022.

X. Gao, H. Liu, S. Roy, G. Xing, A. Alansari, and Y. Luo, “Learning to detect open carry and concealed object with 77 GHz radar,” IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 4, pp. 791-803, 2022.

J. A. Stratton, Electromagnetic Theory. Hoboken, NJ: John Wiley & Sons, 2007.

J. A. Kong, Electromagnetic Wave Theory. Hoboken, NJ: John Wiley & Sons, 1986.

J.-P. Berenger, “A perfectly matched layer for the absorption of electromagnetic waves,” Journal of Computational Physics, vol. 114, no. 2, pp. 185-200, 1994.

S. Gabriel, R. Lau, and C. Gabriel, “The dielectric properties of biological tissues: II. Measurements in the frequency range 10 Hz to 20 GHz,” Physics in Medicine & Biology, vol. 41, no. 11, p. 2251, 1996.

M. Tan and Q. Le, “Efficientnet: Rethinking model scaling for convolutional neural networks,” in International Conference on Machine Learning, pp. 6105-6114, PMLR, 2019.

Downloads

Published

2025-01-30

How to Cite

[1]
M. . Asri, A. . Morgenthaler, and C. M. . Rappaport, “Semantic Segmentation on FDFD-generated Wideband Radar Images of Potential Shooters”, ACES Journal, vol. 40, no. 01, pp. 1–9, Jan. 2025.

Issue

2025: Vol 40 Iss 1

Section

General Submission

Categories