Enhancing Network Communication Security Using Hybrid Cryptographic Techniques

Abstract

The security requirements of networked systems have become increasingly critical due to the growing need for intelligent systems that can detect intrusions and protect data during transmission. This study presents a network security system that combines deep learning-based traffic trust assessment with two different cryptographic protection methods. The system employs a DenseNet–BiGRU design to capture network traffic patterns across different spatial and temporal dimensions, enabling the system to distinguish between normal and malicious traffic before the data is encrypted. The system uses Elliptic Curve Cryptography (ECC) to secure session establishment for trusted traffic, which enables key exchange and implements Advanced Encryption Standard (AES) for data encryption that requires low computational resources. The proposed framework reaches an accuracy of 94.5%, together with a precision of 88.8%, recall of 82.3%, F1-score of 85.4% and Matthews Correlation Coefficient (MCC) of 0.82, which demonstrates its ability to detect under conditions of class imbalance. The model demonstrates exceptional ability to differentiate between classes, which results in an ROC-AUC of 0.96 and PR-AUC of 0.93. The analysis of cryptographic performance shows that encryption and decryption process times remain minimal while system performance maintains consistent throughput, which increases with larger payloads. The framework demonstrates its ability to detect attacks in real time while maintaining secure communication, which makes it suitable for modern network protection and IoT security frameworks.