Research on a Network Security Situation Intelligent Awareness and Prediction Model Based on Big Data Technology and Its Supporting Applications

Abstract

The traditional network security situational awareness method is difficult to deal with high-speed multi-source data flow because it relies on a centralized data processing architecture, resulting in poor real-time performance and weak data association. Therefore, building a perception prediction model that can fuse multi-source data in real time, understand the internal structure of the network, and have cognitive reasoning capabilities similar to experts is of great theoretical significance for realizing active intelligent network security defense. The study hypothesizes that the integration of distributed flow processing, network structure mode decomposition, and a cognitive decision-making framework can build a security perception model with high-precision perception and forward-looking prediction ability. The core of this method is to integrate the above components into a unified model of “data structure cognition” three-tier collaboration: Kafka+spark stream processing layer is responsible for real-time data fusion and feature supply. The Structural Modal Modeling and Decomposition (SMMD) layer analyzes the functional topology of the network to achieve fine-grained situation decoupling; The Estimation Memory Control (EMC) cognitive layer simulates the closed-loop process of expert evaluation, experience reuse, and predictive decision-making. This architecture realizes the deep coupling of data-driven, structural understanding, and cognitive intelligence, which is different from the existing situational awareness framework that mainly relies on a single data dimension or lacks explicit cognitive reasoning. To verify this hypothesis, several experiments are designed and implemented. Firstly, a distributed stream processing framework is built based on Kafka+spark to realize the real-time fusion and feature extraction of multi-source security data. Secondly, a safety perception prediction model combining the EMC framework and SMMD is proposed. The network functional topology is analyzed by structural mode decomposition, and the EMC framework is introduced to simulate the expert cognitive decision-making process. The core findings are as follows: Experiments on the Canadian Institute for Network Security network intrusion detection dataset (cic-ids-2017) show that the accuracy of the model in predicting the macro situation level is 93.7%, and the F1-Score for identifying five types of attacks is up to 97.2%. This performance is superior to the mainstream baseline models of LSTM, TCN, and GBDT. In the verification of the real network range, the model can shorten the average response time of high-risk threats to 3.5 minutes, improve the attack containment rate to 95.2%, reduce the false positive intervention rate to 8%, and improve the analysis efficiency by about 83%. The conclusion is that the proposed model is superior to the mainstream methods in perception accuracy, response speed, and operation and maintenance efficiency. This study provides effective technical support for the construction of an active and intelligent network security protection system.

This study also recognizes that the model has some limitations: Its performance depends on high-quality labeled data for initial training and pattern library construction; When dealing with large-scale networks (such as more than 10,000 devices), the computational cost of mode division and state estimation needs to be further optimized; In addition, the prediction ability of the model to the new attack mode (zero day) that has not appeared in the training data remains to be explored. Although the model has been validated on the cic-ids-2017 standard dataset and enterprise-wide, its ability to generalize to other network architectures (such as cloud native, IoT) and more complex real-world operational environments is a direction that needs to be evaluated before actual deployment in the future.