Big data analytics approach for intrusion detection using random forest and support vector machine

The increasing volume and complexity of network traffic have created significant challenges for conventional Intrusion Detection Systems (IDS), particularly in processing large-scale data and detecting malicious activities efficiently. This study develops a Big Data Analytics-based IDS by integrating Apache Spark with machine learning techniques for network intrusion detection. Random Forest (RF) and Support Vector Machine (SVM) were implemented as the classification algorithms, while the CICIDS2017 benchmark dataset was used for training and evaluation. The system was developed using Python, Apache Spark, Scikit-learn, and Streamlit following the Agile Software Development methodology. Network traffic data were preprocessed using Apache Spark to validate the required features and remove missing, invalid, and infinite values before classification. The models were evaluated using accuracy, macro precision, macro recall, macro F1-score, classification reports, and confusion matrices. The experimental results showed that RF achieved the best overall performance, with an accuracy of 99.05%, Macro Precision of 24.91%, Macro Recall of 24.92%, and Macro F1-score of 24.92%. In comparison, SVM achieved 77.05% accuracy, 15.47% Macro Precision, 18.34% Macro Recall, and 16.20% Macro F1-score. The preprocessing module also successfully reduced 445,909 uploaded records to 445,645 valid records by removing 264 invalid records before prediction. The developed web based system provides dataset uploading, security analytics, threat prediction, model comparison, and automated security reporting. Overall, the findings demonstrate that RF provided better classification performance than SVM within the developed IDS, while the integration of Apache Spark and Streamlit enabled an end-to-end platform for network traffic processing, intrusion detection, analysis and reporting.