Autonomous anomaly detection using density-based features in streaming data / Muhammmad Yunus Iqbal Basheer

The rise of Industrial Revolution 4.0 (IR4.0) technology, such as the Internet of Things (IoT), leads to the existence of massive volumes of data. The phenomenon produces a vast volume and variety of data and increases production speed. Consequently, to handle these data, computer algorithms must adapt to their characteristics. Due to its massive volume, variety, and velocity, it contains a lot of insightful patterns. These patterns may include both normal and anomalies data. Anomalies are important to be detected as its existence may require immediate attention and actions. The anomaly data deviate far from normal and may feed wrong information that might lead to wrong decisions and predictions. Hence, it is critical for an anomaly detection algorithm to detect data anomalies patterns. Nonetheless, the process of detecting anomalies in streaming data is laborious. The available algorithms will face difficulties due to the abundance of data produced over time. Furthermore, it needs to operate fast. This research focuses on anomaly detection in streaming data. We built three algorithms to detect anomalies in the streaming data autonomously. These algorithms are data-driven and do not require thresholds or predefined assumptions. They are nonparametric and have no assumptions on the distribution of data. Autonomous anomaly detection (AAD) is enhanced to receive streaming data. It is called multithreaded autonomous anomaly detection for streaming data (MAAD) which works asynchronously while using recursive updates to calculate required mechanisms such as mean and average scalar products. After that, autonomous anomaly detection for streaming data (AADS) is proposed to detect anomalies in any amount of data. The AADS algorithm uses evolving methods which are evolving autonomous data partitioning (eADP) and non-weighted frequency equations. Finally, the AADS is enhanced to operate parallelly, called parallel autonomous anomaly detection for streaming data (PAADS). It is because the parallel mechanism is able to handle high-speed streaming data. The proposed algorithms were evaluated to test their speed in handling streaming data. The performance tests are also conducted to assess whether each algorithm can detect most of the true anomalies. The data is supplied using IoT devices, and benchmark datasets are also presented to test the algorithm's performance.