Ransomware Detection on Windows Systems Using File System Activity Monitoring and a Hybrid XGBoost-Isolation Forest Approach

Ransomware has rapidly evolved into one of the most significant cybersecurity threats, with the ability to encrypt critical data and demand ransoms, causing substantial financial and operational damage to organizations worldwide. The novel approach presented in this paper addresses the limitations of traditional signature-based detection methods through a hybrid model that combines supervised and unsupervised machine learning techniques, offering enhanced accuracy in identifying both known and previously unseen ransomware variants. Through real-time monitoring of file system activities on Windows environments, the model utilizes XGBoost for classifying known ransomware behaviors while leveraging Isolation Forest to detect anomalous activities indicative of novel threats. The experimental results demonstrate that the hybrid model achieves high detection accuracy, reduces false positives, and scales efficiently in dynamic environments with varying system loads, making it a viable solution for proactive ransomware mitigation. Moreover, the ability to generalize across zero-day ransomware variants provides a robust defense mechanism against evolving cyber threats. Overall, the proposed hybrid model offers a significant advancement in the field of ransomware detection, bridging the gap between traditional and contemporary detection strategies.