A scalable and real-time system for disease prediction using big data processing

Abstract The growing chronic diseases patients and the centralization of medical resources cause significant economic impact resulting in hospital visits, hospital readmission, and other healthcare costs. This paper proposes a scalable, big data, real-time health status prediction and analytical system to filter, manage, process, predict and store streaming health data. The proposed system uses Twitter as data source, Apache Kafka as ingestion tool, Apache Spark as computing engine, Apache Cassandra as storage engine and Apache Zeppelin as analytical tool. Proposed Spark Parallel Random Forest (SPRF) is used as data classifier. The idea is to use Twitter as a communication channel to transmit data generated by IoT medical devices to a system to perform analytics in a real-time fashion, using streaming. Thus, Twitter users tweet attributes related to health, Kafka streaming receives all desired tweets attributes and ingest them to Spark streaming. Here, a machine learning algorithm model is applied to predict health status and send back a response message through Kafka. The result will be saved into distributed database for historical data analysis and visualization. The system has been tested with a Heart disease dataset from UCI. RF classification performances are compared to other machine learning classifiers of Apache Spark MLlib library. To test the system scalability, a comparative study between execution time of RF on Spark environment and on WEKA for both training and application stages. The results show significant better performances of Spark cluster in terms of scalability and computing times.