Data Management Optimization in a Real-Time Big Data Analysis System for Intensive Care

Vital signs monitors in intensive and intermediate care units generate large amounts of data, most of which are not recorded nor taken advantage of. We propose a computer system that allows the automatic and early detection of the deterioration of critical patients, through the real-time processing and analysis of digital health data, including physiological waveform data generated by the medical monitors. Our system tries to emulate the behavior of an expert intensivist physician, giving recommendations for clinical decision making to reduce the uncertainty on diagnosis, treatment options and prognosis. In our previous works, we presented an real-time Big Data infrastructure built using free software technologies. In this paper we improve its data management. We present and evaluate three different data representation models in Apache Kafka. One of this models outperforms the other two in storage space use and delivery time of both real-time and historical data. Our results show that Kafka can be used for historical data storage. This in turn allows us to eliminate the NoSQL database of our previous system. Unlike other works, ours attempts to reduce the number of components to lower system overhead and administration complexity.