When Does a Kalman Filter Beat an LSTM ?
Abstract
Faced with the challenges posed by the energy transition, the decentralisation of generation and the growing integration of renewable energy, the energy sector is undergoing a profound transformation. Energy systems must now become more flexible, intelligent and resilient. Concepts such as microgrids, prosumers and local energy exchanges are emerging as innovative solutions, promoting energy self-sufficiency and more efficient resource management. Our work analyses and implements various methods for processing energy data in order to improve the forecasting and optimisation of energy flows. A comparative analysis allows us to identify the most relevant methods based on use cases and system constraints. Short-term energy load forecasting underpins P2P trading decisions in smart microgrids. Whilst the literature favours LSTM-based architectures, their advantage over conventional estimators is rarely quantified using real-world data. Our main conclusion runs counter to intuition: a well-tuned Kalman filter outperforms all LSTM variants by a factor of 4 on the Frobenius norm. We characterise the conditions under which this applies and show that, when LSTM is justified, Kalman pre-processing constitutes a cost-free improvement.
Short Biography
Noria Foukia is currently an Associate Professor at the University of Aplied Sciences and Arts of Western Switzerland. She holds a Ph.D. in Computer System Security from the University of Geneva, Switzerland, and completed a postdoctoral fellowship at the Information Sciences Institute (ISI), University of Southern California (USC), Los Angeles, USA. Her academic background also includes advanced degrees in Computer Science, Applied Mathematics, and Pure Mathematics from leading French universities : Lyon I-Claude Bernard and Ecole Normale Supérieure de Lyon (France). Her research focuses on cryptography, network security, and cybersecurity including machine learning-based predictive systems. She has extensive experience in intrusion detection and response systems, as well as trust, reputation, and privacy management in collaborative and distributed environments. Her work has been applied to a wide range of domains, including e-commerce, Grid computing, ad hoc network and wireless sensor networks. She is particularly interested in developing secure, intelligent, and privacy-preserving solutions for complex distributed systems. More recently, her work has expanded to AI-driven cybersecurity, privacy-preserving machine learning, federated learning, explainable artificial intelligence (XAI), intelligent transportation systems, and secure data-sharing platforms.
