Lessons learned from the Battle of Water Demand Forecasting

A highlight of the conference and a continuing long-standing tradition in the field, the Battle of Water Demand Forecasting (BWDF) aimed to compare the effectiveness of various methods for short-term (one-week ahead) urban water demand forecasting. This competition challenged participants to predict weekly water usage patterns with a hourly resolution across real District Metered Areas (DMAs, metered parts of the city’s water network), using different metrics and covering various periods of the year. The method that proved most robust across all these aspects was crowned the winner. The detailed instructions and data remain available on the website, serving as a powerful benchmark for future solutions addressing this complex and crucial task in urban water management.

Our team’s solution is the result of a strong collaborative effort. We formed a multidisciplinary team that brought together expertise from Bielefeld University, KIOS Research Centre, Athens University of Economics and Business, and our own organization. The collaboration was born during one of the recurrent in-person meetings of the ERC-funded Water-Futures project (from here the team’s name), highlighting the importance of these gatherings.

Takeaways

While we couldn’t attend all 31 presentations from participating teams, we were impressed by the wide-ranging interest this competition attracted, especially beyond the water management field. Many universities and companies working with data-driven techniques participated and showcased their approaches. Making such datasets openly available and organizing these competitions is fundamental to advancing the field and building collective knowledge in water management.

Regression Trees vs Deep Learning

A hot topic in the machine learning community, reflected in this challenge, is the competition between regression trees and deep learning techniques. These two families of approaches emerged as the most promising data-driven methods, but they incorporate information and relationships differently: tree based models are usually less computationally expensive and easier to interpret¹ TLDR: the input-output relationship is broken down into a series of yes/no decisions , while on the other hand, deep learning techniques can approximate almost any relationship between inputs and outputs²TLDR: the input-output relationship is expressed as a combination of several layers of simpler functions. The battle showcased this rivalry, with XGBoost and Random Forest leading the regression trees camp, while Long Short-Term Memory (LSTM) networks and Convolutional Neural Networks (CNNs) demonstrated the potential of deep learning techniques.

The key takeaway? Neither approach is clearly superior across all scenarios. While developing our solution, we reached the same conclusion: each technique is better suited to specific tasks (or DMA-metric combinations in the context of the battle). This observation led us to opt for an ensemble approach, allowing us to leverage the strengths of both worlds.

Stronger together

Our choice of an ensemble approach wasn’t only due to the competition between these techniques but also reflected the collaborative nature of our four-partner team. Indeed, the strength of the ensemble approach was validated by the fact that all three top-finishing teams employed some form of ensemble forecasting. This underscores how, when tackling complex forecasting tasks with uncertain parameters, only an ensemble of models can adequately capture the dynamics of the underlying process.

 

Embracing uncertainty

The most important lesson we learned is that sometimes you must let go of the familiar to make room for innovation. A unique aspect of the BWDF was the gradual release of the dataset. While we could retrain our models on new data, we were prohibited from modifying the model structure. With the latest data, demand distribution shifted (people’s habits changed a lot as the dataset comprised COVID and non-COVID time), requiring a mechanism to adapt and learn online.

We anticipated this challenge when developing our solution, but our preliminary tests on the initial dataset suggested us a more conservative approach – a static one, like averaging the models – would have been safer and more robust across all evaluation scenarios. Our intuition proved partially correct: compared to the competition, our method was indeed the one with the narrower confidence interval, meaning that it was systematically performing at a high level, but this conservatism came at the cost of peak performance, as the winners applied some form of online ensemble learning.

This leaves us to wonder: What if we had implemented our online learning idea?

Concluding remarks

Reflecting on our journey in developing a solution for the BWDF, I can only say how fun and stimulating it was to work on it with my colleagues and how proud I am of the solution we developed together. Our team is also very proud of the effort to make the results and the solution open access and accessible to everyone via these links (solution and results). We hope that our work can contribute beyond the context of the battle and help water managers cope with the complex yet crucial problem of short-term urban water demand forecasting.

We would like to acknowledge the fantastic work of the organisers- the University of Ferrara and their partners- for hosting such a fantastic event with incredible attention to detail and enthusiasm. Also, thanks Prof. Alvisi and his team for organising the competition. We are delighted with how the BWDF provided an incredible platform for showcasing and comparing different approaches while opening the field to a broader community, drawing so much interest worldwide.

Finally, congratulations to our winning colleagues, the ZWUcaster team. We have much to learn from your solution, and we’re pleased to see the similarities in our approaches!

If you want to know more about this year’s WDSA/CCWI 2024 conference, read an overview here.