Germany's dependence on natural gas is undeniable, with consumption of this fossil fuel rising steadily over the past decades. Even amidst the shift towards renewable energy sources, natural gas continues to play a crucial role, primarily as a transition fuel, enabling organisations and policymakers to meet short-term CO2 reduction targets. It's also critical for heat-intensive industrial processes and residential heating, areas that have traditionally been difficult to transition to clean energy sources. Given the unique position of natural gas within our economic system, understanding and forecasting its demand is of paramount importance, particularly in times of supply disruption, such as during the Ukrainian war. Proper planning and procurement of natural gas supplies is essential to prevent severe shortages that could drastically affect a country's gross domestic product.

Attempts to forecast natural gas consumption have a long history, with early models focusing on statistical approaches that took into account aspects such as household income, GDP and ambient temperature. The advent of advanced computing resources led to the application of more sophisticated models, including Artificial Neural Networks (ANNs), Support Vector Regression (SVR) and deep learning models such as Long Short-Term Memory (LSTM) models. While these forecasting techniques have led to unprecedented advances, much of the previous work on natural gas consumption modelling lacks reproducibility due to limited access to the underlying data or exact model parameters. This is often due to licensing restrictions imposed on authors.

This project aims to bridge this gap by providing an open source and publicly available platform for natural gas consumption forecasting in Germany. The focus is on ensuring that critical data on the dependent variable and potential independent variables are publicly available, that forecasts are implemented in a time-efficient manner using state-of-the-art models and frameworks, and that these forecasts are continuously published and productized. The aim is to make energy-related scientific issues, which have a significant impact on everyday life, more accessible to the general public.

Potential regressors

Finding the right data for a model can be challenging. Dependent variables for the model have been established in literature. Recent works offer an overview of factors influencing natural gas consumption in today's energy market. A study of literature surveys and discussions was conducted to identify consumption indicators.

Special challenges are faced in this work. All datasets used must be open-source and programmatically accessible for the model.

The potential regressors considered for the model are:

• Temperature: The relationship between natural gas consumption and temperature is well-established in literature.
• Natural gas prices: The influence of pricing on consumption is well-studied, described through "price elasticity."
• Crude oil prices: Prices of other energy commodities can impact natural gas consumption due to substitution effects.
• Coal prices: Wholesale prices of coal can also affect natural gas consumption.
• Electricity prices: Electricity is an energy commodity that can substitute natural gas to some extent.
• Auction prices of European Emission Allowances (EUAs): The European Emission Trading System caps natural gas consumption, but the relationship can be both positive and negative.
• Natural gas storage levels: Storage levels of natural gas can serve as an indicator of demand and consumption.

To obtain the necessary data, various sources are used. Consumption data for natural gas in Germany is provided by Trading Hub Europe GmbH. Temperature data is sourced from the atmospheric reanalysis model ERA-5 and real-time weather information from Open-meteo. Natural gas prices are obtained from Trading Hub Europe GmbH. Brent crude oil prices are available from the U.S. Energy Information Administration. Electricity prices data is provided by Ember. Auction prices of European Emission Allowances (EUAs) are obtained from the European Energy Exchange AG (EEX). Natural gas storage levels data is sourced from Engie.

The data analysis reveals seasonal patterns in natural gas consumption and the importance of temperature as a factor. The historic data of consumption, temperature, and other variables are examined in the figures provided.

Short-term forecast

After understanding the dynamics of natural gas consumption, a model was designed, trained, and evaluated to forecast Germany's natural gas consumption. Previous models, such as a simple seasonal model and a piecewise-linear temperature model, showed promising results, but more sophisticated models are now established in literature. Several models are discussed, including multiple linear regression, Autoregressive Exogenous Model (ARX), Seasonal Autoregressive Integrated Moving Average Model with exogenous regressors (SARIMAX), XGBoost, and NeuralProphet.

The performance of these models is compared using metrics like Mean Absolute Percentage Error (MAPE). The temperature-only versions of ARX and NeuralProphet are also evaluated to create a fully automated forecasting system. The NeuralProphet model performs well in both short-term and long-term forecasts. A hybrid approach is proposed for temperature forecasting, combining short-term forecasts from weather services with a mid-term model trained on historic data. The combined temperature forecasts, along with the temperature-only NeuralProphet model, enable the forecasting of natural gas consumption for the next 365 days.

This approach strikes a balance between predictive performance and deployability, making it suitable for practical applications.

Long-term forecast

The primary aim of this project was to develop and deploy a public-facing natural gas forecasting model, which re-trains and updates predictions daily. The resultant data needs to be stored and served to the public in a straightforward manner. In terms of architecture, the design leans towards simplicity and efficiency. Both temperature and natural gas consumption prediction models are encapsulated in a Docker container as Python code. The results are saved using a storage service that channels forecasts via a backend to be accessed by a frontend. This entire backend operation is hosted on Google Cloud Platform.

A scheduling service is set up to activate daily model re-training and run the model at a certain time. The new forecasts are stored in Google Cloud Storage, from where they can be accessed by users through the frontend. The frontend comprises a web application built with TypeScript and the Next.js framework, with Tailwind CSS aiding in styling. The app's primary function is to visualize data retrieved from the storage endpoint on Google Cloud Storage, avoiding the need for implementing any additional logic.

While the model re-training and runs take place on the Docker container hosted on Google Cloud Platform, the frontend is also hosted on a fully managed platform, Vercel, to reduce costs and administrative tasks. Vercel, which offers free service for non-commercial use, links to a Git repository of a web application built with one of many approved frameworks, managing the deployment and serving of the web application. Though utilizing fully-managed infrastructure solutions may not be a necessity or suit everyone's unique requirements, in this instance, it helps to keep the solution simple and easily reproducible.

The entire work presented is fully reproducible. Both backend (GitHub) and frontend (GitHub) code is available on GitHub, together with detailed instructions on how to run the code.

Acknowledgements

This work was only possible due to the large number of already existing works in the realm of data science, energy forecasting and time series modelling.