End-to-End Weather Forecasting System from scratch

Introduction

I’ve been working on a weather forecasting system that combines IoT hardware with machine learning to predict local temperature patterns. This is an academic project I developed to explore the intersection of embedded systems and predictive modeling, inspired by machine learning concepts that helped shape the system’s approach.

The system demonstrates how you can collect real-time sensor data, process it through a neural network, and generate short-term forecasts - all running on a modest VPS setup.

The complete source code and documentation for this project is available on GitHub.

What It Does

The weather forecasting system consists of three main parts:

ESP32 Sensor Node: Collects temperature readings every 10 minutes and transmits them via MQTT to a central server. The sensor runs on battery power and connects wirelessly to handle data collection in remote locations. Multiple nodes can be deployed to cover different areas.

Prediction Engine: Uses an LSTM neural network to analyze historical temperature data and generate 48-hour forecasts. The model runs on the server and updates predictions every 10 minutes as new data arrives.

Web Dashboard: Provides a simple interface to view current conditions, historical data, and temperature predictions with confidence scores.

Technical Approach

The project uses a straightforward architecture: ESP32 sensors send data over MQTT to a Python server running FastAPI. The server stores sensor readings in a MySQL database and feeds this data to a PyTorch-based LSTM model for temperature prediction.

I kept the model complexity intentionally low to fit within the constraints of a small VPS, focusing more on the end-to-end pipeline than achieving state-of-the-art accuracy. The LSTM processes recent temperature trends to predict future values, with each forecast including a confidence score based on model uncertainty.

The web interface shows both actual sensor readings and predicted temperatures, making it easy to see how well the model performs over time.

How It Started

You know the feeling when you wake up, and you want to go for a bike ride, but you’re not sure if it’s going to rain, or if it’s going to be too hot? Maybe you don’t even know the temperature outside. Well, if you live in middle of nowhere, the local weather forecast might not be very accurate. Or maybe there is no forecast at all. That was the situation I found myself in, and I thought: “Why not build my own weather forecasting system?” So, here we are. I have some experience with electronics, and designing PCBs, and I wanted to learn more about machine learning. Combining these two interests seemed like a perfect project. You can also just order a weather station, but where is the fun in that?

What you Need

You can start with a ESP32 development board, a temperature sensor (like the BME280), and a battery pack. You will also need a server to run the prediction engine and host the web dashboard, but that comes later. First you need to set up the ESP32 to read temperature data from the sensor and transport it via any protocol you like. I chose MQTT because it’s lightweight and well-suited for IoT applications. I did make a custom PCB for the sensor module to make it more compact and reliable, but you can also use a breadboard for prototyping. The main advantage you have when you do your own PCB is that you can optimize power consumption and size. Also it just does what you want, and no less or more. That sounds a bit silly, but I think that the simpler the system is, the easier it is to maintain and understand. Also, you can always revise the design later if you want to add more features. Connecting your sensor to the MQTT broker is straightforward. The ESP32 can connect to WiFi and publish temperature readings every 10 minutes. You only need to setup the I2C communication with the sensor and the MQTT client library. After you have at least one sensor up and running, you can start working on the server side. I used FastAPI for the web framework because it’s fast to develop with, and I did use it before for other projects. You can use whatever you like, though. The MQTT is just a message broker, so you need to subscribe to the temperature topic and store the readings in a database. I used MySQL because why not, but after a few months of storing my influx data there, I would probably choose something more appropriate for time series data, like InfluxDB or TimescaleDB. I’m too lazy to migrate my data now, so MySQL it is for the time being. It works fine for this project, a few million rows of temperature data is not a big deal for MySQL. Your platform/backend is next.

So, how to predict the temperature? I wanted to try out LSTM networks because they are well-suited for time series data, and very lightweight compared to other deep learning models. You can implement an LSTM in PyTorch with just a few lines of code. Very simple, not as accurate as more complex models, but good enough for a proof of concept. The idea is to use the last few hours of temperature data to predict the next 48 hours. So, every 10 minutes, the model gets updated with the latest readings and generates a new forecast. A bit of PyTorch magic, and you have a working prediction engine. The model is trained on historical data, and you can fine-tune it as you collect more data over time. Finally, the web dashboard is just a simple FastAPI app that displays the current temperature, historical data, and the predicted temperatures with confidence scores. You can use any frontend framework you like, or just stick to server-side rendering with Jinja2 templates.

Current Status

This is still an ongoing project. I’m working on improving the model accuracy and expanding the system to include additional weather parameters like humidity and pressure. The modular design makes it relatively easy to add new sensor types or modify the prediction algorithms. A few MB of storage are needed for the database, and the server runs on a small VPS with minimal resource usage. The ESP32 sensor nodes are battery-powered and can run for weeks on a single charge, depending on the sampling interval, and the power optimization of the program itself, and the hardware design.

Future Development

Any system can be improved, and this one is no exception. Here are some areas I’m focusing on for future development:

• Updgrade the machine learning model to incorporate more features and improve accuracy. Yeah, I know why not use transformers from the start, but I wanted to keep it simple.
• Add more sensor nodes to cover a wider area and gather more data for training. Currently only two nodes are deployed.
• Implement alerting features to notify users of significant weather changes based on predictions.
• Upgrade the data storage to a time-series optimized database for better performance with large datasets.

Future Development: Experimental Sensors

One of the most exciting aspects of this project is the development of experimental sensors that haven’t been integrated into the main system yet. I’ve been working on two specialized sensor modules that will significantly enhance the forecasting capabilities:

Ground Vibration Sensor

The ground vibration sensor is designed to detect seismic activity and ground movement patterns that can correlate with weather changes. This experimental module uses piezoelectric sensors combined with operational amplifiers to measure micro-vibrations in the ground, which can provide early indicators of atmospheric pressure changes and weather pattern shifts.

The complete sensor assembly includes a custom PCB with signal conditioning circuits using op-amps, a waterproof housing for outdoor deployment, and a low-power microcontroller for data processing. However, there’s still significant work to be done on the piezo sensor calibration and signal processing algorithms. The goal is to correlate ground vibration patterns with temperature and pressure changes to improve forecast accuracy.

Wind Detection Sensor

The wind detection sensor represents another experimental approach to weather prediction. This sensor uses a crystal, and an amplifier to create a comprehensive wind profile around the measurement site. Wind patterns are often strong indicators of upcoming weather changes, and this sensor aims to capture those patterns at a very local level.

Integration Challenges

Integrating these experimental sensors into the main system presents several interesting challenges:

• Data Fusion: Combining temperature, vibration, and wind data requires sophisticated algorithms to identify meaningful correlations
• Power Management: Each additional sensor increases power consumption, requiring careful optimization for battery-powered operation
• Signal Processing: The vibration and wind sensors generate much more data than simple temperature readings, necessitating edge processing capabilities
• Calibration: Experimental sensors need extensive calibration to ensure data quality and reliability

The modular architecture of the current system makes it relatively straightforward to add these new sensor types, but the real challenge lies in developing the machine learning models that can effectively utilize this multi-modal data for improved weather prediction.

The code is available on GitHub for anyone interested in the implementation details or wanting to adapt it for their own weather monitoring needs.

Learning Outcomes

Building this system taught me a lot about integrating different technologies - from embedded programming on ESP32 to machine learning with PyTorch to web development with FastAPI. The project demonstrates how academic concepts can be applied to create practical IoT solutions, even when working within resource constraints.