Engineering project
From market publication to SQL time series
Turn electricity-market publications into historical price and volume series that can be queried in a SQL database.
The project
Development of electricity-market data collectors for an energy group. Python services retrieve day-ahead and intraday publications, interpret delivery periods and populate a SQL database. Additional processing consolidates ENTSO-E energy and reserve time series.
Engineering approach
The work concerns the meaning of a time interval as much as retrieving a price. A publication may label the end of a quarter-hour while the database expects its start, and a value may be revised after publication. The routines make these conventions and update rules explicit. The diagram separates market collectors from ENTSO-E consolidation as distinct inputs.
Work delivered
Executable collectors, a historical-backfill procedure and SQL routines transforming market publications into structured, updateable time series.
My contribution
- Containerised Python services for Azure Functions, with HTTP triggers and scheduled execution
- Collection of published tables through browser automation, value conversion and handling of hourly and fifteen-minute intervals
- Interpretation of delivery dates, interval boundaries and time offsets; historical backfill over a date range
- SQL MERGE loading and ENTSO-E curve consolidation, selecting the latest revision for each period
Functional overview
Python functions on Azure collect price publications while a SQL workflow consolidates ENTSO-E series. The scripts handle historical backfill, interval normalisation and record inserts or updates through MERGE.
Azure
Docker
Python
SQL Server
Components and flows
- HTTP and scheduled triggers; browser-based extraction of published tables
- Value and delivery-interval normalisation followed by SQL updates
- ENTSO-E processing with latest-revision selection per interval and curve consolidation
Related capabilities
Services by sector
Other projects
Discuss your project
A discussion of your project’s design, development and production deployment.