Real-time energy trading and analytics on Databricks

Real-time energy trading and analytics on Databricks | Databricks Blog Skip to main content

Summary

Energy trading teams often run separate systems for live trading and analytics, creating stale risk data, duplicated governance, and synchronization overhead.

Databricks runs this on one platform. Spark Structured Streaming Real-Time Mode streams market data into Lakebase, a Serverless Postgres for apps and agents. That data feeds the live trading app, machine learning models, dashboards and Genie One, a natural-language query tool.

Unity Catalog, the Databricks governance layer, controls it all in one place. Permissions, lineage and audit are set up once instead of for each system.

Why does real-time energy trading need more than faster data? Energy trading desks work in markets where conditions change quickly and decisions are measured in milliseconds. Two changes make this harder: higher price volatility, and the move EU’s day-ahead electricity market move from hourly to 15-minute trading intervals . The 15-minute market time unit is part of the Single-Day Ahead Coupling (SDAC), the system that connects day-ahead electricity markets across EU countries. This creates four times as many market periods per hour and increasing the volume of records, orders, and risk recalculations. Keeping those calculations accurate requires data that is current – which points directly to how data is stored and governed, not only how fast it arrives. Stale data is more than an inconvenience for a trading desk. It can make risk controls reflect an earlier state of the market, contribute to missed intraday moves, and put desk limits out of sync with current positions. Many organizations respond with two stacks. An operational system supports the live desk and millisecond transactions. A separate analytical system supports strategy backtesting, machine learning and regulatory reporting. The split matches two different workloads, but it creates four recurring problems: Two versions of the truth A freshness gap in risk controls Duplicated governance and lineage Multiple synchronization pipelines between the different systems

Machine learning adds another dependency. Models need historical data for training, but their outputs must be available in the live trading workflow. Moving models and data between separate systems adds complexity to an already time-sensitive process. The alternative is a unified platform for data, analytics and AI – one that Real-Time Mode streams in with millisecond latency and Lakebase Postgres holds ready – so both the live desk and the model pipeline draw from the same source without an extra transfer step. How can operational and analytical workloads run together on Databricks? Operational and analytical workloads run together by sharing one data foundation. The solution presented below by Victoria Sicking, Solutions Architect at Databricks, described in ' Run Live Energy Trading, Risk, and Analytics in Databricks ' below combines six components on the Databricks Data + AI Platform, and the key design choice is that no component maintains its own separate copy of the data. All of them work from what Spark Structured Streaming Real-Time Mode has written into Lakebase and the lakehouse. Spark Structured Streaming Real-Time Mode for continuous event processing with millisecond latency Lakebase , a serverless Postgres for apps and agents, for the operational database Databricks Apps for the live trading interface. Machine learning models for real-time price forecasts and trade signals. Genie One for historical analysis. Unity Catalog for governance across the workflow.

Spark Structured Streaming Real-Time Mode receives market data and streams it into Lakebase , used here as the operational (OLTP) database for millisecond reads, transactions, and upserts. The same data can then support the analytical workloads built on the lakehouse . Teams no longer need to coordinate parallel pipelines, one for the live desk and another for backtesting and reporting. Time previously burned on reconciliation and coordination is released by creating a single ingestion pathway into your data estate for market and position data. Unity Catalog – the Databricks governance layer for permissions, lineage and audit – applies that governance across the app, business intelligence and machine learning workloads, so they are not configured independently in multiple systems. What does a live energy trading desk look like in Databricks Apps? It looks like a single application showing prices, positions, profit and loss (P&L) and risk, all updating as the market moves. The live experience is built on Databricks Apps – the way to build and host data applications, such as a live trading interface, on the platform – bringing together what a desk needs to monitor the market and act on changing conditions, served directly from Lakebase so the desk is always reading from the same governed data that feeds the analytical workloads, with no separate operational copy required. The demo displays instrument prices for Brent crude, German and French power spots, EUR-to-USD conversion, EU carbon allowances (EUAs) and TTF gas. Risk is monitored against the desk's limits in the same workflow. When a measure approaches a limit, the app raises a risk alert so the desk can see the change and respond immediately. This replaces a nightly batch process that calculates risk from an earlier snapshot of the data. The application is not a reporting surface disconnected from the data platform. Lakebase provides the operational layer behind the app, and the streaming pipeline keeps the information current. How do you generate trade signals from the same data? Train machine learning models on the same governed data that supports the live desk, track their development and lineage with MLflow , and deploy approved versions through Databricks Model Serving . Their outputs can then surface directly in the app. Energy trading teams need models that can anticipate risk and identify potential trade signals, and running everything on one governed, unified copy of the data removes the handoff between a training system and a live system – the model sees exactly what the desk sees. The demo shows a real-time price forecast that can contribute to a trade signal. The path between the data used to train a model and the workflow that consumes its output gets shorter. Machine learning becomes part of the end-to-end trading workflow...