Delivering a real-time, high-availability, end-to-end data solution to generate new market insights for Elexon
The Challenge
Build the data insights platform used by decision-makers across the British electricity market, while helping Elexon to adopt agile, user-centric product development
Elexon is at the heart of Britain’s electricity system and is responsible for administering the Balancing and Settlement Code. Their reporting service is the primary channel for providing operational data on the state of the market and is used extensively by third parties to understand market dynamics and inform trading decisions.
Elexon has a legal requirement to publish this data accurately, reliably, and in real time. This means that the system must be ready to scale up and ingest large amounts of data at any time; that this data must be secure from tampering or manipulation; and that downtime is simply not an option.
New offerings
The data has long been available through Elexon’s Balancing Mechanism Reporting Service (BMRS), but they had ambitious goals to expand the offering to include more datasets, richer visualisations linked to contextual news and analysis, and a more flexible API, all running on robust, scalable foundations.
This required the building of a new platform, known as the Insights Solution. This sophisticated data platform will sit at the heart of UK electricity analytics and trading for years to come, supplying data to decision-makers through interactive visualisations, as well as via an API and a data publishing service.
Secure and extensible
As a key piece of national infrastructure, the electricity market sets extremely strict security requirements that the Insights Solution must conform with. Elexon also wanted the Insights solution to be easily extensible and were keen to minimise ongoing infrastructure costs.
Agile upskilling
In addition, Elexon wanted its product-development processes to become more agile so that it could dramatically reduce the time taken to get new features through to market. Insights was the first agile project undertaken by Elexon, and they asked Softwire to demonstrate the benefits and help their people to adopt new ways of working.
Our solution
We architected and built a comprehensive Azure data platform which delivers on Elexon’s ambitions, and helped them to adopt an agile delivery mindset along the way. We helped Elexon to develop their cloud operating model and paved the way for future cloud deliveries.
We delivered the Insights Solution by forming a combined team with Elexon, including their technical and product specialists. The solution draws data from over 30 distinct sources, with differing data types, schemas, and update cadences.
As we architected the new solution, we worked closely with Elexon’s experts to understand the business domain and the reasons behind the structure of each dataset, and to design a solution which stores and publishes data in ways which really add value to end users.
Cloud architecture and security
We designed and provisioned a new Microsoft Azure environment for the platform, adopting cloud-native services such as Azure Functions and API Gateway.
We also ran optimisation exercises to reduce ongoing operational costs, for example identifying that Azure Data Factory was a significant proportion of the total cost, and replacing most of its dataflows with simpler Functions.
The electricity industry has stringent security requirements, and we focused on the robustness and resilience of the platform from day one, working with infosec specialists from Softwire and Microsoft to secure the platform in line with the latest best practices, and validated our approach with independent penetration testing.
Data pipelines, APIs, and Visualisations
Our data engineering team built the pipelines to ingest large volumes of data and ensure it could be reliably output in real time. This involved addressing various complexities associated with the disparate set of sources, and the many different legacy file and data formats being used. We also ensured that we had the necessary monitoring and alerting set up to identify failed loads or data issues.
The loaded data are displayed in real-time through rich custom graphs and visualisations, with extensive capabilities for filtering and data export to allow users to identify the data which is important to them, and to download it for further analysis.
We used Azure API Gateway to build a well-documented set of APIs which enable industry users to directly pull specific Elexon data into their own decision-support tools. We invested in creating a great experience for API consumers, using our own experience and speaking with third parties to create an interactive Developer Portal which made APIs easy to explore in the browser.
We also developed a next-generation publish/subscribe service running on Azure Service Bus to distribute data to third parties. We wrote interactive developer documentation, and have published SDKs in ready-to-go GitHub repositories in five programming languages to encourage uptake. This means that end users can get connected to a production-ready data publishing system in only 20 minutes.
The Result
The Insights Solution has been welcomed by industry stakeholders. And our work to showcase agile, cloud-native development best practices has enabled Elexon to adopt these approaches in other teams.
Successful go-live and superb feedback
Elexon launched the beta of the Insights Solution in 2021, including electricity generation and demand visualisations as well as the Developer Portal. It invited comments from the industry, and feedback has been superb, including around ease of use and quality of the visualisations and the API. The data now provided by the Insights Solution serves as the source of truth for electricity traders across the country.
Clear improvements on the previous generation
Elexon’s existing solution was difficult to change, and was tied to outdated data formats, whereas the Insights solution is highly flexible and uses modern open-source standards, with great documentation and open-source SDKs to encourage wider adoption and more flexible use cases for the platform.
With the Insights Solution being fully cloud-native and bespoke to Elexon’s needs, it’s significantly lower cost to run and maintain than its predecessor, while having the scalability and security required for Elexon’s long-term operational needs.
Happy clients
Zaahir Ghanty, Elexon’s Product Manager for the Insights Solution, explains:
“Several things really stood out about Softwire. The first was how quickly they picked up the necessary domain knowledge, in a very complex industry. And second is the collaborative culture we’ve fostered, where Softwire and Elexon operate as one team, with everyone taking ownership and contributing to the success. Overall, Softwire has gained a really good reputation within Elexon as a supplier that can deliver successful agile projects in this sector.”
The team from Softwire and Elexon is continuing to work on the Insights Solution, including integrating it with other Elexon systems and putting in place a long-term support model. Next steps for the project include adding a further 30 sources of data for a total of 60, including APIs and visualisations for the new datasets.
