The Operational States Platform of the National Electric Coordinator allows tracking the operational states records of the generating units in the electrical system and calculating the main indices that determine the Sufficiency Power (the capacity of a system to meet peak demand) and generation unavailability indices.
Business Challenges
The Coordinator needs to keep a record of the operational states presented by the power supply of the generating units. To carry out this record, they compiled information from multiple sources and performed complex manual processing, extensively using spreadsheets to handle critical information. On the other hand, the quality of information, processing time, and access to information are significant challenges associated with the implementation of this platform.
Arkhotech proposed the creation of a system accessible to generating companies that would facilitate the calculation process, traceability, and review of key indicators that affect Sufficiency Power and Generation Unavailability. With this platform, the Coordinator could:
Mitigate the possibility of errors by replacing manual formula application and data processing with a 5-year window of automatic calculations.
Make the information available. The system has a public and private view that allows access to information and provides greater transparency in the calculation process.
Reduce processing time and improve the quality of information, achieving greater efficiency in the Markets department.
Unified record through the construction of a data lake, consolidating the information as a single source of truth.
Solution
Our solution consisted of building a data exploration platform that enables the management of large volumes of information in a Data Lake. Through an administrative web interface, users can load, visualize, and generate calculations, as well as query the Operational States of power generating plants.
The application's data sources combine Excel templates, APIs, and databases, which are stored (RAW layer) and processed (Staging layer) using Apache NiFi. The information is consolidated in an RDS PostgreSQL database that considers the system's operational model. The web application is built as an accelerated Angular static site using CloudFront and accessed through Cognito authentication. The backend layer employs a microservices architecture, developed in Flask with Python, orchestrated and scalable using Amazon EKS. Data processing is visible to the user in a control panel integrated with notifications via SES.
Results:
We successfully built a data exploration platform that enables the management of large volumes of information, allowing for complex processing of over 10,000 records in less than 5 minutes.
The web interface facilitates data loading, visualization, and calculation generation, specifically determining Power Sufficiency and Generation Unavailability Indices.
This resulted in a remarkable 98.86% reduction in calculation time.
Information is now available publicly and privately, leading to improved data quality and increased data accessibility.