Big data integration: Performance and Scalability Matter

More data, more variation in data, higher and different resolutions of data, scheduled or event-triggered data and the need to process large amounts of data in as close to real-time as possible: these are the real-life data challenges that utilities around the globe are facing as they optimize their grid and satisfy regulatory requirements with all kinds of new data sources and decision support systems.

Greenbird’s Utilihive platform is designed to handle big data. And lots of it! 

Utilihive enables real-time data flows within Advanced Metering Infrastructure (AMI), Smart Grid, Distributed Energy Resource Management (DERM) and other areas such as our Utilihive Datalake.

Performance and scalability are key.

Here are some examples, both from test bench scenarios and real customer cases.

Example 1: Rapid Processing of Smart Metering Data

Ideally, Advanced Metering Infrastructure moves data quickly and securely from one or more Headend Systems (HES) to a Meter Data Management system. Sometimes, data critical for grid operations such as outage events, needs to go into ADMS or other operational solutions. Transporting and transforming the data into the required formats is a core task for our Utilihive platform.

Here’s how this works in practice:

• One of our customers is currently in the middle of a multi-million smart meter roll-out.

• They asked us to perform a scalability test to demonstrate Utilihive’s data handling with messages from nine million meters.

• For this customer, Utilihive runs in a private data center and collects 15-minute values from six registers for every smart meter.
• We used the same metrics in the scalability test, simulating a two-hour data batch with 432 million meter values.

Utilihive processed, transformed, and handed off all data in under 15 minutes.

For full transparency:

• The test for nine million meters was performed in an on-premise environment with 10 nodes (8CPU cores and 96 GB RAM each).

• The hardware proved to be oversized, as CPU utilization never went above 40%.

• Memory utilization didn’t even go significantly above 10%.
  
  
  

Example 2: Processing Data From Multiple Power Grid Sources

• Another customer uses Utilihive Datalake to store and analyze data from the grid.

• In total, we integrated and provisioned data from different sources, amounting to approximately 70 billion readings from substations all the way to house-hold smart meters.

• Our customer had previously developed a reporting system using queries from this data. Before they implemented Utilihive, this report would take them around three days to compute and finish.

When they implemented the same report using Utilihive Datalake, it was generated in just under 10 minutes.

If you find that hard to believe, so did our client. They told us that they had re-run the report several times, because they thought the results were impossible.

How Can Utilihive Achieve Results Like These?

Utilihive is different from other integration platforms or ESBs. How? It is built as network of reactive microservices (service mesh), following the actor model for highly concurrent applications and the principle of event-driven architecture. This allows for high performance, dynamic scalability and ensures high availability by design. In addition, we can often compress data to below 5% of the original data size, further boosting data processing performance.

In the future, utilities will need to handle growing quantities of data from sensors, meters, local producers and much more. We say: Bring it on!