Cloud Simulation and Virtualization for Testing of Critical Energy Infrastructure Components

Cloud computing and cloud infrastructure are beginning to be applied in order to ensure effective functioning of management system of Critical Energy Infrastructure. This is improved informativeness of Supervisory Control and Data Acquisition system by the application of additional Wide-area Measurement and Control Systems (WAMCSs), since most of all processing and transition operations of data information streams of the WAMCSs may be performed by using on clouds. In light of these circumstances, it is importantly to get extensive modeling results considering wide area range of input data and latency time between critical energy infrastructure components and cloud infrastructure components. However, researchers were not, yet, had possibility to receive more adequate results based only mathematical modeling process. In this case, simulation models considering the cost time are more preferable than mathematical models. In the spirit development of this conception, authors are proposing to employ the simulation model based on virtual technology and Matlab Simulink toolkit in order to get modeling results for more important on-line mode of functioning of the WAMCSs.


I. INTRODUCTION
OWADAYS, power consumption is regularly growing due to people seek to use more electric appliances, equipment and new quite energy consuming technologies. As a result, electric grid loads of Critical Energy Infrastructure (CEI) are constantly amplified. This circumstance is a cause of different negative events such as sudden and hidden failures, power outages of diverse CEI components, including accidents and blackouts of power grid [1][2][3]. Therefore, staff of CEI tries to improve functionality and informativeness of a management system of the infrastructure using additional dynamic monitoring digital devices that work in wide-area frequency range. These devices form Wide-area Measurement and Control System (WAMCS) are based on Phasor Measurement Units (PMUs) [4], which are intended to monitor and control technical parameters of the infrastructure components.
These circumstances also reflect a desire to ensure robust functioning of the management system for energy components based on the phasor-oriented principles of control. Important detail, which staff of CEI should consider, namely a most widely used supervisory, control and data acquisition (SCADA) system works more slowly than PMUs. Indeed, it is a serious issue, since different malicious deliberate impacts and invasions would be able to implement during quite short period. For instance, between two relevant moments of information update.
In addition, according to statements of some scientists and experts [5], unlike SCADA data obtained the PMU data N obtained better reflects technical and information states of CEI, since frequency of processing operation in synchrophasor technology systems is higher. Hence, informativeness of PMU systems is as well as higher than SCADA systems. This is why the large energy providers try to use PMUs together with SCADA system. On the other hand, utilization of the WAMCS based on PMUs causes a sharp increase in data information streams. In this case, staff can make mistakes due to an appropriate time restrict. The desire to exclude the CEI staff mistakes might require the additional information and computational resources based on new system resource ensuring model. Such a model is an Infrastructure of a Service (IaaS) Cloud. Note that among cloud infrastructures, which are utilized to solve similar tasks [6,7], IaaS Cloud is frequently used. It could contain both multiple types of servers [8] and multiple pools of physical (PMs) and virtual machines (VMs) [9].
Unfortunately, nowadays energy infrastructure of Ukraine has quite low functional availability and reliability, since obsolescence level of its components is more than 70%. It means that infrastructure works in very unstable modes, that is electrical signal parameters are often changed. If in such conditions CEI staff will resolve to utilize WAMCSs, then the PMUs within dynamic monitoring systems can continually disable grid power nets due to phase swings. In these complex circumstances, Ukrainian energy companies together with members of Ukrainian Government and scientists must take vigorous steps to prevent this problem. Since alternative solution to the problem, pertaining to ensure effective functioning of the CEI based on the use of WAMCSs and advanced dynamic monitoring technologies does not exist. The special understanding of this problem is key moment in order to implement further transition from used power distribution technologies for traditional energy infrastructure to new advanced technologies for Smart Distribution Grids [10]. In view of the above said, remained significantly important creation of simulation toolkit of different critical situations considering possibilities of usage both PMUs and additional cloud services in order to prepare service personal of CEI. In addition, staff of CEI can use these models in order to perform some testing operations to choose the best cloud service provider (CSP).
In the paper authors propose to consider simulation models, which contain two clusters, namely first cluster is a signal simulator for phasor-oriented functioning components of the CEI and second cluster is simulator for IaaS Cloud. In fact, authors described modeling results pertaining to realm of usage cloud infrastructure in feedback circuit of the management system for CEI considering possibilities of simulation virtual process.
The main provisions of this paper are outlined as follows. Section II describes related works, including analytical and stochastic techniques for estimation of availability and reliability of both CEI and IaaS Cloud. Simulation models for CEI and IaaS Cloud considering on-line mode functioning of the PMUs, which are constructed based on virtual resource and technologies, are presented in Section III. Conclusion and perspectives of the future researches are produced in Section IV.

II. RELATED WORK
In real situation staff of the national CEI faces challenges pertaining to low functional availability and dependability of appropriate energy systems, since as most of their equipment has high level of obsolescence and as well as these systems involve very old equipment. Therefore, faults and outages of these systems amply demonstrate growing concern about generic state of the national CEI. It is noticeable, that even advanced energy infrastructures suffer from different negative impacts. Description of accidents and weatherdriven natural disasters considering resulted damage for CEI and economic of the USA are represented in [11][12][13]. Cyber threats are also a serious negative factor of influence on information and computational resources of the CEI. Because, different intruders can even penetrate into management closed-loop of CEI and create physical, economic damage due to disturbances and outages of energy equipment. For instance, in December 2015 sophisticated and synchronized cyber-attacks on power companies of Ukraine caused cascading outages of regional energy grids, which influenced approximately 225 000 customers [14,15]. In order to eliminate these issues, it is necessarily to develop complex and systematic approach considering both analytical and stochastic simulation methods of behavior of cyber-physical CEI assets for diverse situations. As a result of performed researches, some scientists proposed to employ a private cloud service platform for availability assessments of the CEI components [16] that were obtained based on Semi-Markov Modeling Processes [17,18]. Furthermore, some researchers notice that application of the IaaS Cloud will permit to improve operation efficiency of GridCloud Infrastructure [6]. In [19] authors considered an option of resource sharing of the SCADA and WAMCS systems. Fig. 1 shows overall architecture of CEI involving WAMCS components, SCADA system [19] and cloud assets.
As it is shown in Fig. 1, the WAMCS also consists of micro-PMUs, Phasor Data Concentrators (PDCs) and Frequency Disturbance Recorders (FDRs). Micro-PMUs are devices, which are intended to conduct extreme precise, synchronized measurements of voltage magnitudes, phase angles [20] and usually are utilized at distributed renewables of the CEI [19]. At the same time, PDCs and FDRs are devices, which perform processing of data and flag corrupt data at low-voltage distributing systems of CEI [19].
Cloud components can be deployed on Amazon CSP resources and services. In [6,7] authors propose to employ Amazon EC2 service in order to implement some data mining processing operations of the GridCloud Infrastructure.
Accordingly, seeking to improve functionality of IaaS Cloud would be useful to consider and analyze modeling methods and simulation toolkit of cloud infrastructure behavior. Thanks to researches of some groups of the scientists using Stochastic Reward Nets and Markov Modeling Processes were received data, which can be utilized in order to build IaaS Cloud systems in accordance with the performance, availability, and power consumption requirements of both PMs and VMs [21,22]. In addition, it is obvious that researchers should also consider the availability of data storages for some CSPs during a cyber attack or a physical disaster. It is considered in [23][24][25].

Figure 1. Overall architecture of CEI involving WAMCS components, SCADA system and cloud assets
In spite of the fact, that papers reviewed above have described quite interesting scientific results, at the same time, in these papers authors neglect possibility to create simulation models for complex systems. Surely, this is unacceptable from the standpoint of extension range of the modeling results, when complexness described physical processes is quite high in order to build their mathematical models. In particularly, it is difficult to build adequate mathematical model for different using scenarios of PMUs and GridCloud Infrastructure. In this case, researchers resort to the use of different tools in order to simulate the CEI components operation. The first thing that needs to be noted is that some scientists have been able to perform simulation of the PMU operation using MATLAB toolkit. A MATLABbased simulator for PMU with single-phase estimation performance was developed in [26]. A comparative analysis of the use of MATLAB Simulink and LABVIEW toolkits was made in [27]. It should be also noted that Laboratory Virtual Instrument Engineering Workbench (LABVIEW) is a development environment based on a graphical programming language, which can be used by developers in order to simulate applications and radioelectronic devices [28], including virtual components of the IaaS Cloud. According to the received modeling results, MATLAB simulation model considering computational efforts was preferable.
Reverting to the problem of transition from traditional energy transmission and distribution system to Smart Grid system [29,30], it should be emphasized on papers, which addressed to estimation of survivability metrics for a power distribution grid [31] considering reactive and active constituents [32] that were received based on utilizing Continuous Time Markov Chains. In [33] authors described a technique for simulation modeling of scalable power grids using Möbius toolkit, which focuses on the failure dynamic of infocommunication components considering different scenarios for development of their failures. In fact, Möbius is a multi-paradigm multi-solution framework that can be used by researchers in order to build and apply stochastic models to determine availability level and performance of dynamic systems [34].
One of the attempts to employ Cloud Simulator in order to perform simulate Smart Grid systems considering various virtual parameters, influencing on cost and overall processing data time is presented in [35]. Thus, performed analysis of related works confirms a necessity for building and implementing the Cloud Simulation models for critical energy infrastructure.

III. CLOUD SIMULATION MODELS FOR TESTING OF CEI COMPONENTS
Let us seek to build the simulation models considering requirements, which influence on effective functioning of management system of the critical energy infrastructure reviewing three types of its assets namely, physical assets, cyber assets and cloud assets. Solving this quite complex task, researchers should see the issue of ensuring functional availability of the CEI as one of the most important. Fig. 2 shows taxonomy of ensuring functional availability of the CEI considering three types of its assets.
As indicated in Fig. 2, the issue of ensuring functional availability of critical energy infrastructure provides for assess of property hierarchy, indicators hierarchy, as well as realization of simulation and optimization operations as regards interacting assets of the energy infrastructure. Fig. 3 vividly presents property hierarchy of the critical energy infrastructure considering some peculiarity creation of Electronic Security Perimeter (ESP) [36], Physical Security Perimeter (PSP) for both cyber assets and physical assets.
How to perform simulation and further to execute an optimization operation of CEI assets in order to ensure functional availability of the energy infrastructure [16,18] is the next significant issue that has to be reviewed. In this situation authors propose to set the indicators hierarchy in order to estimate functional availability level of the CEI. The analysis results [6,7] confirmed that latency parameter is one of the most important performance characteristics of the cloud infrastructure, which can be looked upon as an element of management system of the CEI with reciprocal connection between IaaS Cloud and energy infrastructure. Not stopping on some separate features of the transition from property hierarchy to the indicators hierarchy of the energy infrastructure, makes sense to transit to building of simulation models. Figure 2. Taxonomy of functional availability of the CEI considering three types of its assets Figure 3. Property hierarchy of the CEI At the same time, it is noteworthy that latency assessment may be included as an additional requirement for implementation of an advanced cloud platform for the CEI. It means, according to [16] a logical chain that forms the generalized service as an implementation result of concrete transition from Procedure as a Service (PRCaaS) to the Property as a Service (PRPaaS) can be written as where PRDAAs -Product as a Service.
In accordance with the equation (1) (2) Thus, the logical chain (1), (2) was extended additional procedure in order to estimate performance characteristics of CEI considering reciprocal connection between management system of the energy infrastructure and IaaS Cloud based on estimation of latency parameters (Fig. 4) for the closed-loop circuit.
Before beginning building the simulation models for concrete modes of the CEI functioning, we will examine an overall architecture of the simulator for testing of energy infrastructure.

A. OVERALL ARCHITECTURE OF GRID CLOUD SIMULATOR
According to Fig. 2, the simulator for CEI should accomplish testing operations and analysis for all infrastructure components based on cloud assets that influence on functional availability level of the energy infrastructure. Figure 4. Indicators hierarchy of the CEI Furthermore, it is necessarily to remember that cloud assets are part of the Grid Cloud Infrastructure [6,7]. Concurrently, Grid Cloud Infrastructure forms part of a closed-loop management system of the CEI with feedback connection between WAMCS and IaaS Cloud. In this connection, it is pertinent to call the simulator how a Grid Cloud Simulator (CloudSim) for the CEI. Fig. 5 presents overall architecture of the Grid CloudSim for CEI, which contains two main modules: on-line mode simulation module (ONMS) and off-line mode processing (OLMP) module.

Figure 5. Overall architecture of the Grid CloudSim for CEI
In case of on-line simulation, a Signal Simulator (SignalSim) generates input signals considering behavior of devices that constitute the WAMCS, when they perform quite rapid and CEI staff has a possibility to take decision within quite short time period. This simulation mode is inspired by external data information streams of the PMUs, micro-PMUs and PDCs. To implement the off-line mode with an effect of the additional processing, the corresponding module should get the data information streams from both SignalSim and on-line mode simulation module. In this case, we assume that WAMCS based on PMUs and PDCs operates slowly and staff of CEI will have enough time in order to perform a deep analysis of different negative events.

B. ON-LINE MODE OF THE GRID CLOUD SIMULATOR
In this subsection, functioning of the proposed Grid Cloud Simulator in on-line mode is applied to the operative processing of simulated data information streams based on utilizing virtualization technology. Goal of simulation is achieved from testing result for assess of effectiveness functioning of the Grid Cloud Infrastructure in accordance with a criterion for minimizing the latency time for bidirectional data flow.
Proposed simulation model was received by us based on Linux virtualization technology using a general principle of data information interchange between Server and Client. In our case, need to be measure after what time (that is value of t  ) the data information stream sent by the Server to the Client will return to the Server. As a result, researchers can analyze latency time of data information stream between CEI components and virtual objects that might be deemed as components of IaaS Cloud. Taking into account abovementioned, let us try to perform an experiment. Fig. 6 describes a simplified scheme of the experiment for the following components: 1. Data Modeling Module (DMM). The SignalSim module generates data information streams that researchers leverage as transmissible signals.
2. Data Transmitting Module (DTM). Task of the module as Server is dispatching of the data information stream (DIS) to the Client with fixation sending and returning timestamps, after that latency time t  is calculated.
3. Data Receiving and Redirecting Module (DRM). According to the principle of Client and Server, this module receives DIS from DTM and sends it back. In order to implement this simplified scheme (Fig. 6) in accordance with plan of the experiment, researchers should build infrastructure based on virtualization technology. Fig. 7 shows an implementation architecture of on-line mode for this Grid CloudSim considering the simplified scheme of experimental pilot-project. First of all, pursuant to Fig. 7 researchers should create virtual machines. In this case, better to use a special technology called virtual machine monitor -VirtualBox. The Oracle VirtualBox toolkit is a powerful x86/x64 virtualization hypervisor for enterprise and home use, that is freely available as Open Source Software under the terms of the GNU General Public License (GPL) version 2 [37]. Accordingly, as the result researchers will deploy two guest virtual machines namely: a) Server that contains DMM and DTM modules; b) Client that consists of DRM module based on Ubuntu Server 18.04.1 LTS. Each guest machine contains relational database MySQL 5.7. In the case, researchers will also provide informational exchange between these virtual objects using the mentioned technology.
Thus, in order to get simulation results in accordance with the represented models (Fig. 5-7), researchers need to organize DIS between two remote databases. For this goal we propose to use appropriate technology called Federated Storage Engine. The Federated storage engine lets researchers access data from a remote MySQL database without using replication cluster technology. Querying a local Federated table automatically pulls the data from the remote table, and at the same time, data is stored on the local table [38]. Fig. 8 describes the Data Queries formation workflow and general schema of the Server and Client databases (DBs).
Note that when the output signal of ONMS module is sent to input of OLMP module, researchers receive possibility to perform an additional processing operation in order to identify the factors of negative impacts or invasions. As we have said, the observations of behavior CEI components during different malicious deliberate impacts reflect growing concern about safety and security of the energy infrastructure and confirm aspiration to utilize advanced analysis methods. In particular, the additional processing procedure based on the joint use of artificial neural networks [39] and the invariant method [40] can be justified. This is an important point, since pooling of these two methods can only increase availability and cybersecurity of both CEI components and IaaS Cloud.

C. NUMERICAL MODELING RESULTS
In compliance with Fig. 8, DBs have two Federated connections and three triggers (After Insert) that provide data information streams exchange pipeline. It is worth noting, when the triggers fires, researchers can keep the current timestamp and record it. SQL The script is presented below.

SQL Script
Further, to assess the latency time t  using equation (3) as an example the simulation procedure for 1000 iterations of DIS is simulated PMU signal was performed. Modeling results for each iteration of the latency time t  in a scatter plot form are presented in Fig. 9. As seen in Fig. 9, great part of values of the latency time t  is within the range from 650 to 1100 microseconds. Fig. 10 also presents hexagonal binning values of the latency time t  .  On the basis of this principle, researchers performed the next test experiment using the proposed Grid CloudSim. Fig. 11 shows the simplified scheme for this experiment. In accordance with the simplified scheme (Fig. 11), the researchers switched off Data Receiving and Redirecting Module and after that the Grid CloudSim signal was sent to input of the appropriate data information centers for a concrete cloud service provider. Figure 11. Simplified scheme of second experiment Figs. 12 and 13 describe the results of testing (in microseconds) of service-oriented resources for two Ukrainian regional IT companies and eight Amazon Web Services (AWS) data information centers in the "clientserver" mode, which were derived using the proposed Grid CloudSim. The results of this comparative analysis are shown in Table 1. Thus, according to the testing of the service-oriented resources and AWS data information centers, considering the criterion of minimizing the latency time, the following results were obtained: a) AVA VPS hosting (Kiev) has quite low latency time, therefore this resource can be used by the service personnel in order to ensure effective functioning of WAMCS-based on Phasor Measurement Units; b) AWS DIC (Frankfurt) is the best data information center, that service personnel can apply in order to solve tasks pertaining to assurance functional availability of the critical energy infrastructure.

IV. CONCLUSIONS
In fact, the proposed cloud simulation models based on the use of virtual technology and resources, implemented in GridCloud simulator, allow performing testing operations of different CEI components and IaaS Cloud, including WAMCS devices. Moreover, thanks to the simulation of online mode of PMUs operation, the GridCloud simulator can be used in order to improve the quality of performing of additional processing operations based on cloud services. A study on latency time determined the best DICs of AWS, which the service personnel can leverage in order to ensure functional availability of CEI. In concrete situation pertaining to concurrent possible use WAMCS-based on PMUs and AWS data centers their latency time is not more than 0.4 second.
The merits of usage of the GridCloud simulator for CEI staff are quite significant due to additional possibilities to receive new knowledge and practical experience as regards actions pertaining to extreme negative situations. We believe that proposed simulation models in conjunction with other toolkits and techniques can be used in order to get the modeling results that will allow improving functionality of CEI management system considering possibilities of dynamic monitoring subsystems.