EFFICIENCY ESTIMATION OF PARALLEL ALGORITHM OF ENHANCED HISTORICAL DATA INTEGRATION ON COMPUTATIONAL GRID

The main feature of neural network using for accuracy improvement of physical quantities (for example, temperature, humidity, pressure etc.) measurement by data acquisition systems is insufficient volume of input data for predicting neural network training at an initial exploitation period of sensors. The authors have proposed the technique of data volume increasing for predicting neural network training using integration of historical data method. In this paper we have proposed enhanced integration historical data method with its simulation results on mathematical models of sensor drift using single-layer and multi-layer perceptrons. We also considered a parallelization technique of enhanced integration historical data method in order to decrease its working time. A modified coarse-grain parallel algorithm with dynamic mapping on processors of parallel computing system using neural network training time as mapping criterion is considered. Fulfilled experiments have showed that modified parallel algorithm is more efficient than basic parallel algorithm with dynamic mapping, which does not use any mapping criterion.


INTRODUCTION
The authors have shown in [1][2], that the error of modern sensor data acquisition systems is much less than sensor's error in many cases.The accuracy improvement of physical quantity measurement is provided by (i) sensor calibration using special calibrator or (ii) sensor's periodic testing by reference sensor directly on exploitation place [3].The frequency of calibration/testing procedure is called as inter-testing interval.However operations implementing calibration/testing procedures are rather laborious.Sensor drift prediction during intertesting interval can reduce the laboriousness tremendously [1].However, well known prediction methods, for example 5-degree polynomial, curvilinear alignment and cubic splines do not provide satisfactory results [2,4].Using artificial intelligence methods, in particularly, neural networks are more effective in this case [5][6][7].
Prediction using neural networks is used very widely and in the same time improvement of prediction accuracy traditionally is reached by improvement of neural network structure, using different neurons' activation functions, training algorithms, etc [8].However mentioned approaches often do not provide satisfactory results and therefore it is necessary to use methods of special forming of neural network training set.Two such methods, additional approximating neural network and integration of historical data using set of Integrating Historical Data Neural Networks (IHDNNs), have been proposed and experimentally investigated in [2,7,[9][10].These methods allow considerably decreasing number of sensor calibration/testing by artificial increasing of the training set of predicting neural network.Experimental results of these methods showed [2,4] that they allow increasing an accuracy of sensor drift prediction in 3-5 times at simultaneous increasing of inter-testing interval in 6-12 times.The enhanced method of integration historical data and its simulation modelling on mathematical models of sensor drift in comparison with the basis method of data integration are considered below.
The works [7,9] show, that the method of integration historical data could require considerable computational recourses and time for its execution.In the previous works Turchenko has developed coarse-grain parallelization algorithms for set of Integrating Historical Data Neural Networks with static [11] and dynamic [12] mapping onto processors of parallel computer.The goal of this paper is to estimate an accuracy of enhanced integration historical data method in comparison with the basic method and to investigate an efficiency of its parallelization on the parallel computer systems.The parallelization allows reducing total execution time of the method in order to be used in real-time intelligent data acquisition systems.
This paper is organised as follows.The basic integration historical data method is described in Section II, the difference of the enhanced integration historical data method is outlined in Section III, the architecture of multi-layer perceptron used as IHDNN model is considered in Section IV, the experimental researches of enhanced algorithm are compared with the basic method in Section V, an approach to enhanced integration historical data method parallelization is considered in Section VI, parallelization experiments are presented in Section VII, summary of the results concludes this paper in Section VIII.

BASIC INTEGRATION HISTORICAL DATA METHOD
It is proposed to use three groups of data of sensor drift in [7]: real, historical and hypothetical data.The real data are not available at the beginning of sensor exploitation.Using historical data, obtained as result of calibration or testing of the same type sensors in the similar operation conditions in the same data acquisition channel, can compensate this disadvantage.
The historical data should be integrated in order to account individual properties of drift of each sensor, those data are used as historical [13].It is proposed to use a set of Integrating Historical Data Neural Networks for such integration.Let us consider the historical data of sensor drift as curves The general algorithm of IHDNNs training and prediction sets forming [13] can be described by the following steps (see Fig. 1): 1.To choose the input data described by ( It is necessary to note, that the length of prediction vector (2.7) should be equal to the length of the training vector (2.6) for appropriate functioning of the integration historical data method.The data set (2.1) for forming the training and prediction sets (2.6)-(2.7)can be considered as "window" of historical data.Execution of step 5 of the algorithm above requires (i) shifting this window on one position to the right and choosing the input data according to (2.2) for xck value prediction, (ii) shifting this window on two positions to the right and choosing the input data according to (2.3) for xdk value prediction, (iii) shifting this window on three positions to the right and choosing the input data according to (2.4) for xek value prediction, (iii) shifting this window on four positions to the right and choosing the input data according to (2.5) for xfk value prediction.These shifts are noted in relation to the current position of the "window" described by (2.1).Thus, according to the basic algorithm of integration historical data, the training and prediction sets of each IHDNN is forming separately and therefore each IHDNN implementation does not relate to any other IHDNN implementation by input data.

ENHANCED INTEGRATION HISTORICAL DATA METHOD
The basic method takes into account the values of sensor drift, which are placed in the "window" only and it is a disadvantage of this method.For example, these values are xai and xbi , n i , 1 = for the calibration point b .At the same time the character of each available sensor drift curve is not accounting in relation to the investigated sensor.The main idea of enhanced integration historical data method is the necessity to take into account all the values of sensor drift located on all available historical sensor drift curves [10].Thus, the goal of IHDNN is to predict value xbk on the basis of xak and historical values xai , , etc (see Fig. 1).The values that should be used to form the training set of the IHDNN according to enhanced method can be described by expressions (3.1)-(3.5).
The general algorithm of IHDNNs training and prediction sets forming within enhanced method can be described by the following steps (see Fig. , where

MATHEMATICAL MODEL OF IHDNNS
IHDNNs should provide a non-linear transfer function because the drift of the majority of modern sensors is non-linear.Therefore the multi-layer perceptron should be used as a model for the IHDNN with a nonlinear activation function, such us the logistic function.This kind of neural networks has the advantage of being simple and to provide nice generalized properties [8].
The output value of three-layer perceptron (Fig. 2) can be formulated as: where N is the number of neurons in the hidden layer, 3 i w is the weight of the synapse from neuron i in the hidden layer to the output neuron, i h is the output of neuron i , T is the threshold of the output neuron and 3 F is the activation function of the output neuron.
The output value of neuron j in the hidden layer is given by: where ij w are the weights from the input neurons to neuron j in the hidden layer, i x are the input values and j T is the threshold of neuron j .The logistic activation function is used for the neurons of the hidden layer and the linear activation function, having a coefficient k , is used for the output neuron.During training, the total error is calculated as: (4.5) The steepest descent method for calculating the learning rate [8] is used for removing the classical disadvantages of the back propagation error algorithm.Thus, the adaptive learning rate for the logistic and linear activation functions are given, respectively, by: where, for the training vector p and iteration t , ) (t p j γ is the error of neuron j and ) (t h p i is the input signal of the linear neuron.
The error of neuron i with logistic activation function can be determined by the relation:

EXPERIMENTAL RESEARCHES OF BASIC AND ENHANCED METHODS
As discussed in Sections II and III, it is necessary to have f copies of the IHDNN to predict f drift values of the new sensor k (Fig. 1).In [4], Turchenko has shown that 6 values on the drift curve of the new sensor k are enough to provide a prediction in the future moments of time.Therefore we set f =6.However, for more complex intelligent data acquisition systems and next generation sensors it may worth investigating innovative solutions with multiple data acquisition channels.Scientific investigations with larger number of sensor drift curves help to discover the limitation conditions of the enhanced data integration method, its potential abilities, how to form training vectors, and to determine the optimal number of hidden neurons, etc.In our case the historical data integration was conducted by using 10 sensor drift curves [4], therefore n is assumed to be 10.
The use of real data of the sensor calibration is not expedient for IHDNNs investigation because real data do not fully describe the behavior of a sensor drift.Thus, mathematical models of sensor drift are usually developed for experimental researches [14].The results of industrial sensors calibrations in real environment are the basis of these mathematical models.The real data about the drift are supplemented by additional components that model non-stationarity and the non-uniformity of the drift, systematic and random errors of standard sensors, methodical errors, noises and other errors.As a result the model "with saturation" (Fig. 3) corresponds to the drift of the thermo-resistor 30К5А1 at a working temperature of 150°C has been used for the experiments.
The total number of IHDNNs used in this study of basic and enhanced integration historical data methods is equal In a case of basic integration historical data method single-layer perceptron has 9 inputs.Multilayer perceptron has 9 input neurons, 7 neurons in the hidden layer with logistic activation function and one output neuron.The number of input neurons can be justified by the fact that in the case of 10 sensor drift curves one of them should play the role of an investigated curve and the other curves are used to form the training vector of the IHDNN according to (2.6).The number of the hidden neurons should be less than the number of the input data (corresponding in our case to 7 neurons) in order to ensure good generalization properties of the IHDNN and to avoid transforming it to an associative memory [8].The output neuron provides the predicted value of the sensor drift for future calibration moments which is the only result expected from each IHDNN.
In a case of enhanced integration historical data method we have used the same number of neurons of hidden and output layers of multi-layer perceptron.The difference in the neural network structure is relating to number of input neurons of single-layer and multi-layer perceptrons used for prediction the sensor drift values in different calibrations . In particular, IHDNN had 9 inputs for the xbk value prediction, 18 inputs for the xck value prediction, 27 inputs for the xdk value prediction, 36 inputs for the xek value prediction and 45 inputs for the xfk value prediction.
The experimental results, comparing an accuracy of the basic and enhanced methods of integration historical data, have showed in Fig. 4. The averaged relative error of data integration within basic method is equal to 14-57% at using single-layer perceptron and 12-20% at using multi-layer perceptron.The averaged relative error of data integration within enhanced method is equal to 5-9%.Taking this result we have got together the best results provided as single-layer as well as multi-layer perceptrons.Thus, enhanced method of integration historical data improves accuracy in two times in comparison with the basic method.However, experiments have shown that enhanced integration historical data method requires considerable computational and time recourses for its execution.The distribution of IHDNN (multilayer perceptron) training times among all calibration points at mentioned scenarios of SSE=10 -3 , 10 -4 , 10 -5 and 10 -7 is presented in Fig. 5 at usage of personal computer with Intel Celeron 1,8 GHz processor, 256 Mb RAM and Windows ® operation system.Taking into account additional time needed for approximation of these values and prediction of the sensor drift in the future moments of time, the time of enhanced integration historical data method execution considerably decreasing an efficiency of its application in the real-time intelligent data acquisition systems.In a case of scientific research using 10 curves of sensor drift increases the computational time in 10 times at least.Therefore let us consider parallelization approaches of the enhanced integration historical data algorithm in the Section V below.

PARALLELIZATION OF ENHANCED INTEGRATION HISTORICAL DATA METHOD
Several approaches to parallelize neural networks have been proposed in the literature: according to the architecture of the network [15], taking advantage of the matrix learning rule calculations [16], or parallelizing the presentation of examples [17].In [18], three nested levels of parallelism in neural algorithms have been considered: connection parallelism (parallel execution on sets of weights), node parallelism (parallel execution of operations on sets of neurons), and example (modular) parallelism (parallel execution of examples on replicated networks).The first two levels are a fine-grain parallelism and the third level is a coarse-grain parallelism.Fine-grain parallel algorithms require a lot of lowlevel communications, for example to combine the results of the parallel calculation of weights of each neuron.Their use is, thus, more effective on processors arrays or network of transputers [19][20].Vice versa coarse-grain algorithms are useful when big independent computation tasks should be processed and communications are rarely required.The use of high-performance computers with powerful of the parallel processors is recommended for the implementation of such parallel algorithms [15].
Turchenko in [11][12] has proposed to use coarsegrain approach to parallelize the basic method of integration historical data and developed two coarsegrain parallel algorithms with static and dynamic mapping of IHDNNs onto processors of parallel computer.The experimental results, achieved on parallel computers SGI Origin 300 and NEC TX-7, showed that parallel algorithm with dynamic mapping has better efficiency on 13% in comparison with parallel algorithm with static mapping for mathematical model of sensor drift "with saturation".Therefore we will use coarse-grain parallel algorithm with dynamic mapping for parallelization of the set of Integrating Historical Data Neural Networks according to enhanced integration historical data method.
Parallel algorithm with dynamic mapping [12] is developed by using a "centralized" planning approach with only one processor Master having the role of task planner and each of the other processors (called Slaves) will train the IHDNNs assigned by the Master.The Master starts with assigning the first M IHDNNs to the M available processors and then continues the execution of the mapping procedure consisting in assigning dynamically the tasks to the Slaves as soon as they become idle.Once a Slave has finished its task, it asks the Master for a new one till no tasks are left.We note here that, besides the role of planner, the Master does not fulfill any further calculation.The communications between the Master and the Slaves are ensured by using the standard MPI sending/receiving functions MPI_Send() and MPI_Recv().More details about this algorithm can be founded in [12].
It was shown in [12] that reducing of the efficiency of this parallel algorithm is caused by a non-uniform training time of the IHDNNs.As a possible solution it was proposed to estimate a priori the training time of each IHDNN module.In this case it will be possible to use the information produced by such a pre-processing procedure in order to improve the load balance among the different Slaves.The analysis of the IHDNNs training time distribution according to enhanced integration historical data (Fig. 5) shown, that the IHDNNs, which predict sensor drift values in b calibration, have biggest training time at SSE value increasing.For example, the average training time is equal to 15 and 25 seconds at SSE=10 -5 and SSE=10 -7 respectively.We propose to use this training time as mapping criteria and modify therefore basic parallel IHDNN parallelization algorithm with dynamic mapping.Thus, the most "long" tasks, i.e.IHDNNs predicting sensor drift in calibration b for all curves, should be mapped first.All other IHDNNs can be mapped after since all of them have practically equal training time (see Fig. 5).The modification of the parallel algorithm consists in calculation of the IHDNN reference number taking into account the training time of this IHDNN module.

PARALLELIZATION EXPERIMENTAL RESULTS
The experimental results have been collected by using the computational grid with Globus middleware [21][22].The computational grid consists in 4 dual-processor personal computers Compaq ML350T01 with Pentium III 933 MHZ processors, 128 Mb PC133 MHz RAM, integrated L2 cash 256 Kb, system bus clock rate 133 MHz, 9.1 Gb SCSI HDD, Fast Ethernet 100 Mbit/s network connection to 24-port 3Com Switch 100Mbit/s.Operation system of each computer is RedHat Linux 9 with Globus toolkit v.3.2.1.
Since the architecture of the computational grid is scalable by n 2 processors we trained the 50 IHDNNs on 2, 4 and 8 processors in order to avoid any eventual overhead due to the architectural characteristics of the system.The sequential and parallel routines have been developed by using C as programming language, standard MPI v.1.2[23] and grid-enabled MPICH-G2 v.1.2.6 [24] as message passing libraries and MPE v.1.9.2. as performance visualization package.
The execution time of 50 IHDNNs parallelization according to enhanced integration historical data method is reported in Table 1 by using the basic dynamic mapping algorithm and in Table 2 by using the modified dynamic mapping algorithm.Speedup and efficiency of the basic mapping algorithm are depicted in Figs.6-7.Speedup and efficiency of the modified mapping algorithm are depicted in Figs.8-9.An efficiency comparison of both algorithms is reported in Table 3.As it is seen, the modified mapping algorithm, which uses IHDNN training time as mapping criterion, shows 5% better efficiency in average in comparison with the basic mapping algorithm and 13-16% better efficiency within two scenarios of IHDNNs training till SSE=10 -5 and SSE=10 -7 , which had low efficiency in the basic mapping case.Efficiency: modified dynamic mapping

CONCLUSIONS
The enhanced integration historical data method using the set of Integrating Historical Data Neural Networks is described in this paper.The main idea of enhanced method is to account all available data on all historical curves of the sensor drift.Experiments showed that enhanced method has better accuracy in two times in comparison with basic integration historical data method.However the enhanced method requires considerable time and computational resources for its execution.The parallelization of the enhanced method has been done using coarse-grain approach with dynamic mapping of IHDNNs onto processors of parallel computer system.We have used a modified mapping strategy which uses the training time of the IHDNN as mapping criterion.Parallelization experiments done on computation grid under Globus middleware have showed that the modified mapping algorithm is more efficient on 5-16% in comparison with the basic mapping algorithm, which does not use any mapping criterion.
n is the number of available historical sensor drift curves.The first calibration of the new sensor allows correcting initial sensor error at 0 moment of time.The second calibration of the new sensor allows receiving the first real value xak of sensor drift in calibration point a .The goal of the IHDNN is to predict value xbk on the basis of xak and xai , next value xck on the basis of xbk and xbi , is necessary to form training and prediction sets of IHDNN for fulfilling this task in special way [13].

2 .
1): 1.To choose the input data described by (3.1). 2. To form the training and prediction sets of IHDNN which will predict value xbk analogously to steps 2 and 3 of the basic method from Section II. 3. To choose the input data described by (3.2). 4. To form IHDNN training set in order to predict xck value by the following: 4.1.To choose one curve of sensor drift xi which will describe real data about sensor drift, all other curves xj , To calculate absolute deviation of xbi values from all other xbj values according to form IHDNN prediction set in order to predict xck value by the following expression (3.7).6.To choose the input data described by the next expression from the expressions (3.3)-(3.5). 7. To execute steps 4-6 above in order to form the training and prediction sets for all IHDNNs, which will predict the next values of sensor drift xfk xek xdk , , .The training and prediction sets should be formed by (3.8)-(3.9)for the xdk value, by (3.10)-(3.11)for the xek value and by (3.12)-(3.13)for the xfk value.
Fig. 4. Accuracy comparison of basic and enhancedintegration historical data (IHD) method.

Fig. 5 .
Fig. 5. Distribution of the training times of enhancedIHD method among calibration points.

Fig. 6 .
Fig.6.Speedup vs number of processors using basic dynamic mapping algorithm.

Fig. 7 .
Fig. 7. Efficiency vs number of processors using basic dynamic mapping algorithm.

Fig. 8 .
Fig. 8. Speedup vs number of processors using modified dynamic mapping algorithm.

Fig. 9 .
Fig. 9. Efficiency vs number of processors using modified dynamic mapping algorithm.
Thus, the values that should be used to form the training set of the IHDNN in order to predict each value