Distributed detection of anomalies Sample Clauses

Distributed detection of anomalies. 5.3.2.1 Performance objectives, and evaluation criteria 5.3.2.2 Methodology: scenarios and tools 5.3.2.3 Experimental results

Distributed detection of anomalies. The classical setting in networking generally assumes that the information that is exchanged is statistically independent. However very frequently this is not strictly the case and nodes do exchange information that is correlated. Distributed anomaly detection has to deal precisely with such a setting: each monitoring node in the network maintains a vector of numerical state information. As the different monitoring nodes observe interacting traffics the state maintained by them is correlated. Distributed monitoring and anomaly detection involves exchanging information between nodes such that each node obtains an approximate view of the states of all other nodes. Through the information exchanged the local observation of a node is extended to the global state of the network. One important challenge here is to deal with nodes selfishness, i.e. a node wants to achieve the best approximation of other nodes states consuming itself the lowest amount of resources. This means that we need an incentive/punishment cooperative mechanism to motivate node to exchange information. One should also assume that the quality of approximation about other states is not defined a priori; it needs to be defined online during the operation of the distributed system. The distributed anomaly detection, a communication scheme initially designed by Lancaster University enables the sharing of correlated node states; a punishment scheme has been proposed that solves the node selfishness issue; a signalling method has been proposed that enables a node to announce to other nodes its state variable of interest as well as their importance. The proposed scheme merges approximation obtained at different nodes into a single consistent approximation with better quality. We will show that the proposed scheme implements a negotiation between neighbours that trade-off an increase of the node transmission rate to its neighbour with a mechanism achieving a better approximation of the state of other nodes.

Distributed detection of anomalies. The integration of the distributed anomaly detection component into the global ECODE architecture is taking advantage of all parts of the architecture. Two major parts of the overall ECODE architecture cooperate during the distributed anomaly detection: the MLE (machine learning engine) and the ME (Monitoring Engine). However there is a major interaction between distant MLE in our case.

Distributed detection of anomalies. The class of parametric statistical anomaly detectors needs to implement three phases: 1. A modelling phase that consists of capturing essential correlation structure of the state vector to monitor. This phase has a strong machine learning flavour and generally uses statistical model calibration techniques as Expectation Minimization to implement Maximum Likelihood fitting or Principal Component Analysis to derive approximate low dimensional models. 2. A filtering phase that consists of using the correlation structure model obtained in the first phase in order to reduce the entropy of the observation by removing the information’s irrelevant to anomaly detection. The approach assumes that everything that goes along with the correlation structure obtained in the first phase is irrelevant to anomaly detection. 3. A decision phase that takes the signal filtered in the second phase and applies to it a statistical test that will decide if there was an anomaly somewhere in the network or not. These three phases are now well investigated in the centralized case where all observations are available at a single point. However, moving from a centralized setting to a distributed setting -our ambition in this project- necessitates implementing the two above defined phases in a distributed way. Our machine learning approach consisted of developing distributed machine learning techniques to deal with distributed modelling. The approach followed consisted of developing a network wide state-sharing scheme that enables each node to obtain an approximation with a controllable precision of other nodes states. This approximation is obtained through a distributed optimization that could be seen as a distributed Principal Component Analysis; a technique widely used in centralized machine learning. This technique was chosen because of the nice and rich theoretical framework that exists around it.