Introduction to information theory Sample Clauses

Introduction to information theory. The first description of information theory was introduced by Xxxxxxx in 1948 [2] who defined the information I of an event x as: I(x) = log 1 P(x) (1.1) This measure associates the information of the event with its uncertainty, given by its prob- ability. That means the less often this event occurs, the larger is its information. When we have a source of the discrete alphabet X with elements {x1, x2, ..., xN} we can compute the average amount of information given from this source by its expectation value. Thus Xxxxxxx introduced the entropy H like: H(X) = E[I(X)] = .N P(xi) · I(xi) = .N P(xi) · log P(x ) (1.2) with 0 ≤ H(X) ≤ log N where the maximal value belongs to a uniform distribution. Now, we can introduce the conditional entropy which is the entropy of a source X given the side information from source Y as: H(X|Y) = . . x∈X y∈Y P(x, y) log P(x|y) = H(X, Y) − H(Y) (1.3) where P(x, y) is the joint distribution of X and Y and P(x|y) the conditional distribution of X given Y. A measure for the correlation between two random variables X and Y distributed according