Download Error-Correcting Codes Lab: Understanding Symmetric Channels and Capacity and more Slides Applied Mathematics in PDF only on Docsity!
18.413: Error-Correcting Codes Lab February 5, 2004
Lecture 2
Lecturer: Daniel A. Spielman
2.1 Channels
A channel is given by a set of input symbols a 1 ,... , am, a set of output symbols b 1 ,... , bn, and a
set of transition probabilities pi,j , where
pi,j = Pr
[
bj is received
∣ai is sent
]
Given a general channel, we now examine how to determine the probability that a particular input
was transmitted given that a particular output was received. That is, we examine how to interpret
the output of the channel. Our derivation follows from many applications of the law of conditional
probability. We let x be the random variable corresponding to the input to the channel, and y be
the output. We assume that x is chosen uniformly from a 1 ,... , am.
Pr
[
x = ai
∣y = b j
]
Pr [x = ai and y = bj ]
Pr [y = bj ]
Pr
[
y = bj
∣x = a i
]
Pr [x = ai]
Pr [y = bj ]
Pr
[
y = bj
∣x = a i
]
Pr [x = ai] ∑
k Pr [y^ =^ bj^ and^ x^ =^ ak]
Pr
[
y = bj
∣x = a i
]
Pr [x = ai] ∑
k
Pr
[
y = bj
∣x = a k
]
Pr [x = ak]
Pr
[
y = bj
∣x = a i
]
(1/m) ∑
k Pr^
[
y = bj
∣x = a k
]
(1/m)
Pr
[
y = bj
∣x = a i
]
k Pr^
[
y = bj
∣x = a k
]
pj,i ∑
k
pj,k
2.2 Capacity and Mutual Information
The capacity of a channel provides a sharp threshold: If one communicates over a channel using
any code of rate greater than the capacity, then the probabiliy that one will have a communication
error tends to 1. On the other hand, there exist codes of every rate less than capacity that drive
Lecture 2: February 5, 2004 2-
the probability of communication error to zero. It is easy to compute the capacity of symmetric
channels.
Definition 2.2.1. A channel is symmetric if
- For all i 1 and i 2 , the vectors (pi 1 , 1 ,... , pi 1 ,n) and (pi 2 , 1 ,... , pi 2 ,n) are permutations of each
other, and
- For all j 1 and j 2 , the vectors (p 1 ,j 1 ,... , pm,j 1 ) and (p 1 ,i 2 ,... , pm,j 2 ) are permutations of each
other.
Most of the channels we consider will be symmetric.
Let x be the random variable uniformly chosen from a 1 ,... , am and let y be the random variable
giving the output of the channel on input x. Then, the capacity of the channel is the mutual
information of x and y, written I(x; y) and defined by
I(x; y)
def
i,j
Pr [x = ai and y = bj ] log 2
Pr [x = ai and y = bj ]
Pr [x = ai] Pr [y = bj ]
If a channel has a simple description, the one can compute I(x; y) directly. Otherwise, one can
estimate I(x; y) by experiment if one can compute the quantity Pr [y = bj |x = ai] / Pr [y = bj ]:
repeatedly choose x at random, generate y, compute
i(x; y)
def = log 2
Pr [x = ai and y = bj ]
Pr [x = ai] Pr [y = bj ]
and take the average of all the i(x; y) values obtained.
2.3 What I should have said
We will extend the definition of a symmetric channel to the following:
Definition 2.3.1. A channel is symmetric if its output symbols can be partitioned into sets such
that for each set S in the partition,
- For all i 1 and i 2 , the vectors (pi 1 ,j )j∈S and (pi 2 ,j )j∈S are permutations of each other, and
- For all j 1 ∈ S and j 2 ∈ S, the vectors (p 1 ,j 1 ,... , pm,j 1 ) and (p 1 ,i 2 ,... , pm,j 2 ) are permutations
of each other.
x
In particular, this defition includes the “nice” channels that I defined in class, which satisfy:
- The channel has two input symbols, a 1 and a 2 = 1, and
