Download Lab: Analysis of LDPC Codes on Binary Erasure Channel and more Slides Applied Mathematics in PDF only on Docsity!
18.413: Error-Correcting Codes Lab March 18, 2004
Lecture 13
Lecturer: Daniel A. Spielman
13.1 Related Reading
- “Efficient Erasure Correcting Codes”, from Stellar web page. I’m not suggesting that you read this paper right now. But, this is where the material comes from.
13.2 The Binary Erasure Channel
The binary erasure channel can be understood to output three symbols: “0”, “1”, and “?”. The symbols 0 and 1 indicated that the input was a 0 or 1 respectively, and the symbol? indicates that the input was equally likely to have been a 0 or 1. Thus, upon receiving these symbols, the demodulator tells us that the probability that the input symbol was a 1 is 0 or 1 if a 0 or 1 was received, or 1/2 if a? was received.
I claim that if one runs the belief propagation decoding algorithm for a LDPC code after trans- mitting over the erasure channel, then these three probabilities are the only quantities that will be transmitted over edges. To see this, let’s first consider the formula for parity nodes:
pext 1 =
∏k i=2(1^ −^2 p int i ) 2
We see that if pinti = 1/2 for some i, then the product will equal zero, and we will get pext 1 = 1/2. On the other hand, if pinti = ±1 for all i, then each term in the product will be ±1, the product will be ±1, and we will get pext 1 ∈ { 0 , 1 }.
Similarly, for the equality nodes the update formula is:
pext 1 =
∏k i=2 p int ∏ i k i=2 p int i +^
∏k i=2(1^ −^ p int i )^
Here, we see that if pinti ∈ { 0 , 1 / 2 } for all i, and pinti = 0 for some i, then we will get pext 1 = 0 /(0 + 1) = 0. Similarly, if pinti ∈ { 1 , 1 / 2 } for all i and pinti = 1 for some i, then we get pext 1 = 1. Finally, if pinti = 1/2 for all i, then we get
pexti = 2 −(k−1) 2 −(k−1)^ + 2−(k−1)^
One might wonder if we can possibly have both a 0 and a 1 among the values of pinti. If we did, it would clearly be bad as we would then get pext 1 = 0/(0 + 0) = undefined. However, if there are
no numerical errors, then this should be impossible: a 0 is an indication that the bit is definitely 0 and a 1 is an indication that the bit is definitely 1. By the correctness of the belief propagation algorithm on trees, it should be impossible for us to reach both conclusions simultaneously. (While the graph is not a tree, the output of the algorithm corresponds to the output for some tree).
13.3 Analysis of LDPC on BEC
The nice thing about using Low-Density Parity-Check codes on the BEC is that it is possible to analyze their performance. We’ll begin with heuristic approach to the analysis. Let p 0 denote the probability of erasure in the channel. We are going to track the proportion of messages being sent at each iteration that correspond to probability 1/2. We let at denote the proportion of 1/2 messages leaving equality nodes during the tth iteration, and bt denote the proportion of 1/2 messages leaving parity nodes during the tth iteration. As we expect that p 0 of the message bits will be erased, and in the first stage these are the only inputs to the equality nodes, we expect to have a 1 = p 0.
The computation of the expected value of b 1 will be more interesting. Note that a parity node will output 1/2 along an edge if any of its incoming messages on other edges were 1/2. As the graph is random, let’s assume that the probability that any of these incoming messages is 1/2 is a 1. Then, the probability that a particular edge does not see a 1/2 is 1 − a 0. As there are 5 incomming messages, the probability that none of them are 1/2 is roughly (1 − a 1 )^5. Similary the probabiility that some one of the incomming messages is a 1/2, and thus the outgoing message is a 1/2 is 1 − (1 − a 1 )^5. Thus, we obtain
b 1 = 1 − (1 − a 1 )^5.
There was nothing special about this being the first stage. So, in general we obtain
bi = 1 − (1 − ai)^5.
We now perform the analogous computation at the equality nodes. The value output by an equality node along an edge is a function of three inputs: the incomming messages on the other two edges and the value received by the channel. The output value will be a 1/2 only if all three of these are 1 /2. The probability the input from the channel is 1/2 is p 0 , and, assuming the graph is random, we expect the probability that each of the incomming messages is 1/2 to be bi. So, we get
ai+1 = p 0 b^2 i.
13.3.1 Discussion
The above computation is heuristically reasonable. However, the above justification does not really hold up. In particular, the repeated assertion “because the graph is random” needs some mathematics behind it.
One might also wonder why we bother to do the computation in terms of the messages on edges rather than by fractions of nodes. The main reason is that you get the wrong answer this way. I would explain why, but I fear that the exercise of debunking a bad analysis would only serve to teach a bad analysis. If there is demand, I’ll explore why this would be wrong later.
for i = 1:20, a(i+1)= p0 * (1-(1-a(i))^5)^2; end a
a =
Columns 1 through 4
0.40000000000000 0.34021064704000 0.30622652487968 0.
Columns 5 through 8
0.26169574373974 0.24375339697360 0.22659100186113 0.
Columns 9 through 12
0.19090062329355 0.17069720280653 0.14774283981889 0.
Columns 13 through 16
0.09053460661436 0.05709275330846 0.02594402938913 0.
Columns 17 through 20
0.00035931960004 0.00000128925140 0.00000000001662 0.
Column 21
0
p0 = .44; a(1) = p0; for i = 1:20, a(i+1)= p0 * (1-(1-a(i))^5)^2; end a
a =
Columns 1 through 4
0.44000000000000 0.39287014786402 0.37040180678629 0.
Columns 5 through 8
0.34875815840488 0.34295307142171 0.33883631866955 0.
Columns 9 through 12
Columns 13 through 16
0.32902085739940 0.32845676540375 0.32802151644683 0.
Columns 17 through 20
0.32742403093216 0.32722162169403 0.32706438559285 0.
Column 21
So, we can tell that the limit of p 0 that we can handle seems to be between .4 and .44.
Of course, we would like to determine the threshold. There are two ways to go about doing this: algebraic and visual. I think it helps to work visually. We consider the function
f (x) = p 0
1 − (1 − x)^5
and plot it. Our iterated application of f (x) can be understood by reading off the point (p 0 , f (p 0 )) on the plot, then taking (f (p 0 ), f (f (p 0 ))), etc.
(^00) 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.
Figure 13.2: Generated by code http://math.mit.edu/∼spielman/ECC/lect13b.m
A better way to visualize this is to place a mirror on the line x = y in the plot. We can then view the plot as bouncing around off this mirror and the curve. It is easy to show that this is equivalent
(^00) 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.
1
ai
bi
bi = (1−(1−ai))^5 ai = p0 * bi^2
Figure 13.4: Generated by code http://math.mit.edu/∼spielman/ECC/lect13d.m
so the capacity of this symmetric channel is
∑^ k
i=
qiH 2 (pi).
What that means for us is that to estimate the capacity of a channel, we need merely take the probabilities p that pop out from it, and average the resulting values 1 − H 2 (p) (recall that H 2 (p) = −p log 2 (p) − (1 − p) log 2 (1 − p)). We don’t actually need to know whether the estimate is leaning towards correct or incorrect because H 2 (p) = H 2 (1 − p).
I would like to revise project 1 by asking for you to treat the messages being sent at some iteration as a meta-channel, and empiracally computing its capacity in this way instead of keeping the statistics that I asked for.
