Download Error-Correcting Codes Lab 03, Lecture Slide - Engineering and more Slides Applied Mathematics in PDF only on Docsity!
18.413: Error-Correcting Codes Lab February 10, 2004
Lecture 3
Lecturer: Daniel A. Spielman
3.1 Analysis of repetition code meta-channel
When we specialize our interpretation of the output of a channel to the meta channel formed by encoding using the repitition code and transmitting over another channel, we solve a fundamental problem of probability: how to combine the results of independent experiments.
That is, let w be a random varible taking values in { 0 , 1 }. Imagine encoding w using the repeat- 2-times code to (x 1 , x 2 ) = (w, w), and transmitting x 1 and x 2 over a memoryless channel (so each transmission is independent). Equivalently, we could assume that x 1 is transmitted over one channel and x 2 is transmitted over another. Let y 1 and y 2 be the random variables corresponding to the outputs of the channel, let b 1 and b 2 be the values actually received, and let
p 1 = P [x 1 = 1|y 1 = b 1 ] , and p 2 = P [x 2 = 1|y 2 = b 2 ].
We would like to know the probability that w = 1 given both y 1 and y 2. As before, we will assume that w was uniformly distributed (half chance 0 and half chance 1). I think of each channel transmission as an experiment, and I now want to determine the probability that w was 1 given the results of both experiments.
By the theorem from last class, we have
P [w = 1|y 1 = b 1 and y 2 = b 2 ] = P [y 1 = b 1 and y 2 = b 2 |w = 1] P [y 1 = b 1 and y 2 = b 2 |w = 1] + P [y 1 = b 1 and y 2 = b 2 |w = 0] (3.1)
To evaluate this probability, we first note that
P [y 1 = b 1 |w = 1] = P [w = 1|y 1 = b 1 ] P [y 1 = b 1 ] /P [w = 1] = p 1 P [y 1 = b 1 ] /P [w = 1].
While we do not necessarily know P [y 1 = b 1 ], it will turn out not to matter.
Since the two channel outputs are independent given w, we have
P [y 1 = b 1 and y 2 = b 2 |w = 1] = P [y 1 = b 1 |w = 1] P [y 2 = b 2 |w = 1]
=
p 1 P [y 1 = b 1 ] p 2 P [y 2 = b 2 ] P [w = 1] P [w = 1]
Applying P [w = 0|y 1 = b 1 ] = 1 − P [w = 1|y 1 = b 1 ], we can also comput
P [y 1 = b 1 and y 2 = b 2 |w = 0] (1 − p 1 )P [y 1 = b 1 ] (1 − p 2 )P [y 2 = b 2 ] P [w = 0]^2
Combining these equations, and P [w = 0] = P [w = 1] = 1/2, we obtain
(3.1) =
p 1 p 2 p 1 p 2 + (1 − p 1 )(1 − p 2 )
In particular, the terms we don’t know cancel!
3.2 Capacity of meta-channel
Consider the meta-channel obtained by encoding a bit w via the repeat-2-times code to obtain (x 1 , x 2 ), and then passing these bits through the BSCp to obtain (y 1 , y 2 ). We will now compute the capacity of this meta-channel. We begin with the computation of the quantities that appear in the formula for I(w; (y 1 , y 2 )):
P [w = 1|(y 1 , y 2 ) = (1, 1)] = (1 − p)^2 p^2 + (1 − p)^2
P [w = 1|(y 1 , y 2 ) = (1, 0)] = p(1 − p) p(1 − p) + (1 − p)p
P [w = 1|(y 1 , y 2 ) = (0, 0)] =
p^2 p^2 + (1 − p)^2 )
P [w = 0|(y 1 , y 2 ) = (1, 1)] = p^2 p^2 + (1 − p)^2 )
P [w = 0|(y 1 , y 2 ) = (1, 0)] = p(1 − p) p(1 − p) + (1 − p)p
P [w = 0|(y 1 , y 2 ) = (0, 0)] =
(1 − p)^2 p^2 + (1 − p)^2
If you know the prior probability, then you can combine this knowledge with the extrinsic prob- ability to achieve the posterior probability: then actual probability of w = 1 given the channel output. Treating the prior and extrinsic probabilities as independent observations, and applying the calculation of the previous section, we obtain
Ppost^ [w = 1|y = b] = Pext^ [w = 1|y = b] Pprior^ [w = 1] Pext^ [w = 1|y = b] Pprior^ [w = 1] + Pext^ [w = 0|y = b] Pprior^ [w = 0]
A useful exercise would be to re-derive the probability that w = 1 given y = b assuming that w is not uniformly distributed, and to observe that one obtains the above formula.
We will occasionally also see the term intrinsic probability. This will usually be treated in the same way as the prior, but will be distinguished from the prior in that it will often be determined from previous experiments.
