Download Error-Correcting Codes: Definitions, Properties, and Existential Bounds and more Study notes Number Theory in PDF only on Docsity!
CS225: Pseudorandomness Prof. Salil Vadhan
Lecture 14: Error-Correcting Codes
April 3, 2007
Based on scribe notes by Sasha Schwartz and Adi Akavia.
1 Basic Definitions
The field of coding theory is motivated by the problem of communicating reliably over noisy chan- nels — where the data sent over the channel may come out corrupted on the other end, but we nevertheless want the receiver to be able to correct the errors and recover the original message. There is a vast literature studying aspects of this problem from the perspectives of electrical engi- neering (communications and information theory), computer science (algorithms and complexity), and mathematics (combinatorics and algebra). In this course, we are interested in codes as ‘pseu- dorandom objects,’ ones that are intimately related with the other objects we are studying. In particular, we will see how to use ideas from coding theory to construct the condensers and unbal- anced expanders that we assumed in the previous lectures (for our construction of extractors).
The approach to communicating over a noisy channel is to restrict the data we send to be from a certain set of strings that can be easily disambiguated (even after being corrupted).
Definition 1 A q-ary code is a set C ⊆ Σˆn, where Σ is an alphabet of size q. Elements of C are called codewords. Some key parameters:
- nˆ is the block length.
- n = log 2 |C| is the message length.
- ρ = n/(ˆn · log |Σ|) is the (relative) rate of the code.
An encoding function for C is an injective mapping Enc : { 0 , 1 }n^ → C (for n a positive integer). Given such an encoding function, we view the strings in { 0 , 1 }n^ as messages. The code is explicit if Enc is computable in polynomial time.
Note that every code C whose message length is an integer has an encoding function Enc. We view C and Enc as being essentially the same object (with Enc merely providing a ‘labelling’ of codewords), with the former being useful for studying the combinatorics of codes and the latter for algorithmic purposes. Our notation differs from the standard notation in coding theory in several ways. Typically in coding theory, the input alphabet is taken to be the same as the output alphabet (rather than { 0 , 1 } and Σ, respectively), the blocklength is denoted n, and the message length (over Σ) is denoted k and is referred to as the rate.
So far, we haven’t talked at all about the error-correcting properties of codes. Here we need to specify two things: the model of errors (as introduced by the noisy channel) and the notion of a successful recovery.
For the errors, the main distinction is between random errors and worst-case errors. For random errors, one needs to specify a stochastic model of the channel. The most basic one is the binary symmetric channel (over alphabet Σ = { 0 , 1 }), where each bit is flipped independently with prob- ability δ. People also study more complex channel models, but as usual with stochastic models, there is always the question of how well the theoretical model correctly captures the real-life dis- tribution of errors. We, however, will focus on worst-case errors, where we simply assume that at most some δ fraction of symbols have been changed. That is, when we send a codeword c ∈ Σ nˆ over the channel, the received word r ∈ Σˆn^ differs from c in at most δˆn places.
Definition 2 For two strings x, y ∈ Σˆn, their (relative) Hamming distance dH (x, y) equals Pri[xi 6 = yi]. For a string x ∈ Σˆn^ and δ ∈ [0, 1], the Hamming ball of radius δ around x is the set B(x, δ) of strings y ∈ Σˆn^ such that dH (x, y) ≤ δ. Define Hq(δ, nˆ) to be such that |B(x, δ)| = qHq^ (δ,nˆ)·nˆ.
For the notion of a successful recovery, the traditional model requires that we can uniquely decode the message from the received word (in the case of random errors, this need only hold with high probability). Our main focus will be on a more relaxed notion which allows us to produce a small list of candidate messages. As we will see, the advantage of such list-decoding is that it allows us to correct a larger fraction of errors.
Definition 3 Let Enc: { 0 , 1 }n^ → Σˆn^ be an encoding algorithm for a code C. A δ-decoding algo- rithm for Enc is a function Dec : Σnˆ^ → { 0 , 1 }n^ such that for every m ∈ { 0 , 1 }n^ and r ∈ Σnˆ^ such that dH (Enc(m), r) ≤ δ, we have Dec(r) = m. If such a function Dec exists, we call the code δ-decodable.
A (δ, L)-list-decoding algorithm for Enc is a function Dec : Σnˆ^ → ({ 0 , 1 }n^ )L^ such that for every m ∈ { 0 , 1 }n^ r ∈ Σˆn, such that dH (Enc(m), r) ≤ δ, we have m ∈ Dec(r). If such a function Dec exists, we call the code (δ, L)-list-decodable.
Note that a δ-decoding algorithm is the same as a (δ, 1)-list-decoding algorithm. It is not hard to see that, if we do not care about computational efficiency, the existence of such decoding algorithms depends only on the combinatorics of the code.
Definition 4 The (relative) minimum distance of a code C ⊆ Σˆn^ equals minx 6 =y∈C dH (x, y).
Proposition 5 Let C ⊆ Σnˆ^ be a code with any encoding function.
- C is δ-decodable iff its minimum distance is greater than 2 δ.
- C is (δ, L)-list-decodable iff for every r ∈ Σˆn, we have |B(r, δ) ∩ C| ≤ L.
The factor of 2 in Item 1 is the reason that list-decoding can provide an advantage over unique decoding in the fraction of errors corrected.
The main goals in constructing codes are to have infinite families of codes (e.g. for every message length n) in which we:
- Maximize the fraction δ of errors correctible (e.g. constant independent of n and ˆn).
Proposition 7 (Johnson Bound)
- If C has minimum distance 1 − ε, then it is (1 − O(
ε), O(1/
ε))-list-decodable.
- If a binary code C has minimum distance 1 / 2 − ε, then it is (1/ 2 − O(
ε), O(1/ε))- list- decodable.
Proof: We prove Item 1. The proof is by inclusion-and-exclusion. Let r ∈ Σ nˆ, say C 1 , ..., Cs are codewords at distance at most 1 − ε′^ from r, for ε′^ =
2 ε and s = 2/ε′.
1 ≥ fraction of positions where r agrees with some Ci ≥
i
agreement(r, Ci) −
1 ≤i<j≤s
agreement(Ci, Cj )
≥ sε′^ −
s 2
· ε
2 − 1 = 1
where the last inequality is by our setting of parameters. We reached a contradiction, implying s < (^) ε^2 ′.
Note that the quadratic loss in the distance parameter. This means that optimal codes with respect to minimum distance are not necessarily optimal with respect to list-decoding. Nevertheless, if we do not care about the exact tradeoff between the rate and the decoding radius, in both cases, the above can yield codes where the decoding radius is as large as possible (approaching 1 and 1/2, respectively).
3 Explicit Codes
As usual, most applications of error-correcting codes (in particular the original motivating one) require computationally efficient encoding and decoding. For now, we focus on only the efficiency of encoding.
3.1 Hadamard Code
- Codewords: truth tables of all linear functions L : Zn 2 → Z 2.
- nˆ = 2n
- For m ∈ { 0 , 1 }n^ , Enc(m) is the linear function Lm(x) =
i mixi^ mod 2.
- Minimum distance δ = 1/ 2
- List-decodable up to radius 1/ 2 − ε with lists of size O(1/ε^2 ).
3.2 Reed-Solomon Codes
- Σ = Fq for the finite field Fq of size q.
- Codewords: truth tables of all polynomials p : Fq → Fq of degree at most d.
- n = (d + 1) · log q, ˆn = q.
- Minimum distance δ = 1 − d/q
- List-decodable up to radius 1 − O(
d/q) with lists of size O(
q/d).
- Common settings: |F| = O(d) (constant rate and distance), |F| = poly(d) (ˆn = poly(n), δ = 1 − 1 /nΩ(1)).
3.3 Reed-Muller Code
- Σ = Fq for the finite field Fq of size q.
- Codewords: truth tables of all multivariate polynomials p : Fmq → Fq of total degree at most d.
- n =
(m+d d
· log q, ˆn = qm.
- Minimum distance δ = 1 − d/q
- List-decodable up to radius 1 − O(
d/q) with lists of size O(
q/d).
- Generalization of both Reed-Solomon (m = 1) and Hadamard (essentially d = 1 and q = 2, except also have complements of the linear functions).
