Download Compressive Sensing and the CoSaMP Algorithm - Prof. Volkan Cevher and more Exams Statistics in PDF only on Docsity!
The CoSamp Algorithm
Sunday, October 12, 2008 Rice University STAT 631 / ELEC 639: Graphical Models Instructor: Dr. Volkan Cevher
Scribe: Andrew Waters
1 Motivation
In previous classes, we have explored the topic of compressive sensing. Tra- ditional techniques for signal acquisition involve acquiring N samples of a signal sampled at a rate faster than twice the Nyquist rate in order to guarantee perfect signal reconstruction. Many signals of practical interest, however, are sparse in some basis, meaning that in some basis they can be represented with K N samples. Rather than initially acquire N samples and then throw away ≈ N − K samples during compression, we would like to be able to sample the compressed signal directly.
This can be accomplished via compressive sensing techniques to produce a compressed signal y by taking M ≈ K linear projections of the desired signal, x, via a projection matrix Φ, as follows:
y = Φx, (1)
where x is an N x1 (sparse) vector, Φ is an M xN matrix, and y is an M x vector.
For signal reconstruction to be successful, Φ must satisfy the restricted isom- etry property or RIP:
(1 − δK )‖x‖^22 ≤ ‖Φx‖^22 ≤ (1 + δK )‖x‖^22 , (2)
where ‖ ‖ 2 defines the ` 2 norm. Finding matrices that satisfies this property is, in general, an NP-complete problem. However, random matrices chosen with i.i.d random entries (Gaussian, Bernouli, etc) will satisfy the RIP with high probability subject to M = O(K log(N/K)) N.
Several algorithms exist for performing the signal reconstruction problem. Some of these include:
- Convex Optimization:
- Basis Pursuit (BP)
- Basis Pursuit De-Noising (BPDN)
- Second-Order Cone Programming
- Iterative Greedy Algorithms
- Matching Pursuit (MP)
- Orthogonal Matching Pursuit (OMP)
- CoSaMP
Here, we will focus our attention on the CoSaMP (Compressive Sampling Matching Pursuit) algorithm. In section 2 we briefly discuss some notational conventions that will be used throughout this document. In section 3 we present the steps of CoSaMP algorithm. Finally, in Section 4 we derive sev- eral lemmas regarding the convergence properties of the CoSaMP algorithm. In particular, we will show that the error in the CoSaMP estimate of a given sparse vector decays exponentially in the number of algorithm iterations.
2 Notation
We define some basic notation used throughout this scribe to eliminate con- fusion.
Vectors:
-Bolded lower case letters (x, y, etc) are used exclusively for vectors.
-Double bars around a vector with an subscript p (ex. ‖x‖p) indicate the ell-p norm of the vector x. ‖x‖p = p
ÈP i |xi|p
Matrices:
-Matrices are denoted with upper-case, bold letter (Φ, etc).
- Φ∗^ indicates the Hermitian transpose of Φ
- Φ†^ indicates the pseudo-inverse of Φ such that Φ†^ = (Φ∗Φ)−^1 Φ∗
- Initialization: x− 1 = 0 (xJ is the estimate of x) at the Jth^ iteration r = y (the current residual)
- Loop until convergence i) Compute the current error: (Note that for Gaussian Φ, ΦT^ Φ is ∼ diagonal) e = Φ∗r.
ii) Compute the best 2K support set of the error (index set): Ω = e 2 K.
iii) Merge the the strongest support sets: T = Ω
S supp(xJ− 1 ).
iv) Perform a Least-Squares Signal Estimation: b|T = Φ†|T y, b|T c = 0.
v) Prune xJ and compute r for next round: xJ = bk, r = y − ΦxJ.
4 Performance Bounds on CoSaMP algorithm
In this section, we prove several important lemmas regarding the CoSaMP algorithm and use these to prove convergence bounds for the algorithm. Specifically, we will show that the error in the CoSaMP estimate decays exponentially until achieving a lower bound dominated primarily by the noise power present in the measured signal.
4.1 Lemma 1
Suppose that Φ obeys the RIP and has an isometry constant δr. Also, define T to be a set of r indices. Then:
‖Φ∗ Tu‖ 2 ≤
È 1 + δr‖u‖ 2 , (3)
‖Φ† Tu‖ 2 ≤
1 + δr
‖u‖ 2 , (4) È 1 − δr‖u‖ 2 ≤‖Φ∗ TΦTu‖ 2 ≤
È 1 + δr‖u‖ 2 , (5) 1 √ 1 − δr
‖u‖ 2 ≤‖Φ∗ TΦTu‖ 2 ≤
1 + δr
‖u‖ 2. (6)
Proof: This follows directly from the RIP property which indicates that the singular values of Φ|T lie between
1 + δr and
1 − δr.
4.2 Lemma 2
Suppose that we have index sets S and T that are disjoint (S
T T = ∅). Now define R = S
S T with |R| ≤ r.
‖Φ∗|S Φ|T ‖ 2 ≤ δr. (7)
Proof: This follows from the spectral norm inequality, which states that any given matrix M has a spectral norm greater than or equal to the spectral norm of any of its submatrices. For our case, if we define M = Φ|R∗Φ|R − I we can see that:
M =
" Φ∗|S Φ|T − I Φ∗|S Φ|T Φ∗|T Φ|S Φ∗|T Φ|T − I
. (8)
Hence, Φ∗|S Φ|T is a submatrix of M. By the spectral norm inequality:
‖Φ∗|S Φ|T ‖ ≤ ‖M‖. (9)
Applying the triangle inequality we also can see that:
‖M‖ ≤ ‖Φ∗|RΦ|R‖ + ‖I‖. (11)
We combine the above with the results of Lemma 1 in order to obtain:
Defining R = supp(s) we can show that
‖e − e|Ω‖^22 ≤ ‖e − eR‖^22 , (20) ⇒
X n
(e(n) − eΩ(n))^2 ≤
X n
(e(n) − eR(n))^2 , (21)
⇒
X
n /∈Ω
(e(n))^2 ≤
X
n /∈R
(e(n))^2 , (22)
X n∈Ω
(e(n))^2 ≥
X n∈R
(e(n))^2 , (23)
⇒
X
n∈Ω\R
(e(n))^2 ≥
X
n /∈R\Ω
(e(n))^2 , (24)
⇒ ‖e|Ω\R‖^22 ≥ ‖e|R\Ω‖^22. (25)
Expanding the LHS of (25):
‖e|Ω\R‖^22 = ‖Φ∗|Ω\R (Φs + n) ‖ 2 , (26) ≤ ‖Φ∗|Ω\RΦs‖ 2 + ‖Φ|Ω\Rn‖ 2 , (27) ≤ δ 4 K ‖s‖ 2 +
È 1 + δ 2 K ‖n‖ 2. (28)
Expanding the RHS (25):
‖e|R\Ω‖^22 = ‖Φ∗|R\Ω (Φs + n) ‖ 2 , (29) ≥ ‖Φ∗|R\ΩΦs‖ − ‖Φ|R\Ωn‖ 2 , (30) = ‖Φ∗|R\ΩΦs|R\Ω‖ − ‖Φ∗|R\ΩΦs|(R\Ω)C ‖ − ‖Φ|R\Ωn‖ 2 , (31)
≥ (1 − δ 2 K )‖s|R\Ω‖ 2 − δ 2 K ‖s‖ 2 −
È 1 + δ 2 K ‖n‖ 2. (32)
Substituting the expansions of both the LHS and RHS into the inital in- equality we find that:
1 + δ 2 K 1 − δ 2 K
δ 2 K + δ 4 K 1 − δ 4 K
δ 2 K ≤ δ 4 K ≤. 1. (35)
4.5 Lemma 5
Define:
b|T = Φ†|T y b|T C = 0
then:
‖x − b‖ 2 ≤ 1. 112 ‖x|T C ‖ 2 + 1. 06 ‖n‖ 2. (36)
Proof:
‖x − b‖ 2 ≤ ‖x|T C ‖ 2 + ‖x|T − b|T ‖ 2 , (37) = ‖x|T C ‖ 2 + ‖x|T − Φ†|T
Φx|T + Φx|T C + n
‖ 2 , (38) = Φ†|T
Φx|T C + n
‖ 2 , (39) ≤ ‖Φ†|T Φx|T C ‖ 2 + ‖Φ†|T n‖ 2. (40)
We now recall that Φ†|T =
Φ∗|T Φ|T
− 1 Φ∗|T and apply the results of Lemma 1 to arrive at:
‖x − b‖ 2 ≤ δ 4 K 1 − δ 3 K ‖x|T C ‖ 2 + ‖n‖ 2 √ 1 − δ 3 K
‖n‖ 2 , (41)
= 1. 112 ‖x|T C ‖ 2 + 1. 06 ‖n‖ 2. (42)
4.6 Lemma 6
We wish to show that:
‖x − xJ ‖ 2 ≤ 2 ‖x − b‖ 2. (43)
Proof:
We prove this lemma through application of the triangle inequality:
