Download Bregman Iteration: Techniques and Algorithms for Constrained Minimization and more Study notes Cryptography and System Security in PDF only on Docsity!
Notes on Bregman Iteration
Pascal Getreuer
February 5, 2009
Abstract
These notes discuss the Bregman iteration technique for constrained minimization. We review the basic theoretical results (with proofs), in particular, the monotonic convergence of the constraint. Second, we discuss several Bregman-related algorithms.
1 Bregman Iteration
Bregman iteration is a technique for solving constrained minimizations of the form
arg min u
J(u) subject to H(u) = 0
or arg min u J(u) subject to H(u) < σ
where J and H are convex and min H(u) = 0.
Definition 1. The Bregman distance of J between u and v is
DpJ (u, v) = J(u) − J(v) − 〈p, u − v〉 , p ∈ ∂J(v).
Result 1. Bregman distance satisfies
(a) DpJ (v, v) = 0
(b) DpJ (u, v) ≥ 0
(c) DpJ (u, v) + DpJ˜ (v, ˜v) − DpJ˜ (u, v˜) = 〈p − p, v˜ − u〉
Proof. Results (a) and (c) follow immediately from Definition 1. Result (b) follows by definition of subgradient (∂J(v) :=
p : J(u) ≥ J(v) + 〈p, u − v〉 , ∀u
Definition 2. Given parameter λ > 0, Bregman iteration is
uk+1 = arg min u D Jpk (u, uk) + λH(u), pk ∈ ∂J(uk). (1)
Result 2. Suppose that u?^ is a minimizer of H, then Bregman iteration (1) decreases H(uk) monotonically with
H(uk+1) ≤ H(uk+1) + (^1) λ D Jpk (uk+1, uk) ≤ H(uk).
Proof. Since uk+1 minimizes Dp Jk (u, uk) + λH(u),
λH(uk+1) ≤ Dp Jk (uk+1, uk) + λH(uk+1) ≤ Dp Jk (uk, uk) + λH(uk) = λH(uk).
Result 3. Suppose H is differentiable. Then pk − λ∇H(uk+1) ∈ ∂J(uk+1) and a special case of Bregman iteration is { uk+1 = arg min u Dp Jk (u, uk) + λH(u)
pk+1 = pk − λ∇H(uk+1).
Furthermore, let u?^ be a minimizer of H, then iteration (2) satisfies
H(uk) ≤ H(u?) +
Dp J^0 (u?, u 0 ) λk
Proof. Since uk+1 minimizes Dp Jk (u, uk) + λH(u), it satisfies Euler-Lagrange equation
0 ∈ ∂u
[
D Jpk (u, uk) + λH(u)
]
(uk+1) 0 ∈ ∂J(uk+1) − pk + λ∇H(uk+1) pk − λ∇H(uk+1) ∈ ∂J(uk+1).
From Result 1(c), iteration (2) satisfies
Dp Jk (u?, uk) − Dp Jk −^1 (u?, uk− 1 ) ≤ Dp Jk (u?, uk) + Dp Jk −^1 (uk, uk− 1 ) − D Jpk −^1 (u?, uk− 1 ) = 〈pk − pk− 1 , uk − u?〉 = 〈λ∇H(uk), u?^ − uk〉 ≤ λ
H(u?) − H(uk)
where the last inequality follows by the convexity of H. Summing over k = 1,... K and applying Result 2,
∑^ K
k=
Dp Jk (u?, uk) − D Jpk −^1 (u?, uk− 1 ) + λ
H(uk) − H(u?)
D JpK (u?, uK ) − D Jp^0 (u?, u 0 ) + λK
H(uK ) − H(u?)
−D Jp^0 (u?, u 0 ) + λK
H(uK ) − H(u?)
See [1] for more details and other results.
2.2 Split Bregman
Suppose E is convex and Φ is convex and differentiable. Split Bregman [3] solves the problem
min u ‖Φ(u)‖ 1 + E(u)
by the operator splitting
min u,d
‖d‖ 1 + E(u) subject to Φ(u) = d.
Applying Bregman iteration with J(u, d) = ‖d‖ 1 + E(u) and H(u, d) = 12 ‖d − Φ(u)‖^22 yields
(uk+1, dk+1) = min u,d J(u, d) −
pku, u − uk
pkd, d − dk
pk u+1 = pku − λ∇uH(uk+1, dk+1) pk d +1= pkd − λ∇dH(uk+1, dk+1)
where ∇uH(u, d) =
∇Φ(u)
Φ(u) − d
and ∇dH(u, d) = d − Φ(u). Analogous to Result 4, Split Bregman is equivalently
(uk+1, dk+1) = min u,d ‖u‖ 1 + E(u) +
λ 2
‖d − Φ(u) − bk‖^22
bk+1 = bk +
Φ(uk+1) − dk+
The subproblems may then be solved by alternatingly minimizing u and then d,
uj+1 = arg min u E(u) +
λ 2
‖dj − Φ(u) − bk‖^22
dj+1 = arg min d
‖d‖ 1 +
λ 2
‖d − Φ(uj+1) − bk‖^22
In the step for dj+1, the minimizer can be expressed in closed-form as a shrinkage.
References
[1] Stanley Osher, Martin Burger, Donald Goldfarb, Jinjun Xu, and Wotao Yin, “An Iterative Regularization Method for Total Variation Based Image Restoration,” UCLA CAM Report 04–13.
[2] Wotao Yin, Stanley Osher, Donald Goldfarb, and Jerome Darbon, “Breg- man Iterative Algorithms for ` 1 -Minimization with Applications to Compressed Sensing,” UCLA CAM Report 07–37.
[3] Tom Goldstein and Stanley Osher, “The Split Bregman Method for L1 Regular- ized Problems,” UCLA CAM Report 08–29.
