Download Vector-Valued Image Regularization: Anisotropic PDEs and Numerical Schemes and more Papers Computer Science in PDF only on Docsity!
Vector-Valued Image Regularization with PDE’s :
A Common Framework for Different Applications
D. Tschumperl´e R. Deriche
INRIA, Odyss´ee Lab, 2004 Rte des Lucioles, BP 93, 06902 Sophia-Antipolis, France. { David.Tschumperle,Rachid.Deriche } @sophia.inria.fr
Abstract
(^1) We address the problem of vector-valued image regu-
larization with variational methods and PDE’s. From the study of existing formalisms, we propose a unifying frame- work based on a very local interpretation of the regulariza- tion processes. The resulting equations are then specialized into new regularization PDE’s and corresponding numeri- cal schemes that respect the local geometry of vector-valued images. They are finally applied on a wide variety of image processing problems, including color image restoration, in- painting, magnification and flow visualization.
1. Introduction & Motivation
Anisotropic regularization PDE’s raise a strong interest in the field of image processing. The ability to smooth data while preserving large global features such as contours and corners (discontinuities), has opened new ways to handle classical image-related issues (restoration, segmentation). Thus, many regularization schemes have been presented so far in the literature, particularly for the case of 2 D scalar images I : Ω ⊂ R^2 → R ( [1, 17, 18, 28] and references therein). Extensions of these algorithms to vector-valued images I : Ω → Rn^ have been recently proposed, leading to more elaborated diffusion PDE’s : a coupling between image channels appears in the equations, through the con- sideration of a local vector geometry , given pointwise by the spectral elements λ+, λ− (positive eigenvalues) and θ+, θ− (orthogonal eigenvectors) of the 2 × 2 symmetric and semi positive-definite matrix G =
∑n j=1 ∇Ij^ ∇I
T j (also called structure tensor [25, 26, 28, 29]). The λ± respectively de- fine the local min/max vector-valued variations of I in cor- responding spatial directions θ±, i.e. the local configura- tion of the image discontinuities. (note that λ+=‖∇I‖ and θ+ = ∇I/‖∇I‖ for scalar images , n = 1). Regulariza- tion schemes generally lie on one of these three following approaches, related to different interpretation levels :
(1) Functional minimization : Regularizing an image I may be seen as the minimization of a functional E(I) mea- suring a global image variation. The idea is that minimizing this variation will flatten the image, then remove the noise :
minI:Ω→Rn^ E(I) =
Ω φ(N^ (I))^ dΩ^ (1) (^1) This article was published in CVPR2003 - IEEE Conference on Com- puter Vision and Pattern Recognition , Madison/USA June 2003
where φ : R → R is an increasing function and N (I) is a norm related to local image variations , for instance N (I) =
λ+ + λ− = trace (G)
(^12)
. The minimization of (1) is performed with a gradient descent (PDE) given by the Euler-Lagrange equations of E(I). Useful references for vector image regularization are [5, 12, 16, 18, 20, 22, 26]. (2) Divergence expressions : A regularization process may be also designed more locally, as the diffusion of pixel val- ues - viewed as chemical concentrations [11, 28] - driven by a 2 × 2 diffusion tensor D (symmetric and positive matrix) :
∂Ii ∂t = div (D∇Ii)^ (i^ = 1..n)^ (2)
It is generally assumed that the spectral elements of D give the two weights and directions of the local smoothing per- formed by (2). D is then usually designed from the spectral elements of G in order to smooth I anisotropically, while respecting its intrinsic local geometry by preserving its dis- continuities. Anyway, this interpretation of (2) should not be systematic, as pointed out in further paragraphs. (3) Oriented Laplacians : 2D image regularization may be finally seen as the juxtaposition of two oriented 1D heat flows , i.e two monodimensional gaussian smoothing along orthonormal directions u⊥v, with corresponding weights c 1 and c 2 [14, 19, 25, 26] :
∂I ∂t =^ c^1
∂^2 I ∂u^2 +^ c^2
∂^2 I ∂v^2 =^ c^1 Iuu^ +^ c^2 Ivv^ (3)
Like divergence expressions, c 1 , c 2 and u, v are usually de- signed from the spectral elements λ± and θ± of G, in order to perform edge-preserving smoothing, mainly along the di- rection θ− orthogonal to the vector image discontinuities. The link between these three formulations (1),(2),(3) is generally not trivial, especially for vector-valued images. Actually, it is well known for the classical case of φ- functional regularization of scalar images (n = 1). In this case, the three following formulations are equivalent :
(1) minI:Ω→R
Ω φ(‖∇I‖)^ dΩ^ (4)
⇒ (2) ∂I ∂t = div
φ′ (‖∇I‖) ‖∇I‖ ∇I
⇒ (3) ∂I ∂t = φ
′ (‖∇I‖) ‖∇I‖ Iξξ^ +^ φ
′′ (‖∇I‖) Iηη
where η=∇I/‖∇I‖ and ξ⊥η. Note that this regularization leads to anisotropic smoothing (in the sense that it is per- formed in privileged spatial directions ξ and η with different weights), despite the isotropic shape of the corresponding divergence-based tensor D = φ
′ (‖∇I‖)/‖∇I‖ Id. In this paper, we propose a way to find such equivalences for the more general case of vector-valued regularization. We tackle each of the three interpretation levels (1),(2),(3) in its more general form, and derive the corresponding equa- tions. We particularly show that the oriented-Laplacian for- malism has an interesting interpretation in terms of local filtering , and represents the right smoothing geometry per- formed by the PDE’s. Then, we design a new vector-valued regularization approach respecting desired local smoothing properties as well as adapted numerical schemes (section 4 and 5). Finally, we apply it for color image restoration, in- painting, magnification, and flow visualization (section 6).
2. From Variational to Divergence Forms
We first consider vector-valued image regularization as a variational problem. We want to find the corresponding divergence-based expression , i.e. the link (1)⇒(2).
- A generic functional : Instead of considering a functional such as (1) depending on a variation norm N (I), we rather propose to minimize this more general ψ-functional :
minI:Ω→Rn^ E(I) =
Ω ψ(λ+, λ−)^ dΩ^ (5)
where the λ± are the eigenvalues of the structure tensor G =
∑n j=1 ∇Ij^ ∇I
T j , and^ ψ^ :^ R (^2) → R is an increas-
ing function. This is a natural and generic extension of the scalar φ-function formulation (4) for vector-valued images.
- Gradient descent : The Euler-Lagrange equations of (5) can be derived, and reduce to a simple form of divergence- based expression ∂I ∂ti = div (D∇Ii ) , (i = 1..n) (see [27] for a full demonstration), where the 2 × 2 tensor D is :
D = (^) ∂λ∂ψ+ (λ+, λ−) θ+θT + + (^) ∂λ∂ψ− (λ+, λ−) θ−θT −
D is simply defined from the partial derivatives of ψ, and has the same eigenvectors θ+, θ− as G.
- Link with existing approaches : Particular choices of functions ψ leads to previous vector-valued regular- ization approaches defined as variational methods, such as the whole range of Vector φ-functionals [16, 22] : ψ(λ+, λ−) = φ(
λ+ + λ−), or the Beltrami flow frame- work [12] : ψ(λ+, λ−) =
(1 + λ+)(1 + λ−). More gen- erally, our approach shows that eigenvalues of a divergence tensor D define the gradient of a potential function ψ (if such a ψ exists), linked to the functional (5). Anyway, the shape of D is still giving a wrong estimation of the local smoothing performed by the process : For instance, the φ- functional case leads to isotropic tensors D, while the ef- fective local smoothing is anisotropic.
3. From Divergences to Oriented Laplacians
We rather want to develop divergence forms as (2) into their corresponding oriented Laplacian formulations, i.e. find the link (2)⇒(3). Indeed, it is particularly understandable in terms of local geometric smoothing :
- Geometric meaning of oriented Laplacians : Let us consider the oriented Laplacian-based equation (3). As u⊥v, this PDE can be equivalently written as : ∂Ii ∂t = trace (THi)^ (i^ = 1..n)^ (6)
where Hi is the Hessian matrix of the vector component Ii and T is the 2 × 2 tensor defined as : T = c 1 uuT^ + c 2 vvT^ , characterized by its two eigenvalues c 1 , c 2 and its corre- sponding eigenvectors u⊥v. Suppose that T is a constant tensor over the definition domain Ω. Then, it can be shown [24, 27] that the formal solution of the PDE (6) is :
Ii(t) = Ii(t=0) ∗ G(T,t)^ (i = 1..n) (7)
where ∗ stands for the convolution operator and G(T,t)^ is an oriented gaussian kernel , defined by :
G(T,t)(x) = (^41) πt exp
− x
T (^) T− (^1) x 4 t
with x = (x y)T
It is a generalization of the Koenderink’s idea [13], who proved this property for the isotropic diffusion tensor T = Id, resulting in the well-known heat flow equation : ∂Ii ∂t = ∆Ii^. The top row of Fig.1 illustrates a gaussian ker- nel G(T,t)(x, y) obtained with an anisotropic tensor T (top left) and the corresponding evolution of the PDE (6) on a color image (top right). It is worth to notice that the rep- resentation of G(T,t) gives exactly the classical ellipsoid drawing of T. Conversely, it is clear that T represents the effective smoothing performed by the PDE (6).
100 200 300 400 500 600 700 800
100 200 300 400 500 600 700 800
Figure 1: Behavior of trace-based PDE’s (6) (right) with constant or spatially varying tensors T (respectively top and bottom left).
- We don’t want to mix diffusion contributions between image channels. The desired coupling between vector com- ponents Ii should only appear in the diffusion PDE through the computation of the structure tensor G. This means that we have to define only one diffusion tensor A, then choose Aij^ = δij A. Undesired coupling terms are then avoided.
- On homogeneous regions (corresponding to low vec- tor variations), we want to smooth isotropically , in order to remove the noise efficiently with no-preferred directions : ∂Ii ∂t '^ ∆Ii^ = trace (Hi). It means that the tensor^ A^ must be isotropic in these regions : lim(λ++λ−)→ 0 A = αId.
- On vector edges (corresponding to high vector varia- tions), we want to perform an anisotropic smoothing along the vector edges θ−, in order to preserve them while remov-
ing the noise : ∂I ∂ti = trace
βθ−θ−T^ Hi
, where β is a
decreasing function, avoiding corners over-smoothing any- way. This corresponds to an anisotropic tensor A in these regions : lim(λ++λ−)→ 0 A = βθ−θ −T.
The following multivalued regularization PDE respects all these local geometric properties :
∂Ii ∂t = trace (THi)^ (i^ = 1..n)^ (12)
where T is the tensor field defined pointwise as :
T = f+
λ∗ + + λ∗−
θ−∗θ∗−T^ + f−
λ∗ + + λ∗−
θ +∗θ∗ +T
λ∗± and θ±∗ are the spectral elements of Gσ = G ∗ Gσ , a gaussian smoothed version of the structure tensor G, giving a more coherent approximation of the vector variation direc- tions and magnitudes (see [28]). For our experiments in sec- tion 6, we chose f+(s) = (^) 1+^1 s 2 and f−(s) = √1+^1 s 2. This is of course one possible choice (inspired from the hyper- surface formulation of the scalar case [1]) that verifies the above geometric properties, relying on practical experience. The point is that we can freely choose the weighting func- tions f± to obtain specific regularization behaviors, since we are sure of the local smoothing performed by (12).
5. Numerical schemes
The PDE (12) can be implemented with classical numeri- cal schemes, based on centered spatial discretizations of the gradients and the Hessians [15]. Here we propose an al- ternative approach based on the local filtering interpretation of trace-based equations (6), (section 3) : the PDE veloc- ity can be locally estimated by applying a spatially varying gaussian smoothing mask G(T,t)^ over the image I :
trace (THi) =
k,l=− 1 G
(T,dt)(k, l) Ii(x − k, y − l)
Main advantages of this numerical scheme are :
- It preserves the maximum principle , since the local filter- ing is done only with normalized gaussian kernels.
- It is more precise, since the computed kernels G(T,t)^ do not depend on a discretization in privileged axis x and y. In particular, no (imprecise) second derivatives (in the Hes- sians Hi) have to be computed (Fig.2). As for shortcomings of this scheme, we have to mention that it is specially time-consuming, since it needs the com- putation of several exponentials for each (x, y) and each iteration. For our experiments, we chose 5 × 5 kernels.
(a) Noisy image (b)sian discretizations^ Scheme using^ Hes- (c)filtering techniques^ Scheme using local
Figure 2: Comparisons of numerical schemes.
6. Applications
We applied our particular regularization PDE (12) to handle these important image-processing issues :
- Color image restoration : Fig.3a represents a digital pho- tograph with real noise , due to the bad lightning conditions during the snapshot. Our vector-valued regularization PDE (12) can successfully remove the noise, while preserving the important global features of the image.
- Improvement of lossy compressed images : Lossy com- pression algorithms often introduce visible image artefacts : for instance, bloc effects are classical JPEG drawbacks. Us- ing our flow (12) significantly improves the quality of such degraded images (Fig.3b).
- Color image inpainting : Image inpainting, recently pro- posed in [4, 7, 8, 9] consists in filling undesired holes (de- fined by the user) in images by interpolating the data lo- cated at the hole’s neighborhood. It is possible to do that by applying our regularization PDE (12) only in the holes to fill : boundaries pixels will be diffused until they com- pletely fill the missing regions, in a structure-preserving way. We used it to suppress text on images (Fig.3c), re- move real objects in photographs (Fig.3d) and reconstruct partially coded images for compression purposes (Fig.3e).
- Color image magnification : In the same way, we per- formed nonlinear image magnification : An image can be magnified by applying our PDE (12) on a linear interpola- tion of the corresponding small image, excepted on the orig- inal known pixels. It suppresses the usual jagging or bloc effects, inherent to classical interpolation methods (Fig.3g).
- Flow visualization : Considering a 2D vector field F : Ω → R^2 , we have several ways to visualize it. We can first use vectorial graphics, but we have to subsample the field since this kind of representation is not adapted to rep- resent big flows. A better solution is as follows. We smooth a completely noisy image I, with a regularizing flow equiv- alent to (12) but where T is directed by the F, instead of
the spectral elements of the structure tensor G :
∂Ii ∂t = trace
([
1 ‖F ‖ FF
T
]
Hi
(i = 1..n) (13)
Whereas the PDE evolution time t goes by, more global structures of the flow F appear, i.e. a visualization scale- space of F is constructed. Here, our used regularization equation (13) ensures that the smoothing of the pixels is done exactly in the direction of the flow F (Fig.3f). This is not the case in [3, 6, 10], where the authors based their equa- tions on divergence expressions. Using similar divergence- based techniques would raise a risk of smoothing the image in false directions, as this has been pointed out in section 3.
Conclusion & Perspectives
We proposed a unifying formalism expressing a large set of existing vector-valued regularization approaches within a common framework, adapted to understand the local behav- ior of regularization PDE’s, by explaining the link between diffusion tensors in divergence or trace-based equations and corresponding local gaussian filtering processes. From this study, we defined a specific regularization equation, based on the respect of a coherent anisotropic smoothing preserv- ing the global features of vector images, as well as specific numerical schemes adapted for accurate implementations. The successful application to several image processing is- sues demonstrated the efficiency of our regularization ap- proach More results can be found in the author’s web page : http://www-sop.inria.fr/odyssee/team/David.Tschumperle
References
[1] G. Aubert and P. Kornprobst. Mathematical Problems in Image Pro- cessing: PDE’s and the Calculus of Variations , vol. 147 of App. Mathem. Sciences. Springer-Verlag, January 2002. [2] D. Barash. Bilateral filtering and anisotropic diffusion : Towards a unified viewpoint. Technical report, HP Laboratories Israel, 2000. [3] J. Becker, T. Preusser, and M. Rumpf. PDE methods in flow sim- ulation post processing. Computing and Visualization in Science , 3(3):159–167, 2000. [4] M. Bertalmio, G. Sapiro, V. Caselles, and C. Ballester. Image in- painting. In Kurt Akeley, editor, Proceedings of the SIGGRAPH , pages 417–424. ACM Press, Addison Wesley Longman, 2000. [5] P. Blomgren and T.F. Chan. Color TV: Total variation methods for restoration of vector-valued images. IEEE Trans. Imag. Proc. , 7(3):304–309, 1998. Special Issue. [6] D. Buerkle, T. Preusser, and M. Rumpf. Transport and diffusion in timedependent flow visualization. In Proceedings IEEE Visualiza- tion , 2001. [7] T. Chan, S.H. Kang, and J. Shen. Euler’s elastica and curvature based inpainting. SIAM J. Appl. Math. , 2002. [8] T. Chan and J. Shen. Mathematical models for local deterministic inpaintings. Technical Report 00-11, Department of Mathematics, UCLA, Los Angeles, March 2000.
[9] T.F. Chan and J. Shen. Non-texture inpainting by curvature-driven diffusions (CDD). J. Visual Comm. Image Rep. , 12(4):436–449,
[10] U. Diewald, T. Preusser, and M. Rumpf. Anisotropic diffusion in vector field visualization on euclidian domains and surfaces. IEEE Trans. on Visualization and Comp. Graphics , 6(2):139–149, 2000. [11] G. Gerig, O. Kubler, R. Kikinis, and F. Jolesz. Nonlinear anisotropic filtering of mri data. IEEE TMI , 11(2):221–231, 1992. [12] R. Kimmel, R. Malladi, and N. Sochen. Images as embedded maps and minimal surfaces: movies, color, texture, and volumetric medical images. Internat. Journal of Comp. Vision , 39(2):111–129, 2000. [13] J.J. Koenderink. The structure of images. Biological Cybernetics , 50:363–370, 1984. [14] P. Kornprobst, R. Deriche, and G. Aubert. Nonlinear operators in image restoration. In Proceedings of the International Conference on Computer Vision and Pattern Recognition , pages 325–331, Puerto Rico, June 1997. IEEE Computer Society, IEEE. [15] Laurence Lucido, Rachid Deriche, Luis Alvarez, and Vincent Rigaud. Sur quelques sch´emas num´eriques de r´esolution d’´equations aux d´eriv´ees partielles pour le traitement d’images. Rapport de Recherche 3192, INRIA, June 1997. [16] A. Pardo and G. Sapiro. Vector probability diffusion. In Proceedings of the International Conference on Image Processing. IEEE Signal Processing Society, September 2000. [17] P. Perona and J. Malik. Scale-space and edge detection using anisotropic diffusion. IEEE Transactions on Pattern Analysis and Machine Intelligence , 12(7):629–639, July 1990. [18] G. Sapiro. Geometric Partial Differential Equations and Image Anal- ysis. Cambridge University Press, 2001. [19] G. Sapiro and D.L. Ringach. Anisotropic diffusion of multivalued images with applications to color filtering. IEEE Transactions on Image Processing , 5(11):1582–1585, 1996. [20] J. Shah. Curve evolution and segmentation functionals: Applications to color images. In Proceedings of the International Conference on Image Processing , pages 461–464, 1996. [21] N. Sochen, R. Kimmel, and A.M. Bruckstein. Diffusions and con- fusions in signal and image processing. Journal of Mathematical Imaging and Vision , 14(3):195–209, 2001. [22] B. Tang, G. Sapiro, and V. Caselles. Diffusion of general data on non- flat manifolds via harmonic maps theory : The direction diffusion case. Internat. Journal of Comp. Vision , 36(2):149–161, 2000. [23] C. Tomasi and R. Manduchi. Bilateral filtering for gray and color images. In Proceedings of the IEEE International Conference on Computer Vision , pages 839–846, January 1998. [24] D. Tschumperl´e. PDE’s Based Regularization of Multivalued Images and Applications. PhD thesis, Universit´e de Nice-Sophia Antipolis, December 2002. [25] D. Tschumperl´e and R. Deriche. Constrained and unconstrained PDE’s for vector image restoration. In Ivar Austvoll, editor, Pro- ceedings of the 10th Scandinavian Conference on Image Analysis , pages 153–160, Bergen, Norway, June 2001. [26] D. Tschumperl´e and R. Deriche. Diffusion PDE’s on Vector-Valued images. IEEE Signal Processing Magazine , 19(5):16–25, 2002. [27] D. Tschumperl´e and R. Deriche. Vector-valued image regularization with PDE’s : A common framework for different applications. RR 4657, INRIA Sophia-Antipolis, December 2002. [28] J. Weickert. Anisotropic Diffusion in Image Processing. Teubner- Verlag, Stuttgart, 1998. [29] S. Di Zenzo. A note on the gradient of a multi-image. Computer Vision, Graphics, and Image Processing , 33:116–125, 1986.
