CSE 494 CSE/CBS 598 (Fall 2007): Numerical Linear Algebra for Data
Exploration— Nonnegative Matrix Factorization
Instructor: Jieping Ye
1 Nonnegative Matrix Factorization
•An overview of SVD
–Let Abe an m×nmatrix, with m≥n. It can be factorized as
A=UÃΣ
0!VT,
where U∈IRm×mand V∈IRn×nare orthogonal, and Σ ∈IRm×nis diagonal
Σ = diag (σ1, σ2,· · · , σn), σ1≥σ2≥ · · · ≥ σn≥0.
•Weak interpretability: The entries in Uand Vmay be negative, even when Acontains
nonnegative entries only, such as term-document matrix.
•Nonnegative Matrix Factorization
–Let Abe an m×nmatrix, with nonnegative entries. NMF computes two matrices
W∈IRm×kand H∈IRk×n, which solve the following optimization problem:
min
W≥0,H≥0||A−W H ||F.
–Here, W≥0 indicates that all entries in Ware nonnegative.
•(Lee and Seung, nature 1999) started a flurry of research into the Nonnegative Matrix Fac-
torization. There are hundreds of papers in the area since 1999.
–NMF is shown to induce parts-based representation.
•Many applications:
–Image Processing and Computer Graphics
–Text analysis
–Bioinformatics
1.1 Alternating Least Squares (ALS) Algorithm
•This minimization problem above is nonlinear considered as an optimization problem for W
and H at the same time. However, if one of the unknown matrices were known, W, say, then
the problem of computing H, would be a standard, non-negatively constrained, least squares
problem with a matrix right hand side. Therefore the most common way of solving it is to
use an alternating least squares (ALS) procedure:
–Guess an initial value W(1).
