Download Statistical Image Segmentation: Object Category Discovery and Segmentation and more Papers Art in PDF only on Docsity!
Shared Segmentation of Natural Scenes
Using Dependent Pitman-Yor Processes
Erik B. Sudderth and Michael I. Jordan Electrical Engineering & Computer Science, University of California, Berkeley [email protected], [email protected]
PROCEEDINGS OF NEURAL INFORMATION PROCESSING SYSTEMS 2008
Abstract
We develop a statistical framework for the simultaneous, unsupervised segmenta- tion and discovery of visual object categories from image databases. Examining a large set of manually segmented scenes, we show that object frequencies and segment sizes both follow power law distributions, which are well modeled by the Pitman–Yor (PY) process. This nonparametric prior distribution leads to learning algorithms which discover an unknown set of objects, and segmentation methods which automatically adapt their resolution to each image. Generalizing previ- ous applications of PY processes, we use Gaussian processes to discover spatially contiguous segments which respect image boundaries. Using a novel family of variational approximations, our approach produces segmentations which compare favorably to state-of-the-art methods, while simultaneously discovering categories shared among natural scenes.
1 Introduction
Images of natural environments contain a rich diversity of spatial structure at both coarse and fine scales. We would like to build systems which can automatically discover the visual categories (e.g., foliage, mountains, buildings, oceans) which compose such scenes. Because the “objects” of interest lack rigid forms, they are poorly suited to traditional, fixed aspect detectors. In simple cases, topic models can be used to cluster local textural elements, coarsely representing categories via a bag of visual features [ 1 , 2 ]. However, spatial structure plays a crucial role in general scene interpretation [ 3 ], particularly when few labeled training examples are available.
One approach to modeling additional spatial dependence begins by precomputing one, or several, segmentations of each input image [ 4 – 6 ]. However, low-level grouping cues are often ambiguous, and fixed partitions may improperly split or merge objects. Markov random fields (MRFs) have been used to segment images into one of several known object classes [ 7 , 8 ], but these approaches require manual segmentations to train category-specific appearance models. In this paper, we instead develop a statistical framework for the unsupervised discovery and segmentation of visual object categories. We approach this problem by considering sets of images depicting related natural scenes (see Fig. 1 (a)). Using color and texture cues, our method simultaneously groups dense features into spatially coherent segments, and refines these partitions using shared appearance models. This extends the cosegmentation framework [ 9 ], which matches two views of a single object instance, to simultaneously segment multiple object categories across a large image database. Some recent work has pursued similar goals [ 6 , 10 ], but robust object discovery remains an open challenge.
Our models are based on the Pitman–Yor (PY) process [ 11 ], a nonparametric Bayesian prior on infinite partitions. This generalization of the Dirichlet process (DP) leads to heavier-tailed, power law distributions for the frequencies of observed objects or topics. Using a large database of manual scene segmentations, Sec. 2 demonstrates that PY priors closely match the true distributions of natural segment sizes, and frequencies with which object categories are observed. Generalizing the hierarchical DP [ 12 ], Sec. 3 then describes a hierarchical Pitman–Yor (HPY) mixture model which shares “bag of features” appearance models among related scenes. Importantly, this approach coherently models uncertainty in the number of object categories and instances.
100 101 102
10 −
10 −
10 −
10 −
100
Segment Labels (sorted by frequency)
Proportion of forest Segments
Segment Labels PY(0.39,3.70) DP(11.40)
(^1010) −2 10 −1 100 0
101
102
103
Proportion of Image Area
Number of forest Segments
Segment Areas PY(0.02,2.20) DP(2.40)
(^01 2 3 4 5 6 7 )
20
40
60
80
100
120
Number of Segments per Image
Number of forest Images
Segment Counts PY(0.02,2.20) DP(2.40)
100 101 102
10 −
10 −
10 −
10 −
100
Segment Labels (sorted by frequency)
Proportion of insidecity Segments
(^) Segment Labels PY(0.47,6.90) DP(33.00)
(^1010) −2 10 −1 100 0
101
102
103
Proportion of Image Area
Number of insidecity Segments
(^) Segment Areas PY(0.32,0.80) DP(2.90)
(^01 2 3 4 5 6 7 )
20
40
60
80
100
120
Number of Segments per Image
Number of insidecity Images
Segment Counts PY(0.32,0.80) DP(2.90)
(a) (b) (c) (d) Figure 1: Validation of stick-breaking priors for the statistics of human segmentations of the forest (top) and insidecity (bottom) scene categories. We compare observed frequencies (black) to those predicted by Pitman– Yor process (PY, red circles) and Dirichlet process (DP, green squares) models. For each model, we also display 95% confidence intervals (dashed). (a) Example human segmentations, where each segment has a text label such as sky , tree trunk , car , or person walking. The full segmented database is available from LabelMe [ 14 ]. (b) Frequency with which different semantic text labels, sorted from most to least frequent on a log-log scale, are associated with segments. (c) Number of segments occupying varying proportions of the image area, on a log-log scale. (d) Counts of segments of size at least 5,000 pixels in 256 × 256 images of natural scenes.
As described in Sec. 4 , we use thresholded Gaussian processes to link assignments of features to regions, and thereby produce smooth, coherent segments. Simulations show that our use of contin- uous latent variables captures long-range dependencies neglected by MRFs, including intervening contour cues derived from image boundaries [ 13 ]. Furthermore, our formulation naturally leads to an efficient variational learning algorithm, which automatically searches over segmentations of varying resolution. Sec. 5 concludes by demonstrating accurate segmentation of complex images, and discovery of appearance patterns shared across natural scenes.
2 Statistics of Natural Scene Categories
To better understand the statistical relationships underlying natural scenes, we analyze manual seg- mentations of Oliva and Torralba’s eight categories [ 3 ]. A non-expert user partitioned each image into a variable number of polygonal segments corresponding to distinctive objects or scene elements (see Fig. 1 (a)). Each segment has a semantic text label, allowing study of object co-occurrence fre- quencies across related scenes. There are over 29,000 segments in the collection of 2,688 images.^1
2.1 Stick Breaking and Pitman–Yor Processes
The relative frequencies of different object categories, as well as the image areas they occupy, can be statistically modeled via distributions on potentially infinite∑ partitions. Let ϕ = (ϕ 1 , ϕ 2 , ϕ 3 ,.. .), ∞ k=1 ϕk^ = 1, denote the probability mass associated with each subset. In nonparametric Bayesian statistics, prior models for partitions are often defined via a stick-breaking construction:
ϕk = wk
k∏− 1
ℓ=
(1 − wℓ) = wk
k∑− 1
ℓ=
ϕℓ
wk ∼ Beta(1 − γa, γb + kγa) (1)
This Pitman–Yor (PY) process [ 11 ], denoted by ϕ ∼ GEM(γa, γb), is defined by two hyperparam- eters satisfying 0 ≤ γa < 1 , γb > −γa. When γa = 0, we recover a Dirichlet process (DP) with concentration parameter γb. This construction induces a distribution on ϕ such that subsets with more mass ϕk typically have smaller indexes k. When γa > 0 , E[wk] decreases with k, and the resulting partition frequencies follow heavier-tailed, power law distributions.
While the sequences of beta variables underlying PY processes lead to infinite partitions, only a random, finite subset of size Kε =
∣{k | ϕk > ε}
∣ (^) will have probability greater than any threshold ε.
Implicitly, nonparametric models thus also place priors on the number of latent classes or objects. (^1) See LabelMe [ 14 ]: http://labelme.csail.mit.edu/browseLabelMe/spatial envelope 256x256 static 8outdoorcategories.html
xji
tji T
kjt
D J
k
Nj J
f
f
U
vjt wk
(^00) 0.2 0.4 0.6 0.8 1
1
2
3
4
5
6
Stick−Breaking Proportion
Probability Density (^00) 0.2 0.4 0.6 0.8 1
1
2
3
4
5
6
Stick−Breaking Proportion
Probability Density (^00) 0.2 0.4 0.6 0.8 1
1
2
3
4
5
6
Stick−Breaking Proportion
Probability Density
GEM(0, 10) GEM(0. 1 , 2) GEM(0. 5 , 5)
−4^0 −2 0 2 4
Stick−Breaking Threshold
Probability Density −4^0 −2 0 2 4
Stick−Breaking Threshold
Probability Density −4^0 −2 0 2 4
Stick−Breaking Threshold
Probability Density
Figure 2: Stick-breaking representation of a hierarchical Pitman–Yor (HPY) model for J groups of features. Left: Directed graphical model in which global category frequencies ϕ ∼ GEM(γ) are constructed from stick- breaking proportions wk ∼ Beta(1 − γa, γb + kγa), as in Eq. ( 1 ). Similarly, vjt ∼ Beta(1 − αa, αb + tαa) define region areas πj ∼ GEM(α) for image j. Each of the Nj features xji is independently sampled as in Eq. ( 2 ). Upper right: Beta distributions from which stick proportions wk are sampled for three different PY processes: k = 1 (blue), k = 10 (red), k = 20 (green). Lower right: Corresponding distributions on thresholds for an equivalent generative model employing zero mean, unit variance Gaussians (dashed black). See Sec. 4.1.
3.2 Variational Learning for HPY Mixture Models
To allow efficient learning of HPY model parameters from large image databases, we have devel- oped a mean field variational method which combines and extends previous approaches for DP mixtures [ 21 , 22 ] and finite topic models. Using the stick-breaking representation of Fig. 2 , and a factorized variational posterior, we optimize the following lower bound on the marginal likelihood:
log p(x | α, γ, ρ) ≥ H(q) + Eq [log p(x, k, t, v, w, θ | α, γ, ρ)] (3)
q(k, t, v, w, θ) =
[ K
k=
q(wk | ωk)q(θk | ηk)
]
∏^ J
j=
[ T
t=
q(vjt | νjt)q(kjt | κjt)
] (^) Nj ∏
i=
q(tji | τji)
Here, H(q) is the entropy. We truncate the variational posterior [ 21 ] by setting q(vjT = 1) = 1 for each image or group, and q(wK = 1) = 1 for the shared global clusters. Multinomial assignments q(kjt | κjt), q(tji | τji), and beta stick proportions q(wk | ωk), q(vjt | νjt), then have closed form update equations. To avoid bias, we sort the current sets of image segments, and global categories, in order of decreasing aggregate assignment probability after each iteration [ 22 ].
4 Segmentation with Spatially Dependent Pitman–Yor Processes
We now generalize the HPY image segmentation model of Fig. 2 to capture spatial dependencies. For simplicity, we consider a single-image model in which features xi are assigned to regions by indicator variables zi, and each segment k has its own appearance parameters θk (see Fig. 3 ). As in Sec. 3.1, however, this model is easily extended to share appearance parameters among images.
4.1 Coupling Assignments using Thresholded Gaussian Processes
Consider a generative model which partitions data into two clusters via assignments zi ∈ { 0 , 1 } sampled such that P[zi = 1] = v. One representation of this sampling process first generates a Gaussian auxiliary variable ui ∼ N (0, 1), and then chooses zi according to the following rule:
zi =
1 if ui < Φ−^1 (v) 0 otherwise
Φ(u) ,
2 π
∫ (^) u
−∞
e−s
(^2) / 2 ds (4)
Here, Φ(u) is the standard normal cumulative distribution function (CDF). Since Φ(ui) is uniformly distributed on [0, 1], we immediately have P[zi = 1] = P
[
ui < Φ−^1 (v)
]
= P[Φ(ui) < v] = v.
We adapt this idea to PY processes using the stick-breaking representation of Eq. ( 1 ). In particu- lar, we note that if zi ∼ π where πk = vk
∏k− 1 ℓ=1 (1^ −^ vℓ), a simple induction argument shows that vk = P[zi = k | zi 6 = k − 1 ,... , 1]. The stick-breaking proportion vk is thus the conditional prob- ability of choosing cluster k, given that clusters with indexes ℓ < k have been rejected. Combining
x 1 6
,
k
B
B
7
vk
x 2
x 3 x 4
z 1 z 2
z 3 z 4
uk3 uk
uk1 uk2 (^5) 1 (^5) 2 (^5) 3 (^5) 4 (^5) 1 (^5) 2 (^5) 3 (^5) 4
u 1
u 2
u 3
Figure 3: A nonparametric Bayesian approach to image segmentation in which thresholded Gaussian processes generate spatially dependent Pitman–Yor processes. Left: Directed graphical model in which expected segment areas π ∼ GEM(α) are constructed from stick-breaking proportions vk ∼ Beta(1 − αa, αb + kαa). Zero mean Gaussian processes (uki ∼ N (0, 1)) are cut by thresholds Φ−^1 (vk ) to produce segment assignments zi, and thereby features xi. Right: Three randomly sampled image partitions (columns), where assignments (bottom, color-coded) are determined by the first of the ordered Gaussian processes uk to cross Φ−^1 (vk ).
this insight with Eq. ( 4 ), we can generate samples zi ∼ π as follows:
zi = min
k | uki < Φ−^1 (vk)
where uki ∼ N (0, 1) and uki ⊥ uℓi, k 6 = ℓ (5)
As illustrated in Fig. 3 , each cluster k is now associated with a zero mean Gaussian process (GP) uk, and assignments are determined by the sequence of thresholds in Eq. ( 5 ). If the GPs have identity covariance functions, we recover the basic HPY model of Sec. 3.1. More general covariances can be used to encode the prior probability that each feature pair occupies the same segment. Intuitively, the ordering of segments underlying this dependent PY model is analogous to layered appearance models [ 23 ], in which foreground layers occlude those that are farther from the camera.
To retain the power law prior on segment sizes justified in Sec. 2.3, we transform priors on stick proportions vk ∼ Beta(1 − αa, αb + kαa) into corresponding random thresholds:
p(¯vk | α) = N (¯vk | 0 , 1) · Beta(Φ(¯vk) | 1 − αa, αb + kαa) ¯vk , Φ−^1 (vk) (6)
Fig. 2 illustrates the threshold distributions corresponding to several different PY stick-breaking priors. As the number of features N becomes large relative to the GP covariance length-scale, the proportion assigned to segment k approaches πk, where π ∼ GEM(αa, αb) as desired.
4.2 Variational Learning for Dependent PY Processes
Substantial innovations are required to extend the variational method of Sec. 3.2 to the Gaussian pro- cesses underlying our dependent PY processes. Complications arise due to the threshold assignment process of Eq. ( 5 ), which is “stronger” than the likelihoods typically used in probit models for GP classification, as well as the non-standard threshold prior of Eq. ( 6 ). In the simplest case, we place factorized Gaussian variational posteriors on thresholds q(¯vk) = N (¯vk | νk, δk) and assignment surfaces q(uki) = N (uki | μki, λki), and exploit the following key identities:
Pq [uki < v¯k] = Φ
νk − μki √ δk + λki
Eq [log Φ(¯vk)] ≤ log Eq [Φ(¯vk)] = log Φ
νk √ 1 + δk
The first expression leads to closed form updates for Dirichlet appearance parameters q(θk | ηk), while the second evaluates the beta normalization constants in Eq. ( 6 ). We then jointly optimize each layer’s threshold q(¯vk) and assignment surface q(uk), fixing all other layers, via backtracking conjugate gradient (CG) with line search. For details and further refinements, see [ 17 ].
Figure 5: Segmentation results for two images (rows) from each of the coast , mountain , and tallbuilding scene categories. From left to right, columns show LabelMe human segments, image with boundaries inferred by PY-Edge, and segments for PY-Edge, PY-Dist, PY-BOF, NCut(3), NCut(4), and NCut(6). Best viewed in color.
0.2 0.2 0.4 0.6 0.8 1
1
Normalized Cuts
PY Gaussian (Edge Covar) 0.5 2 4 6 8 10
1
Number of Normalized Cuts Regions
Average Rand Index
Normalized Cuts PY Gaussian (Edge Covar) PY Gaussian (Distance Covar) PY Bag of Features
0.2 0.2 0.4 0.6 0.8 1
1
Normalized Cuts
PY Gaussian (Edge Covar) 0.5 2 4 6 8 10
1
Number of Normalized Cuts Regions
Average Rand Index
Normalized Cuts PY Gaussian (Edge Covar) PY Gaussian (Distance Covar) PY Bag of Features
(a) (b) (c) (d) Figure 6: Quantitative comparison of segmentation results to human segments, using the Rand index. (a) Scat- ter plot of PY-Edge and NCut(4) Rand indexes for 200 mountain images. (b) Average Rand indexes for moun- tain images. We plot the performance of NCut(K) versus the number of segments K, compared to the variable resolution segmentations of PY-Edge, PY-Dist, and PY-BOF. (c) Scatter plot of PY-Edge and NCut(6) Rand indexes for 200 tallbuilding images. (d) Average Rand indexes for tallbuilding images.
ance models to be flexibly shared among natural scenes, and leads to efficient variational inference algorithms which automatically search over segmentations of varying resolution. We believe this provides a promising starting point for discovery of shape-based visual appearance models, as well as weakly supervised nonparametric learning in other, non-visual application domains.
Acknowledgments We thank Charless Fowlkes and David Martin for the Pb boundary estimation and seg- mentation code, Antonio Torralba for helpful conversations, and Sra. Barriuso for her image labeling expertise. This research supported by ONR Grant N00014-06-1-0734, and DARPA IPTO Contract FA8750-05-2-0249.
References
[1] L. Fei-Fei and P. Perona. A Bayesian hierarchical model for learning natural scene categories. In CVPR , volume 2, pages 524–531, 2005. [2] J. Sivic, B. C. Russell, A. A. Efros, A. Zisserman, and W. T. Freeman. Discovering objects and their location in images. In ICCV , volume 1, pages 370–377, 2005. [3] A. Oliva and A. Torralba. Modeling the shape of the scene: A holistic representation of the spatial envelope. IJCV , 42(3):145–175, 2001.
Figure 7: Most significant segments associated with each of three shared, global visual categories (rows) for hierarchical PY-Edge models trained with 200 images of mountain (left) or tallbuilding (right) scenes.
[4] L. Cao and L. Fei-Fei. Spatially coherent latent topic model for concurrent object segmentation and classification. In ICCV , 2007. [5] B. C. Russell, A. A. Efros, J. Sivic, W. T. Freeman, and A. Zisserman. Using multiple segmentations to discover objects and their extent in image collections. In CVPR , volume 2, pages 1605–1614, 2006. [6] S. Todorovic and N. Ahuja. Learning the taxonomy and models of categories present in arbitrary images. In ICCV , 2007. [7] X. He, R. S. Zemel, and M. A. Carreira-Perpi˜n´an. Multiscale conditional random fields for image labeling. In CVPR , volume 2, pages 695–702, 2004. [8] J. Verbeek and B. Triggs. Region classification with Markov field aspect models. In CVPR , 2007. [9] C. Rother, V. Kolmogorov, T. Minka, and A. Blake. Cosegmentation of image pairs by histogram match- ing: Incorporating a global constraint into MRFs. In CVPR , volume 1, pages 993–1000, 2006. [10] M. Andreetto, L. Zelnik-Manor, and P. Perona. Non-parametric probabilistic image segmentation. In ICCV , 2007. [11] J. Pitman and M. Yor. The two-parameter Poisson–Dirichlet distribution derived from a stable subordina- tor. Ann. Prob. , 25(2):855–900, 1997. [12] Y. W. Teh, M. I. Jordan, M. J. Beal, and D. M. Blei. Hierarchical Dirichlet processes. J. Amer. Stat. Assoc. , 101(476):1566–1581, December 2006. [13] C. Fowlkes, D. Martin, and J. Malik. Learning affinity functions for image segmentation: Combining patch-based and gradient-based approaches. In CVPR , volume 2, pages 54–61, 2003. [14] B. C. Russell, A. Torralba, K. P. Murphy, and W. T. Freeman. LabelMe: A database and web-based tool for image annotation. IJCV , 77:157–173, 2008. [15] S. Goldwater, T. L. Griffiths, and M. Johnson. Interpolating between types and tokens by estimating power-law generators. In NIPS 18 , pages 459–466. MIT Press, 2006. [16] Y. W. Teh. A hierarchical Bayesian language model based on Pitman–Yor processes. In Coling/ACL ,
[17] E. B. Sudderth and M. I. Jordan. Shared segmentation of natural scenes using dependent Pitman-Yor processes. Technical report, Dept. of Statistics, University of California, Berkeley. In preparation, 2009. [18] X. Ren and J. Malik. Learning a classification model for segmentation. In ICCV , 2003. [19] Z. Tu and S. C. Zhu. Image segmentation by data-driven Markov chain Monte Carlo. IEEE Trans. PAMI , 24(5):657–673, May 2002. [20] D. R. Martin, C. C. Fowlkes, and J. Malik. Learning to detect natural image boundaries using local brightness, color, and texture cues. IEEE Trans. PAMI , 26(5):530–549, May 2004. [21] D. M. Blei and M. I. Jordan. Variational inference for Dirichlet process mixtures. Bayes. Anal. , 1(1):121– 144, 2006. [22] K. Kurihara, M. Welling, and Y. W. Teh. Collapsed variational Dirichlet process mixture models. In IJCAI 20 , pages 2796–2801, 2007. [23] J. Y. A. Wang and E. H. Adelson. Representing moving images with layers. IEEE Trans. IP , 3(5):625– 638, September 1994. [24] J. A. Duan, M. Guindani, and A. E. Gelfand. Generalized spatial Dirichlet process models. Biometrika , 94(4):809–825, 2007. [25] C. Fern´andez and P. J. Green. Modelling spatially correlated data via mixtures: A Bayesian approach. J. R. Stat. Soc. B , 64(4):805–826, 2002. [26] M. A. T. Figueiredo. Bayesian image segmentation using Gaussian field priors. In CVPR Workshop on Energy Minimization Methods in Computer Vision and Pattern Recognition , 2005. [27] M. W. Woolrich and T. E. Behrens. Variational Bayes inference of spatial mixture models for segmenta- tion. IEEE Trans. MI , 25(10):1380–1391, October 2006. [28] P. Orbanz and J. M. Buhmann. Smooth image segmentation by nonparametric Bayesian inference. In ECCV , volume 1, pages 444–457, 2006. [29] R. D. Morris, X. Descombes, and J. Zerubia. The Ising/Potts model is not well suited to segmentation tasks. In IEEE DSP Workshop , 1996. [30] R. Unnikrishnan, C. Pantofaru, and M. Hebert. Toward objective evaluation of image segmentation algo- rithms. IEEE Trans. PAMI , 29(6):929–944, June 2007.
