Machine Learning (CS 5350/CS 6350) 03 Apr 2007
Bayesian inference
The general problem we face in Bayesian inference is to compute the expectation of some function with
respect to a probabilistic model with unknowns.
In the simplest case, we want the expectation of a single variable. In more complex cases, we want an
expectation of a complex function of all variables.
Suppose p(θ) is our distribution of interest (some θwill be known, some not). We want:
Z=Eθ∼phf(θ)i=Zdθp(θ)f(θ)
For some cases this integral will be available in closed form (eg., HMMs). For many (most?) cases, however,
it will not.
Lets say that θis discrete, univariate. Then we can compute the expectation by just summing over all
possible values. Obviously, though, this won’t scale well to high-dimensional or non-discrete variables. But
let’s see what happens if we try. . .
Integration by Summation
Suppose θis univariate, bounded continuous. Wlog, θ∈[0,1]. If we remember how we first learned
integration, we can break [0,1] into Requally-sized rectangles. Then, we have:
Z≈
R−1
X
i=0
1
Rp(i/R)f(i/R)
As R→ ∞,Zbecomes increasingly more accurate.
One way of thinking about this is that we have a set Scontaining R-many equally spaced points, and the
integral is approximated by:
Z≈1
|S|X
θ∈S
p(θ)f(θ)
Unfortunately, if θis D-dimensional, then we need to sum RDvalues of θ.
Uniform Sampling
Instead of spacing θ∈Sevenly, let’s space them randomly. This is the idea of “Monte Carlo” integration,
which essentially means “randomized” integration. Uniform sampling is the simplest case. Let Sb e a random
sampling of θs. Then, we still have:
Z≈1
|S|X
θ∈S
p(θ)f(θ)
1
