Download Probabilistic Modeling - Lecture Notes | CS 5350 and more Study notes Computer Science in PDF only on Docsity!
Machine Learning (CS 5350/CS 6350) 13 Nov 2008
Probabilistic Modeling
The goal is probabilistic modeling is to set up a statistical model p(data | model) that “explains” our data (given a model) and then try to find the “best” model.
Suppose we believe there is a functional relationship y = w>x between some set of inputs x 1 ,... , xN and some set of outputs y 1 ,... , yn. However, in data that we observe, this relationship is corrupted by noise. That is, the y we observe are not precisely w>x, but is rather corrupted by some noise. We wish to model this noise stochastically; one choise is to say that the noise is Gaussian distributed. Namely, y = w>x + and ∼ Nor(0, σ^2 ).
Here, ∼ Nor(μ, σ^2 ) denotes that is drawn from a Gaussian (“Normal”) distribution with mean μ and variance σ^2. This Gaussian has density:
Nor(x | μ, σ^2 ) =
2 πσ^2
exp
[
2 σ^2 (x − μ)^2
]
To think about y = w>x + , think about a true linear relationship that is then slightly corrupted. Since we assume that the error model is Gaussian with mean zero, this is the same as saying y ∼ Nor(w>x, σ^2 ). (This is not hard to verify: if you don’t see it, try working it out.)
Now, we are ready to talk about p(data | model). Since the inputs x are always given to us, we do not need to explicitly model them. Thus, although our data are the pairs (xn, yn), the xs are always provided. Furthermore, our model (in this case) is totally specified by w. We can therefore consider something of the form:
p(y 1 ,... yN | x 1 ,... , xN , w) (2)
The first step is to apply the chain rule:
p(y 1 ,... yN | x 1 ,... , xN , w) =
∏^ N
n=
p(yn | x 1 ,... , xN , w, y 1 ,... , yn− 1 ) (3)
Next, we make an assumption: conditioned on xn and w, the ys are completely independent of eachother, and yn is independent of all xms for m 6 = n. Thus:
p(y 1 ,... yN | x 1 ,... , xN , w) =
∏^ N
n=
p(yn | x 1 ,... , xN , w, y 1 ,... , yn− 1 ) (4)
∏^ N
n=
p(yn | x 1 ,... , xN , w) (5)
∏^ N
n=
p(yn | xn, w) (6)
Now, we can substitute in our Gaussian model for p(yn | xn, w):
Probabilistic Modeling 2
∏^ N
n=
p(yn | xn, w) =
∏^ N
n=
Nor(yn | w>xn, σ^2 ) (7)
Our first method of “solving” this is by using the maximum likelihood estimator. Namely, we want to choose the w that maximizes the probability of the data given the model (probability of data given model is typically called the “likelihood”). Our standard methods for maximizing (or minimizing) require us to differentiate. Differentiating that product looks nasty. So instead of maximizing the likelihood, we will maximize the log likelihood
``(w) = log p(data | model) (8)
= log
∏^ N
n=
Nor(yn | w>xn, σ^2 ) (9)
∑^ N
n=
log Nor(yn | w>xn, σ^2 ) (10)
We can now plug in the definition of the Gaussian and do some simplification:
``(w) =
∑^ N
n=
log Nor(yn | w>xn, σ^2 ) (11)
∑^ N
n=
log
2 πσ^2
exp
[
2 σ^2
(yn − w>xn)^2
])
∑^ N
n=
log(2πσ^2 ) −
2 σ^2
(yn − w>xn)^2 (13)
Now, this looks slightly messy, but remember that we’re just treating this as a function of w. That means that the additive term in the front is irrelevant. The constant 1/(2σ^2 ) is also irrelevant. Thus, to maximize the likelihood, we could instead maximize −
n(yn^ −^ w
xn) (^2). Or, equivalently, minimize ∑ n(yn^ −^ w
xn).
(Note the missing negative sign.)
This is exactly the linear regression model we came up with before by trying to minimize squared error! One way to interpret this is that by choosing squared error as our loss function, we are implicitly assuming a Gaussian noise model.
We can do the same for classification.
We might want to make some model like y = sign[w>x] for a binary classification problem with y ∈ {− 1 , +1}. However, this is incapable of dealing with noise. Instead, we transform w>x via the sigmoid and use the sigmoid’s output as the probability that y is +1. In other words:
p(y = +1 | w, x) = σ(w>x) (14) p(y = − 1 | w, x) = 1 − σ(w>x) (15)
A convenient property of the sigmoid is that 1 − σ(z) = σ(−z). Thus, we get:
