Study with the several resources on Docsity
Earn points by helping other students or get them with a premium plan
Prepare for your exams
Study with the several resources on Docsity
Earn points to download
Earn points by helping other students or get them with a premium plan
A derivation of bayesian regression for linear models with nonlinear inputs and probabilistic approach. The bayes theorem, gaussian likelihood function, and calculation of mean and covariance matrix of the distribution of weights. It also includes an application to auto mpg data and 1-d data.
Typology: Study notes
1 / 38
This page cannot be seen from the preview
Don't miss anything!
Models (NonlinearInputs; Probabilistic) Chuck Anderson Bayesian Regression DerivationApplication to Auto MPG Data Application to 1-DData
Chuck Anderson
Department of Computer Science Colorado State University
Fall, 2009
Models (NonlinearInputs; Probabilistic) Chuck Anderson Bayesian Regression DerivationApplication to Auto MPG Data Application to 1-DData
Bayesian Regression Derivation Application to Auto MPG Data Application to 1-D Data
Models (NonlinearInputs; Probabilistic) Chuck Anderson Bayesian Regression DerivationApplication to Auto MPG Data Application to 1-DData
The full Bayesian approach to regression does not solve for a single w value. Instead, an expression for the probability of a model, given the data, is formulated. Then, a sum is taken over all possible models of the prediction value for a given model weighted by the probability of that model. So p(tn|xn, X, T) =
p(tn, model|xn, X, T) dmodel
=
p(tn|model, xn, X, T) p(model|X, T) dmodel
Models (NonlinearInputs; Probabilistic) Chuck Anderson Bayesian Regression DerivationApplication to Auto MPG Data Application to 1-DData
The full Bayesian approach to regression does not solve for a single w value. Instead, an expression for the probability of a model, given the data, is formulated. Then, a sum is taken over all possible models of the prediction value for a given model weighted by the probability of that model. So p(tn|xn, X, T) =
p(tn, model|xn, X, T) dmodel
=
p(tn|model, xn, X, T) p(model|X, T) dmodel
As before, let’s choose the model to be y (xn, w) = φ(xn)w + , where ∼ N (0, β−^1 ). With this choice p(tn|model, xn, X, T) = p(tn|w, φ, xn, X, T, β) = N (tn|φ(xn)w, β−^1 )
Models (NonlinearInputs; Probabilistic) Chuck Anderson Bayesian Regression DerivationApplication to Auto MPG Data Application to 1-DData
Bayes Theorem tells us
p(model|X, T) ∝ p(T|X, model)p(model)
or
p(w|φ, X, T, β) ∝ p(T|w, φ, X, β)p(w)
We will model the data likelihood function as a Gaussian,
p(T|w, φ, X, β) = N (T|Φw, β−^1 I )
=
n=
N (tn|φ(xn)w, β−^1 )
Models (NonlinearInputs; Probabilistic) Chuck Anderson Bayesian Regression DerivationApplication to Auto MPG Data Application to 1-DData
Bayes Theorem tells us
p(model|X, T) ∝ p(T|X, model)p(model)
or
p(w|φ, X, T, β) ∝ p(T|w, φ, X, β)p(w)
We will model the data likelihood function as a Gaussian,
p(T|w, φ, X, β) = N (T|Φw, β−^1 I )
=
n=
N (tn|φ(xn)w, β−^1 )
Again, choose the prior distribution of the weights to be the zero-mean Gaussian
p(w|α) = N (w| 0 , α−^1 I )
Models (NonlinearInputs; Probabilistic) Chuck Anderson Bayesian Regression DerivationApplication to Auto MPG Data Application to 1-DData
Now the original equation becomes
p(tn|xn, X, T) = ∫ N (tn|φ(xn)w, β−^1 )
n=
N (tn|φ(xn)w, β−^1 )
N (w| 0 , α−^1 I )dw
Let’s work on the last two terms in the integral. They involve products of Gaussians, which involve products of exponential terms with base e. These products are formed by adding the exponents, so let’s focus on the value of the exponent of e. The sum we get is
(β
n=
(tn−φ(xn)w)^2 + wT^ αI w)
Models (NonlinearInputs; Probabilistic) Chuck Anderson Bayesian Regression DerivationApplication to Auto MPG Data Application to 1-DData
Continuing working with the sum, we get
− 12 (β
X^ N n=
(tn − φ(xn)w)^2 + wT^ αI w)
= − 12 β
X^ N n=
t n^2 +
X^ N n=
tnφ(xn)w − 12 (
X^ N n=
φ(xn)w )^2 − 12 wT^ αI w
= − 12 βTT^ T + βTT^ Φw − 12 βwT^ ΦT^ Φw − 12 wT^ αI w
= − 12 wT^ (βΦT^ Φ + αI )w + wT^ βΦT^ T − 12 βTT^ T
Models (NonlinearInputs; Probabilistic) Chuck Anderson Bayesian Regression DerivationApplication to Auto MPG Data Application to 1-DData
Look at expression (2.71) in Bishop’s book and the text around it.
−
(x − μ)T^ Σ−^1 (x − μ) =
−
xT^ Σ−^1 x + xT^ Σ−^1 μ + const
Models (NonlinearInputs; Probabilistic) Chuck Anderson Bayesian Regression DerivationApplication to Auto MPG Data Application to 1-DData
Look at expression (2.71) in Bishop’s book and the text around it.
−
(x − μ)T^ Σ−^1 (x − μ) =
−
xT^ Σ−^1 x + xT^ Σ−^1 μ + const
Comparing with our previous expression
wT^ (βΦT^ Φ + αI )w + wT^ βΦT^ T −
βTT^ T
Models (NonlinearInputs; Probabilistic) Chuck Anderson Bayesian Regression DerivationApplication to Auto MPG Data Application to 1-DData
Unfortunately, with this definition, the linear terms do not match. To make them match, we can introduce Σ−^1 Σ = I at the right place, so
wT^ βΦT^ T becomes wT^ Σ−^1 ΣβΦT^ T
Models (NonlinearInputs; Probabilistic) Chuck Anderson Bayesian Regression DerivationApplication to Auto MPG Data Application to 1-DData
Unfortunately, with this definition, the linear terms do not match. To make them match, we can introduce Σ−^1 Σ = I at the right place, so
wT^ βΦT^ T becomes wT^ Σ−^1 ΣβΦT^ T
making the full exponent
wT^ (βΦT^ Φ + αI )w+
wT^ (βΦT^ Φ + αI )(βΦT^ Φ + αI )−^1 βΦT^ T− 1 2
βTT^ T
Models (NonlinearInputs; Probabilistic) Chuck Anderson Bayesian Regression DerivationApplication to Auto MPG Data Application to 1-DData
Renaming the mean and covariance matrix of this distribution of w to be
Sw = Σ = (βΦT^ Φ + αI )−^1 μw = μ = βSw ΦT^ T,
Models (NonlinearInputs; Probabilistic) Chuck Anderson Bayesian Regression DerivationApplication to Auto MPG Data Application to 1-DData
Renaming the mean and covariance matrix of this distribution of w to be
Sw = Σ = (βΦT^ Φ + αI )−^1 μw = μ = βSw ΦT^ T,
We have now identified the resulting Gaussian to be
N (μw , Sw ).
Our book uses variable names μN and SN for μw and Sw.
