Optimization in Continuous Time
Jesús Fernández-Villaverde
Unive rsity of Pe nnsylv ania
November 9, 2013
Jesús Fern ández- Villaver de (PENN ) Optimi zation in C ontinuo us Time Novem ber 9, 201 3 1 / 28
Prepare for your exams
Study with the several resources on Docsity
Earn points to download
Earn points by helping other students or get them with a premium plan
Guidelines and tips
Prepare for your exams
Study with the several resources on Docsity
Earn points to download
Earn points by helping other students or get them with a premium plan
Optimization in continuous time, both in deterministic and stochastic environments. It presents three approaches: calculus of variations, optimal control, and dynamic programming, focusing on the last two. necessary and sufficient conditions for the optimal control problem and discusses the solution of the Hamilton-Jacobi-Bellman equation for the dynamic programming approach. It also includes examples of consumption-savings problems and a comparison between the deterministic and stochastic cases. useful for students studying optimization and control theory.
Typology: Study notes
2022/2023
1 / 28
This page cannot be seen from the preview
Don't miss anything!
Related documents
Partial preview of the text
Download Optimization in Continuous Time and more Study notes Calculus in PDF only on Docsity!
Optimization in Continuous Time
Jes˙s Fern·ndez-Villaverde
University of Pennsylvania
November 9, 2013
Motivation
Three Approaches
We are interested in optimization in continuous time, both in
deterministic and stochastic environments.
Elegant and powerful math (di§erential equations, stochastic
processes...).
Three approaches:
1
Calculus of Variations.
2 Optimal Control.
3
Dynamic Programming.
We will focus on the last two:
1 Optimal control can do everything economists need from calculus of
variations.
2
Dynamic programming is better for the stochastic case.
Deterministic Case
Maximization Problem II
Admissible pair: (x (t) , y (t)) s.t. the previous conditions are
satisÖed.
Optimal pair: (bx (t) , yb (t)) that reach V ( 0 , x ( 0 )) < ∞.
Then:
V ( 0 , x ( 0 )) =
Z
∞
0
f (t, bx (t) , by (t)) dt
Two di¢ culties:
1 We need to Önd a whole function y (t) of optimal choices.
2
The constraint is in the form of a di§erential equation.
Deterministic Case Optimal Control
Optimal Control
Pontryagin and co-authors.
Principle of Optimality
If (
bx (
t )
, by (
t ))
is an optimal pair, then:
V (t 0
, x (t 0
Z
t 1
t 0
f (t, bx (t) , by (t)) dt + V (t 1
, bx (t 1
for all t
1
t
0
We will assume that there is an optimal path.
Proving existence is, however, not a trivial task.
Deterministic Case Optimal Control
Exponential Discounting Case I
More speciÖc form:
V (x ( 0 )) = max
x (t ),y (t )
Z
∞
0
e
ρ t
f (x (t) , y (t)) dt
s.t. x˙ = g (x (t) , y (t))
x ( 0 ) = x
0
, lim
t !∞
b (t) x (t) x
1
x (
t )
2 IntX , y (
t )
2 IntY
g (x (t) , y (t)) being autonomous is not needed but it helps to
simplify notation.
Deterministic Case Optimal Control
Exponential Discounting Case II
Hamiltonian:
H (t, x (t) , y (t) , λ (t)) = e
ρ t
f (x (t) , y (t)) + λ (t) g (x (t) , y (t))
= e
ρ t
[f (x (t) , y (t)) + μ (t) g (x (t) , y (t))]
where
μ (t) = e
ρ t
λ (t)
Current-Value Hamiltonian:
b
H (x (t) , y (t) , μ (t)) = f (x (t) , y (t)) + μ (t) g (x (t) , y (t))
Deterministic Case Optimal Control
Su¢ ciency Conditions
Previous theorem only delivers necessary conditions.
However, we also need su¢ cient conditions.
Theorem
Mangasarian Su¢ cient Conditions for Discounted InÖnite-Horizon
Problems. The necessary conditions will be su¢ cient if f and g are
continuously di§erentiable and weakly monotone and
H
t, x (
t )
, y (
t )
, λ (
t ))
is jointly concave in x (
t )
and y (
t )
for 8 t 2 R
We will skip su¢ ciency arguments. They will be relevant later in
models of endogenous growth.
Deterministic Case Optimal Control
Example I
Consumption-savings problem:
V (a) = max
a,c
Z
∞
0
e
ρ t
u (c) dt
a ˙ = ra + w c
Hamiltonian:
b
H (
a, c, μ ) =
u (
c ) +
μ (
ra + w c )
Necessary conditions:
b
H
c
(a, c, μ ) = 0 ) u
0
(c) μ = 0 ) u
0
(c) = μ
b
H
a
(a, c, μ ) = ρμ μ ˙ ) r μ = ρμ μ ˙ )
μ ˙
μ
= (r ρ )
Deterministic Case Dynamic Programming
Dynamic Programming
Dynamic programming is a more áexible approach (for example, later,
to introduce uncertainty).
Instead of searching for an optimal path, we will search for decision
rules.
Cost: we will need to solve for PDEs instead of ODEs.
But at the end, we will get the same solution.
Deterministic Case Dynamic Programming
Hamilton-Jacobi-Bellman (HJB) Equation
When V (t, x (t)) is di§erentiable, (bx (t) , by (t)) satisÖes:
f (t, bx (t) , by (t)) +
V (t, bx (t)) + V
x
(t, bx (t)) g (t, bx (t) , by (t)) = 0
Similar the Euler equation from a value function in discrete time.
Other way to write the formula, closer to the Bellman equation:
V (t, bx (t)) = max
x (t ),y (t )
f (t, x (t) , y (t)) + g (t, x (t) , y (t)) V
x
(t, x (t))
Tight connection between V x
(t, x (t)) and μ (t).
Deterministic Case Dynamic Programming
Solution of the HJB Equation II
Then:
V (t, bx (t)) = ρ e
ρ t
Z
∞
t
e
ρ (s t )
f (bx (s) , by (s)) ds
= ρ
Z
∞
t
e
ρ s
f (bx (s) , by (s)) ds
= ρ V (t, bx (t))
SimplyÖng notation:
ρ V (x ) = max
x ,y
f (x, y ) + g (x, y ) V
0
(x )
Deterministic Case Dynamic Programming
Solution of the HJB Equation III
Characterized by a necessary condition:
f
y
(x, y ) + g
y
(x, y ) V
0
(x ) = 0
and an envelope condition:
( ρ g
x
(x, y )) V
0
(x ) f
x
(x, y ) = g (x, y ) V
00
(x )
Then:
V
0
(x ) =
f y
(x, y )
g y
(x, y )
= h (x, y )
and
V
00
(x ) = h
x
(x, y ) + h
y
(x, y )
dy
dx
Deterministic Case Dynamic Programming
Solution of the HJB Equation V
With our previous example:
ρ V (a) = max
a,c
u (c) + (ra + w c) V
0
(a)
Then:
h (a, c) = u
0
(c)
and:
(r ρ ) u
0
(c) = u
00
(c)
dc
da
a˙ = u
00
(c) c˙
Therefore, as before:
u
00
(c)
u
0
(c)
c˙ = (r ρ )
Deterministic Case Dynamic Programming
Comparison with Discrete Time
HJB versus Bellman equation:
ρ V (a) = max
a,c
u (c) + (ra + w c) V
0
(a)
V (a) = max
a,c
u (c) + β V (( 1 + r ) a + w c)
Optimality conditions:
u
00
(c)
u
0
(c)
c˙ = (r ρ )
u
0
(c
0
u
0
(c)
= β ( 1 + r )