Dynamic Programming in Continuous Time, Study notes of Calculus

Download Dynamic Programming in Continuous Time and more Study notes Calculus in PDF only on Docsity!

Dynamic programming in continuous time

Jes´us Fern´andez-Villaverde^1 and Galo Nu˜no^2 October 15, 2021 (^1) University of Pennsylvania

(^2) Banco de Espa˜na

Basic ideas

Why dynamic programming in continuous time?

• Continuous time methods transform optimal control problems into partial differential equations (PDEs): 1. The Hamilton-Jacobi-Bellman equation, the Kolmogorov Forward equation, the Black-Scholes equation,... they are all PDEs. 2. Solving these PDEs turns out to be much simpler than solving the Bellman or the Chapman-Kolmogorov equations in discrete time. Also, much knowledge of PDEs in natural sciences and applied math. 3. Key role of typical sets in the “curse of dimensionality.”
• Dynamic programming is a convenient framework:
  1. It can do everything economists could get from calculus of variations.
  2. It is better than Hamiltonians for the stochastic case.

The development of “continuous-time methods”

• Differential calculus introduced in the 17th century by Isaac Newton and Gottfried Wilhelm Leibniz.
• In the late 19th century and early 20th century, it was extended to accommodate stochastic processes (“stochastic calculus”). - Thorvald N. Thiele (1880): Introduces the idea of Brownian motion. - Louis Bachelier (1900): Formalizes the Brownian motion and applies to the stock market. - Albert Einstein (1905): A model of the motion of small particles suspended in a liquid. - Norbert Wiener (1923): Uses the ideas of measure theory to construct a measure on the path space of continuous functions. - Andrey Kolmogorov (1931): Diffusions depend on drift and volatility, Kolmogorov equations. - Wolfgang D¨oblin (1938-1940): Modern treatment of diffusions with a change of time. - Kiyosi Itˆo (1944): Itˆo’s Lemma. - Paul Malliavin (1978): Malliavin calculus. 3

The development of “dynamic programming”

• Calculus of variations: Issac Newton (1687), Johann Bernoulli (1696), Leonhard Euler (1733), Joseph-Louis Lagrange (1755).
• 1930s and 1940s: many problems in aerospace engineering are hard to tackle with calculus of variations. Example: minimum time interception problems for fighter aircraft.
• Closely related to the Cold War.
• Lev S. Pontryagin, Vladimir G. Boltyanskii, and Revaz V. Gamkrelidze (1956): Maximum principle.
• Magnus R. Hestenes, Rufus P. Isaacs, and Richard E. Bellman at RAND (1950s):
  1. Distinction between controls and states.
  2. Principle of optimality.
  3. Dynamic programming.

Figure 2. The mathematicians at Steklov: Lev Semyonovich Pontryagin,

Vladimir Grigor’evich Boltyanskii, and Revaz Valerianovich Gamkrelidze

Figure 1: Lev S. Pontryagin, Vladimir G. Boltyanskii, and Revaz V. Gamkrelidze

Hamilton-Jacobi-Bellman

William Hamilton (1805-1865) Carl Jacobi (1804-1851) Richard Bellman (1920-1984)

The Hamilton-Jacobi-Bellman equation

• If we define the value function:

V (t, x) = max {αs }s≥t

t

e−ρ(s−t)u (αs , xs ) ds,

then, under technical conditions, it satisfies the Hamilton-Jacobi-Bellman (HJB) equation:

ρVt (x) = ∂V ∂t

• max α

u (α, x) +

∑^ N

n=

μt,n (x, α) ∂V ∂xn

with a transversality condition limT →∞ e−ρT^ VT (x) = 0.

Example: consumption-savings problem

• A household solves: max {ct }t≥ 0

0

e−ρt^ log (ct ) dt,

subject to: dat dt =^ rat^ +^ y^ −^ ct^ ,^ a^0 = ¯a, where r and y are constants.

• The HJB is: ρV (a) = max c

log (c) + (ra + y − c) dV da

• Intuitive interpretation.

Example: solution

• We guess V (a) = (^1) ρ log ρ + (^) ρ^1

r ρ −^1

• (^) ρ^1 log

a + yr

• The first-order condition is: 1 c

dV da

ρ

a + yr

and hence:

c = ρ

Financial wealth ︷︸︸︷ a +

Human wealth ︷︸︸︷ y r

• Then, we verify the HJB:

ρV (a) = log

ρ

a + y r

ra + y − ρ

a + y r

)) (^) dV da