Dynamic Programming: Functional Equation, Sequence, and Bellman Equation Connection, Study notes of Environmental Science

Download Dynamic Programming: Functional Equation, Sequence, and Bellman Equation Connection and more Study notes Environmental Science in PDF only on Docsity!

Dynamic Programming Under Certainty Contd

Marek Kapicka, Econ 204b

April 1, 2009

Review

I (^) Functional Equation and Sequence Problem:

(SP) : v ∗(k 0 ) = max { 0 ≤kt+ 1 ≤f (kt )}∞ t= 0

∞

t= 0

β t^ U(f (kt ) − kt+ 1 ), k 0 given

(FE ) : v (k) = max 0 ≤y ≤f (k)

U(f (k) − y ) + β v (y )

I (^) Today

2. Connection between (SP) and (FE)
3. Bellman Equation as a Fixed Point

2.2b. A solution to (FE) satisfies (SP):

v (k 0 ) = max 0 ≤k 1 ≤f (k 0 )

[U(f (k 0 ) − k 1 ) + β v (k 1 )]

= max 0 ≤k 1 ≤f (k 0 )

{U(f (k 0 ) − k 1 )

• β max 0 ≤k 2 ≤f (k 1 )

[U(f (k 1 ) − k 2 ) + β v (k 2 )]}

= max { 0 ≤kt+ 1 ≤f (kt )}^1 t= 0

1

t= 0

β t^ [U(f (kt ) − kt+ 1 ) + β^2 v (k 2 )} .... = max { 0 ≤kt+ 1 ≤f (kt )}Tt= 0

T

t= 0

β t^ [U(f (kt ) − kt+ 1 ) + β T^ +^1 v (kT + 1 )}

2.2b. A solution to (FE) satisfies (SP): harder

Theorem

If lim T →∞

β T^ +^1 v (kT + 1 ) = 0 (1)

for all {kT + 1 } such that 0 ≤ kT + 1 ≤ f (kT ), then v satisfies (SP)

Proof.

If the condition is satisfied then the last term vanishes:

v (k 0 ) = (^) Tlim →∞ max { 0 ≤kt+ 1 ≤f (kt )}Tt= 0

T

t= 0

β t^ [U(f (kt ) − kt+ 1 )+ β T+^1 v (kT + 1 )}

= max { 0 ≤kt+ 1 ≤f (kt )}∞ t= 0

∞

t= 0

β t^ [U(f (kt ) − kt+ 1 )

2.3. A Relationship between optimal policies in (SP) and

(FE)

Theorem

If {k t∗+ 1 } attains the maximum of (SP) then k t∗+ 1 = g (k t∗ ) all t ≥ 0

Theorem

If T^ lim →∞ β T^ +^1 v^ (kT^ +^1 )^ ≤^0 then {g (k 0 ).g (g (k 0 )), ....} attains the maximum of (SP).

3. Bellman Equation as a Fixed Point

I (^) Let S be a space of functions. Define a Bellman operator T : S → S by

Tv (k) = max 0 ≤y ≤f (k)

U(f (k) − y ) + β v (y )

I (^) T maps a family of functions to a family of functions. I (^) The solution to (FE) v ∗^ is a fixed point of the operator T :

Tv ∗^ = v ∗

I (^) If we find a fixed point of the operator T , we have solved (FE) and, provided the boundedness condition 2 is satisfied, (SP)

3.1. An Example

I (^) u(c) = ln c, f (k) = k α^ (full depreciation)

1. STEP 1 Set v 0 = 0 : Tv 0 (k) = (^0) ≤maxy ≤k α ln(k α^ − y )

Solution:

g 0 (k) = 0 Tv 0 (k) = ln k α^ = α ln k

3.1. An Example

2. STEP 2

Set v 1 = Tv 0 = α ln k :

Tv 1 (k) = max 0 ≤y ≤k α^

ln(k α^ − y ) + αβ ln y

Solution:

g 1 (k) = (^1) + αβ αβ k α

Tv 1 (k) = α ( 1 + αβ ) ln k + ln

1 + αβ +^ αβ^ ln^

αβ 1 + αβ

3.1. An Example

∞ :THE LIMIT

v ∗^ = (^) slim→∞vs g ∗^ = (^) slim→∞gs

v ∗(k) =

α 1 − β ln^ k^ +^

1 − β [ln(^1 −^ αβ ) +^

αβ 1 − αβ ln^ αβ ] g ∗(k) = αβ k α

Hence, c∗(k) = k α^ − g ∗(k) = ( 1 − αβ )k α

I (^) Consume a fraction 1 − αβ of the resources, save αβ for the future.

3.1. An Example

Results

I (^) By iteration on the value function, we have found

1. The value function v ∗^ that solves (FE)
2. The optimal policy function g ∗(k) I (^) Note: The speed of convergence

α ln k α ( 1 + αβ ) ln k α ( 1 + αβ + ( αβ )^2 ) ln k ...

will hold more generally!

4.1. A General Approach - Mathematical Preliminaries

I (^) Assumption: F (x, y ) is bounded and continuous. I (^) The assumption suggests that the value function might also be bounded and continuous I (^) To study (FE) in general, we need some mathematical tools: I (^) Contraction Mapping Theorem: Will tell us under what conditions will an operator mapping the set of bounded and continuous functions onto itself have a unique fixed point. I (^) Theorem of Maximum: Will tell us under what conditions will the Bellman operator map the set of bounded and continuous functions onto itself.

4.1. A General Approach - Mathematical Preliminaries

I (^) Consider a set of bounded and continuous functions with a sup norm

S = { f : X → R, f is continuous ‖ f ‖= sup x ∈X

|f (x )| < ∞}

I (^) Define a metric

ρ (f , g ) =‖ f − g ‖= sup x ∈X

|f (x ) − g (x )|

Definition

A metric space (S, ρ ) is complete if every Cauchy sequence in S converges to an element in S.

4.1. Example: A set of functions that is not complete

I (^) Let S++^ be a set of bounded continuous functions that are strictly increasing I (^) Consider a sequence

fn(x ) = 1 + 1 1 + n

x x ∈ [0, 1]

I (^) {fn} is a sequence of functions I (^) {fn} is a Cauchy sequence: for any ε > 0,

ρ (fm, fn) = sup x ∈[0,1]

1 + m x^ −^1 −^

1 + n x

= |

1 + m −^

1 + n | = 1 1 + min(m, n)

< ε

for m, n ≥ N ε where N ε = (^1) ε − 1.

4.1. Example: A set of functions that is not complete

I (^) However, nlim→∞fn(x^ ) =^1 which is not strictly increasing. Hence S++^ is not complete.