Stochastic Dynamic Programming, Summaries of Algebra

Download Stochastic Dynamic Programming and more Summaries Algebra in PDF only on Docsity!

Stochastic Dynamic Programming

Jesus Fernandez-Villaverde

University of Pennsylvania

Introducing Uncertainty in Dynamic Programming

 Stochastic dynamic programming presents a very exible framework to handle multitude of problems in economics.

 We generalize the results of deterministic dynamic programming.

 Problem: taking care of measurability.

Environment

 (X; X ): universally measurable space for the endogenous state.

 (Z; Z): universally measurable space for the exogenous state.

 (S; S): (X; X )  (Z; Z) :

 Q: stationary transition function for (Z; Z).

 : X  Z! X: correspondence constraint.

 A = f(x; y; z) 2 X  X  Z : y 2 (x; z)g: graph of .

 F : A! R: one-period return function.

 : discount factor.

Plans

 t : Zt^! X for t = 1; 2 ; :::: sequence of measurable functions.

  = ( 0 2 X; t): plan.

 Interpretation of a plan: contingent decision rules.

 A plan  is feasible from s 0 2 S if:

1.  0 2 (s 0 ) :
2. t 2

 t 1

 zt^1

 ; zt

 for zt^2 Zt, t = 1; 2 ; :::

  (s 0 ): set of all feasible plans from s 0 2 S:

 If  does not depend on t but only on zt, we call the plan stationary or Markov.

Some Preliminary Results II

 Given Q on (Z; Z) and s 0 2 S,

t^ (z 0 ;
) : Zt^! [0; 1] , t = 1; 2 ; :::

 Assumption 2: F : A! R is Ameasurable and either (a) or (b)

holds: a. F  0 or F  0 : b. For each (x 0 ; z 0 ) = s 0 2 S and each plan  2  (s 0 ), F

 t 1

 zt^1

 ; t

 zt

 ; zt

 is t^ (z 0 ;
) integrable, t = 1; 2 ; ::: and the limit:

F (x 0 ;  0 ; z 0 )+ lim t!

X^1 t=

Z Zt

tF  t 1  zt 1  ; t^  zt ; zt^  t (^) (z 0 ;
)

exists (though it may be plus or minus in nity).

Sequential Problem

 De ne un (
; s 0 ) :  (s 0 )! R; n = 0; 1 ; ::: by:

u 0 (; s 0 ) = F (x 0 ;  0 ; z 0 ) un (; s 0 ) = F (x 0 ;  0 ; z 0 )

X^ n t=

Z Zt

tF  t 1  zt 1  ; t^  zt ; zt^  t  z 0 ; dzt

 De ne u (; s 0 ) :  (s 0 )! R 1 by

u (; s 0 ) = (^) nlim!1un (; s 0 )

 De ne v^ : S! R 1 by

v^ (s) = sup  2 (s)

u (; s 0 )

Transversality Condition

 In general, dynamic programming problems require two boundary con- ditions: an initial condition and a nal condition.

 Transversality condition plays the role of the second condition.

 To ensure the equivalence of the sequential and recursive problem, we also need then a transversality condition:

t!1lim^ t^

Z v

 t 1

 zt^1

 ; zt

 t^

 z 0 ; dzt

 = 0; 8  2  (s 0 ) , s 0 2 S

Equivalence of Sequential and Recursive Problem

 Under our previous assumptions:

1. v = v
2. Any plan ^ generated by G obtains the supremum in v^ (s) = sup 2 (s) u (; s 0 )

 Under our previous assumptions and an additional boundness condi- tion, a plan is optimal only if it is generated a.e. by G:

 Our results are equivalent to theorems 4.2-4.5 in SLP for the deter- ministic case.

Results I

Under these assumptions, we can prove that:

1. The Bellman operator:

(T f ) (x; z) = sup y 2 (x;z)

 F (x; y; z) +

Z v

 y; z^0

 Q

 z; dz^0



has a unique xed point.

2. Contractivity: kT nv 0 vk  n^ kv 0 vk, n = 1; 2 ; :::
3. The policy correspondence

G (x; z) =

 y 2 (x; z) : v (x; z) = F (x; y; z) +

Z v

 y; z^0

 Q

 z; dz^0



is non-empty, compact-valued, and u.h.c.

4. The value function will inherit increasing properties from F and Q.

Concavity

 Assumption concavity 1: For each z 2 Z, F (
;
; z) : Az! R satis es:

F

  (x; y) + (1 )

 x^0 ; y^0

 ; z

  F (x; y; z) + (1 ) F

 x^0 ; y^0 ; z



8  2 (0; 1) , 8 (x; y) ;

 x^0 ; y^0

 2 Az and the inequality is strict if x 6 = x^0.

 Assumption concavity 2: For 8 z 2 Z and 8 x; x^0 2 X, y 2 (x; z) and y^0 2

x^0 ; z

y + (1 ) y^0 2

 x + (1 ) x^0 ; z

 ; 8  2 (0; 1)

Unbounded Returns

 What if returns, like in most applications of interest in economics, are unbounded?

 This was already an issue in the deterministic set-up.

 We can get most of the substance of previous results if F is constant returns to scale.

 In the case of CRRA utility functions, we would need to do some ad-hoc work.

Policy Functions and Transition Functions I

 Let us imagine that the decision maker follows g (x; z) given an initial condition s 0 :

 The policy function generates a sequence fstg :

 What do we know about fstg?

 Read chapters 11-14 of SLP.