Download Markov Chains: Contraction Approach and Limiting Probability Vector and more Study notes Operational Research in PDF only on Docsity!
IEOR 4106, Spring 2011, Professor Whitt
Class Lecture Notes: Thursday, February 10.
Markov Chains
The Contraction approach to π = πP
The limit for aperiodic irreducible finite-state DTMC’s. There is a nice simple limit for aperiodic irreducible finite-state Markov chains. For any initial probability vector u ≡ (u 1 ,... , um), the probability vector at time n is
P (Xn = j) = (uP n)j =
∑^ m
i=
uiP (^) i,jn.
The key limiting result is
Theorem 0.1 If P is the transition matrix of an aperiodic irreducible finite-state Markov chain with transition matrix P , then, for any initial probability vector u,
uP n^ → π as n → ∞ ,
where the limiting probability vector π is the unique stationary probability vector, i.e., the unique solution to the fixed-point equation
π = πP or πj =
∑^ m
i=
πiPi,j for all j ,
where πj ≥ 0 for all j and
j πj^ = 1. Note the conditions: Of course, irreducibility is essential. And aperiodicity is essential to get full convergence, as opposed to convergence of averages, or convergence through appropriate subsequences. The method of proof here is designed to apply to finite-state chains. The proof extends to infinite-state chains under the condition that there is some state j such that Pi,j ≥ ≤ > 0 for all states i, or P (^) i,jk ≥ ≤ > 0 for some k. This is a strong extra condition saying that there is a state j such that there is a probability of at least ≤ > 0 of going to j in one step (or in k steps, as a weaker version of the same condition), from any other state. With that extra condition, we not only get convergence, we get convergence quickly, geometrically fast. We actually provide a proof without this condition, but we do not get such quick convergence unless the condition holds.
The Contraction Proof.
One way to prove this result and others is to apply renewal theory. An alternative way to prove the theorem is to consider the transition matrix P as an operator on the space of all probability vectors, here taken to be of dimension m, corresponding to there being m states. An operator on a space maps the space into itself. If u is a probability vector, then P maps
u into the probability vector uP , corresponding to the probability vector starting with u and then taking one step according to P , i.e.,
(uP )j =
∑^ m
i=
uiPi,j for all j.
We want the underlying space to be a complete metric space and the operator to be a contrac- tion map. Then we can apply the Banach fixed-point theorem, also called the Banach-Picard fixed-point theorem or the contraction fixed-point theorem; see pages 220-221 from the blue Rudin book, Principles of Mathematical Analysis, posted on line. The proof can be done in two steps:
step 1. In the first step, you prove that some power of P has all positive entries (using the assump- tion that P is an m × m transition matrix of an irreducible aperiodic Markov chain).
Example 0.1 We remark at the outset that the worst case has P 1 , 2 = P 2 , 3 = · · · = Pm− 1 ,m = 1, while Pm, 1 > 0, Pm, 2 > 0 and Pm,j = 0 for all j ≥ 3. Note that, for this example, P 1 k, 1 > 0 for k = m, k = 2m, k = 2m−1, k = 3m, k = 3m−1, k = 3m−2, k = 4m, k = 4m−1, k = 4m−2, k = 4m − 3, and so forth. In this example, we have P 1 k, 1 > 0 for k = (m − 2)m, but P 1 k, 1 = 0 for k = (m − 2)m + 1, but then we have P 1 k, 1 > 0 for all k ≥ (m − 1)m − (m − 2) = m^2 − 2 m + 2.
Lemma 0.1 For any states i and j 6 = i, P (^) i,jk > 0 for some k, with 1 ≤ k ≤ m − 1.
Proof. We use the fact that the chain is irreducible. Let Si,k be the set of states reachable from state i in at most k steps, with Si, 0 ≡ {i}. By the irreducibility, the sets Si,k have to be strictly increasing in k for every k until Si,k = { 1 , 2 ,... , m}, the full state space. Otherwise, the DTMC would not be irreducible. Since there are only m − 1 other states, all these other states have to be reached in at most m − 1 steps. If the increase is by more than a single state, then the number k will be strictly less than m − 1. Hence, indeed, for any states i and j 6 = i, P (^) i,jk > 0 for some k, with 1 ≤ k ≤ m − 1. But, in general, the value k depends on the states i and j. As a corollary to the last conclusion, we deduce the following:
Corollary 0.1 For any state i, there necessarily is a k ≤ m such that P (^) i,ik > 0.
Proof. To start, there must be some j such that Pi,j > 0. Then, by the reasoning above, P (^) j,il > 0 for some l ≤ m − 1. But then P (^) i,il+1 > Pi,j P (^) j,il > 0. Finally, since l ≤ m − 1, necessarily l + 1 ≤ m. Now we apply the aperiodicity to obtain a further result.
Lemma 0.2 There exists a constant n 0 such that P (^) i,ik > 0 for all k ≥ n 0.
Proof. We have shown above that P (^) i,ik > 0 for some k with 1 ≤ k ≤ m. Since the chain is aperiodic, there necessarily exist constants k 1 and k 2 such that the greatest common de- nominator of k 1 and k 2 is 1, and P (^) i,ik^1 > 0 and P (^) i,ik^2 > 0. But then P (^) i,ik > 0 for all k ≥ k 1 k 2.
We can do better in this last step, but the reasoning is somewhat complicated. We have the following sharper result from Section 2.4 of Seneta, Non-negative Matrices and Markov Chains, second edition, Springer, 1981, in particular from Theorem 2.9 on page 58.
and vP = (0. 01 , 0. 95 , 0. 01 , 0. 01 , 0. 01 , 0 .01). There are three natural norms to consider on Rm: the l∞, l 2 and l 1 norms:
||x||∞ ≡ max {|xi|} ,
||x|| 2 ≡
( (^) m ∑
i=
|xi|^2
||x|| 1 ≡
∑^ m
i=
|xi|. (1)
It is not difficult to see that
||u − v||∞ = 1 / 3 , ||u − v|| 2 =
||u − v|| 1 = 2 (2)
and It is not difficult to see that
||uP − vP ||∞ = 0. 94 , ||uP − vP || 2 =
2 × (0.94)^2 = 1. 329 ,
||uP − vP || 1 = 2 × 0 .94 = 1. 88. (3)
By this example, we prove that the norms ||x||∞ and ||x|| 2 do not work. However, it turns out that the norm ||x|| 1 does work.
Theorem 0.3 Let P be a m×m Markov-chain transition matrix associated with an irreducible Markov chain. Assume that Pi, 1 ≥ ≤ > 0 for all i, 1 ≤ i ≤ m. Then
||uP − vP || 1 ≤ (1 − ≤)||u − v|| 1.
Proof. Note that
||uP − vP || 1 =
∑^ m
j=
∑^ m
i=
uiPi,j −
∑^ m
i=
viPi,j |.
Now write Pi, 1 ≡ ≤ + Qi, 1 and Pi,j ≡ Qi,j for j 6 = i , for all i.
Then Q is a nonnegative m × m matrix with row sums 1 − ≤. Now observe that
∑^ m
j=
∑^ m
i=
uiPi,j −
∑^ m
i=
viPi,j | = |
∑^ m
i=
ui(≤ + Qi, 1 ) −
∑^ m
i=
vi(≤ + Qi, 1 )| +
∑^ m
j=
∑^ m
i=
uiQi,j −
∑^ m
i=
viQi,j |
∑^ m
i=
uiQi, 1 −
∑^ m
i=
viQi, 1 | +
∑^ m
j=
∑^ m
i=
uiQi,j −
∑^ m
i=
viQi,j |
∑^ m
j=
∑^ m
i=
uiQi,j −
∑^ m
i=
viQi,j |
∑^ m
j=
∑^ m
i=
|ui − vi|Qi,j =
∑^ m
i=
∑^ m
j=
|ui − vi|Qi,j = (1 − ≤)||u − v|| 1 .(4)
with the equality in the second line holding because the epsilon terms can be dropped out, using ∑m
i=
(ui − vi) = 0 ,
because u and v are probability vectors, summing to 1. We remark that we would obtain a stronger contraction property if we applied the above reasoning to all m columns. We would strengthen the assumption in the theorem to: Assume that, for each column j, Pi,j ≥ ≤j ≥ 0 for all i, 1 ≤ i ≤ m, with ≤j > 0 for at least one j. We would then use the reasoning above to obtain the stronger conclusion:
||uP − vP || 1 ≤ c||u − v|| 1 ,
where
c = (1 −
∑^ m
j=
≤j ) < 1.
It is not difficult to see that we must have
∑m j=1 ≤j^ <^ 1.
