Download Discrete-time Markov Chains: Probability Transition and Stationary Distributions - Prof. R and more Study notes Electrical and Electronics Engineering in PDF only on Docsity!
1 Discrete-time Markov Chains
Let {Xk} be a discrete-time stochastic process which takes on values in a countable set S, called the state space. {Xk} is called a Discrete-time Markov chain (or simply a Markov chain, when the discrete nature of the time index is clear) if
P (Xk = ik | Xk− 1 = ik− 1 , Xk− 2 = ik− 2 ,.. .) = P (Xk = ik | Xk− 1 = ik− 1 ),
where ij ∈ S. A Markov chain is said to be time homogeneous, if P (Xk = i | Xk− 1 = j) is independent of k. We will only consider time-homogeneous Markov chains here. Associated with each Markov chain is a matrix called the probability transition matrix, denoted by P, whose (i, j)th^ element is given by Pij = P (xk = j | xk− 1 = i). Let p[k] denote a row vector of probabilities with pj [k] = P (Xk = j). This vector of probabilities evolves according to the equation
p[k] = p[k − 1] P.
Thus, p[0] and P capture all the relevant information about the dynamics of the Markov chain. The following questions are important in the study of Markov chains:
- Does there exist a π so that π = πP? If such a π exists, it is called a stationary distribution.
- If there exists a unique stationary distribution, does limk→∞ p[k] = π for all p[0]? In other words, does the distribution of the Markov chain converge to the stationary distribution starting from any initial state?
While the existence of a unique stationary distribution is desirable in the applications studied in these notes, not all Markov chains have a unique steady-state distribution. We will first present an example of Markov chain which does not have a stationary distribution and then another example where a a stationary distribution exists but the probability distribution over the states does not converge to the stationary distribution in steady-state. Motivated by these examples, we will impose some conditions to guarantee the existence of a stationary distribution to which the Markov chain converges in steady state.
Example 1 Consider a trivial two-state Markov chain with states a and b such that the chain remains in the state for time slots k ≥ 0. Thus, the transition probability matrix for this Markov
chain is given by P =
( 1 0 0 1
)
. Therefore, πP = π is true for any distribution π and the
stationary distribution is not unique. The Markov chain in this example is such that if it started in one state, then it remained in the same state forever. In general, Markov chains where one state cannot be reached from some other state will not possess a unique stationary distribution.
The above example motivates the following definitions.
Definition 1 Let P (^) ijn = P (Xk+n = j | Xk = i).
- State j is said to be reachable from state i, if there exists n ≥ 1 so that P (^) ijn > 0.
- A Markov chain is said to be irreducible if a state i is reachable from any other state j.
In these notes, we will mostly consider Markov chains that are irreducible. The Markov chain in Example 1 was not irreducible.
Example 2 Again let us consider a two-state Markov chain with two states a and b. The Markov chain behaves as follows: if it is in state a at the current time-slot, then it jumps to b at the next time-slot and vice-versa. Thus,
P =
( 0 1 1 0
) .
The stationary distribution is obtained by solving πP = π ⇒ which gives π = (1/ 2 1 /2). However, the system does not converge to this stationary distribution starting from any initial condition. To see this, note that if p[0] = (1 0), then p[1] = (0 1) , p[2] = (1 0) , p[3] = (0 1) , p[4] = (1 0) ,.. .. Therefore limk→∞ p[k] 6 = π. The reason that this Markov chain does not converge to the stationary distribution is due to the fact that the state periodically alternated between a and b.
Motivated by the above example, the following definitions lead up to the classification of a Markov chain as being either periodic or aperiodic.
Definition 2 The following definitions classify Markov chains and their states as periodic or ape- riodic:
- State i is said to have a period di ≥ 1 if di is the greatest integer such that P (^) ii(n ) = 0 if n is not a multiple of di. If P (^) ii(n )= 0 ∀n, we say that di = ∞.
- State i is said to be aperiodic if di = 1.
- A Markov chain is said to be aperiodic if all states are aperiodic.
Lemma 1 Every state in an irreducible Markov chain has the same period. Thus, in an irreducible Markov chain, if one state is aperiodic, then the Markov chain is aperiodic.
The Markov chain in Example 2 was not aperiodic. The following theorem states that Markov chains which do not exhibit the type of behavior illustrated in the examples possess a steady-state distribution to which the distribution converges, starting from any initial state.
Theorem 1 A finite state space, irreducible Markov chain has a unique stationary distribution π and if it is aperiodic, limk→∞ p[k] = π, ∀p[0].
We will now show that we cannot find a distribution π that satisfies the above set of equations. Towards this end, note that
π 2 = 2 π 1 − π 0 π 3 = 2 π 2 − π 1 = 2(2π 1 − π 0 ) − π 1 = 3 π 1 − 2 π 0 π 4 = 6 π 1 − 4 π 0 − 2 π 1 + π 0 = 4 π 1 − 3 π 0 .. . πk = kπ 1 − (k − 1)π 0 = k(π 1 − π 0 ) + π 1 , k ≥ 2
Thus,
if π 1 = π 0 > 0 , then πk = π 1 , ∀k ≥ 2 and
∑^ ∞
k=
πk > 1 ,
if π 1 > π 0 , then πk → ∞, if π 1 < π 0 , then πk → −∞, if π 1 = π 0 = 0, then πk = 0 ∀k ≥ 0.
A little thought shows that the last statement is also true for k < 0. Thus, a stationary distribution cannot exist.
Example 4 illustrates that the need for more conditions beyond aperiodicity and irreducibility to ensure the existence of stationary distributions in countable state space Markov chains. Towards this end, we introduce the notion of recurrence and related concepts.
Definition 3 The following definitions classify the states of a Markov chain as recurrent or tran- sient:
- The recurrence time Ti of a Markov chain is defined as
Ti = min{n ≥ 1 : Xn = i given X 0 = i.}
(Note that Ti is a random variable.)
- A state i is said to be recurrent if P (Ti < ∞) = 1. Otherwise, it is called transient.
- The mean recurrence time Mi of state i is defined as Mi = E[Ti].
- A recurrent state i is called positive recurrent if Mi < ∞. Otherwise, it is called null recurrent.
- A Markov chain is called positive recurrent if all of its states are positive recurrent.
The next two lemmas and theorem are stated without proof.
Lemma 2 Suppose {Xk} is irreducible and one of its states is positive recurrent, then all of its states are positive recurrent. (The same statement holds if we replace positive recurrent by null recurrent or transient.)
Lemma 3 If state i of a Markov chain is aperiodic, then limk→∞ pi[k] = (^) M^1 i. (This is true whether or not Mi < ∞ and even for transient states by defining Mi = ∞ when state i is transient.)
Theorem 2 Consider a time-homogeneous Markov chain is irreducible and aperiodic. Then, the following results hold:
- If the Markov chain is positive recurrent, then there exists a unique π so that π = πP and limk→∞ p[k] = π. Further πi = (^) M^1 i.
- If there exists a positive vector π such π = πP and
∑ i πi^ = 1,^ then it must be the stationary distribution and limk→∞ p[k] = π. (From Lemma 3, this also means that the Markov chain is positive recurrent.)
- If there exists a positive vector π such that π = πP and ∑ i πi^ is infinite, then a stationary distribution does not exist and limk→∞ pi[k] = 0 for all i.
The following example is an illustration of the application of the above theorem.
Example 5 Consider a simple model of a wireless link where, due to channel conditions, either one packet or no packet can be served in each time slot. Let s[k] denote the number of packets served in time slot k and suppose that s[k] are i.i.d. Bernoulli random variables with mean μ. Further, suppose that packets arrive to this wireless link according to a Bernoulli process with mean λ,, i.e., a[k] is Bernoulli with mean λ where a[k] is the number of arrivals in time slot k and a[k] are i.i.d. across time slots. Assume that a[k] and s[k] are independent processes. We specify the following order in which events occur in each time slot:
- We assume that any packet arrival occurs first in the time slot, followed by any packet depar- ture.
- Packets that are not served in a time slot are queued in a buffer for service in a future time slot.
Let q[k] be the number of packets in the queue at time slot k. Then q[k] is a Markov chain and evolves according to the equation
q[k + 1] = (q[k] + a[k] − s[k])+^.
We are interested in the steady-stated distribution of this Markov chain. The Markov chain can be pictorially depicted as in Figure 5 where the circles denote the state of the Markov chain (the number of packets in the queue) and the arcs denote the possible transitions with the number on an arc denoting the probability of that transition occurring. For example, Pi,i+1, the probability that
Thus, the stationary distribution is completely characterized. If λ ≥ μ, then let π 0 = 1 and from (2), it is clear that ∑ i πi^ =^ ∞.^ Thus, from Theorem 2, the Markov chain is not positive recurrent if λ ≥ μ.
Unlike the above example, there are also many instances where one cannot easily find the sta- tionary distribution by solving π = πP. But we would still like to know if the stationary distribution exists. It is often easy to verify the irreducibility and aperiodicity of a Markov chain, but, in general, it is difficult to directly verify whether a Markov chain is positive recurrent from the definitions given earlier. Instead, there is a convenient test called Foster’s test or the Foster-Lyapunov test to check for positive recurrence which we state next.
Theorem 3 Let {Xk} be an irreducible Markov chain with a state space S. Suppose that there exists a function V : S → <+^ and a finite set B ∈ S satisfying the following conditions:
- E[V (Xk+1) − V (x) | Xk = x] ≤ − if x ∈ Bc, and
- E[V (Xk+1) − V (x) | Xk = x] ≤ M for some M < ∞, if x ∈ B,
for some M < ∞ and > 0. Then the Markov chain {Xk} is positive recurrent.
Proof: We will prove the result under the further assumption that {Xk} is aperiodic.
E[V (Xk+1) − V (Xk) | Xk = x] ≤ −Ix∈Bc + M Ix∈B = −Ix∈Bc^ + M − M Ix∈Bc = −(M + )Ix∈Bc + M
Taking expectations on both sides, we get
E[V (Xk+1)] − E[V (Xk)] = −(M + )P (Xk ∈ Bc) + M, ∑^ N
k=
(E[V (Xk+1)] − E[V (Xk)]) = −(M + )
∑^ N
k=
P (Xk ∈ Bc) + M N.
Thus,
E[V (XN +1)] − E[V (X 0 )] = −(M + )
∑^ N
k=
P (Xk ∈ Bc) + M N
(M + )
∑^ N
k=
P (Xk ∈ Bc) = M N + E[V (X 0 )] − E[V (XN +1)]
≤ M N + E[V (X 0 )] M + N
∑^ N
k=
P (Xk ∈ Bc) ≤ M +
N
E[V (X 0 )]
lim N →∞ sup
N
∑^ N
k=
P (Xk ∈ Bc) ≤
M
M +
lim N →∞ inf
N
∑^ N
k=
P (Xk ∈ B) ≥
M +
Suppose that any state of the Markov chain is not positive recurrent, all states are not positive recurrent since the Markov chain is irreducible. Thus, Mi = ∞, ∀i and limk→∞ pi[k] = 0, ∀i. Thus,
limk→∞ P (Xk ∈ B) = 0 or lim N →∞ inf
N
∑^ N
k=
P (Xk ∈ B) = 0, which contradicts the fact that it is
≥ (^) M +. Next, we present two extensions of Foster’s theorem without proof.
Theorem 4 An irreducible Markov chain {Xk} is positive recurrent if there exists a function V : S → <+, a positive integer L ≥ 1 and a finite set B ∈ S satisfying the following conditions:
E[V (Xk+L) − V (x) | Xk = x] ≤ −Ix∈Bc + M Ix∈B
for some > 0 and L < ∞ for some > 0 and M < ∞.
Theorem 5 An irreducible Markov chain {Xk} is positive recurrent if there exists a function V : S → <+, a function η : S → <+, and a finite set B ∈ S satisfying the following conditions:
E[V (Xk+η(x)) − V (x) | Xk = x] ≤ −η(x)Ix∈Bc^ + M Ix∈B
for some > 0 and M < ∞.
The following theorem provides conditions under which a Markov chain is not positive recurrent.
Theorem 6 An irreducible Markov chain {Xk} is either transient or null recurrent if there exists a function V : S → <+^ and and a finite set B ∈ S satisfying the following conditions:
- E [V (Xk+1) − V (x) | Xk = x] ≥ 0 , ∀x ∈ Bc
- V (x) > V (y) for some x ∈ Bc^ and all y ∈ B, and
- E [|V (Xk+1) − V (Xk)| | Xk = x] ≤ M for some M < ∞ and ∀x ∈ S.
