Download Lecture Notes on Markov Processes | ECE 6010 and more Study notes Stochastic Processes in PDF only on Docsity!
ECE 6010
Lecture 10 – Markov Processes
Basic concepts
A Markov process {Xt} one such that
P (Xtk+1 = xk+1|X(tk ) = xk , X(tk− 1 ) = xk− 1 ,... , X(t 1 ) = x 1 ) = P (Xtk+1 = xk+1|Xtk = xk )
(for a discrete random process) or
f (xtk+1 |Xtk = xk... , xt 1 = x 1 ) = f (xtk+1 |Xtk = xk )
(for a continuous random process). The most recent observation determines the state of the process, and prior observations have no bearing on the outcome if the state is known.
Example 1 Let Xi be i.i.d. and let Sn = X 1 + · · · + xn = Sn− 1 + Xn. Then
P [Sn+1 = sn+1|Sn = sn,... , S 1 = s 1 ] = P [Xn+1 = Sn+1−Sn] = P [Sn+1 = sn+1|Sn = sn].
2
Example 2 Let N (t) be a Poisson process.
P (N (tk+1) = nk+1|N (tk ) = nk,... , N (t 1 ) = n 1 )] = P [j−i events in tk+1−tk] = P [N (tk+1) = nk+1|N (tk ) = nk ]
2
Let {Xt} be a Markov r.p. The joint probability has the following factorization:
P (Xt 3 = x 3 , Xt 2 = x 2 , Xt 1 = x 1 ) = P (Xt 3 = x 3 |Xt 2 = x 2 )P (Xt 2 = x 2 |Xt 1 = x 1 )P (Xt 1 = x 1 )
(Why?)
Discrete-time Markov Chains
Definition 1 An integer-valued Markov random process is called a Markov chain. 2 Let the time-index set be the set of integers. Let
pj (0) = P [X 0 = j]
be the initial probabilities. (Note that
j pj^ (0) = 1.) We can write the factorization as
P [Xn = in,... , X 0 = i 0 ] = P [Xn = in|Xn− 1 = in− 1 ] · · · P [X 1 = i 1 |X 0 = i 0 ]P [X 0 = i 0 ].
If the probability P [Xn+1 = j|Xn = i] is does not change with n, then the r.p. Xn is said to have homogeneous transition probabilities. We will assume that this is the case, and write pij = P [Xn+1 = j|Xn = i].
Note:
j P^ [Xn+1^ =^ j|Xn^ =^ i] = 1. That is,^
j pij^ = 1. We can represent these transition probabilities in matrix form:
P =
p 00 p 01 p 02 · · · p 10 p 11 p 12 · · · .. .
The rows of P sum to 1. This is called a stochastic matrix. Frequently discrete-time Markov chains are modeled with state diagrams.
Example 3 Two light bulbs are held in reserve. After a day, the probability that we need a light bulb is p. Let Yn be the number of new light bulbs at the end of day n.
P =
p 1 − p 0 0 p 1 − p
Draw the diagram. 2
Now let us look further ahead. Let pij (n) = P [Xn+k = j|xk = i]
If the r.p. is homogeneous, then pij (n) = P [Xk = j|Xi = i]. Let us develop a formula for the case that n = 2.
P [X 2 = j, X 1 = l|x 0 = i] =
P [X 2 = j, X 1 = l, X 0 = i] P [X 0 = i]
=
P [X 2 = j|X 1 = l]P [X 1 = l|X 0 = i]P [X 0 = i] P [X 0 = i] = pil(1)plj (1) = pilplj
Now marginalize:
P [X 2 = j|X 0 = i] =
l
P [X 2 = j, X 1 = l|X 0 = i] =
l
pilPlj.
Let P (2) be the matrix of two-step transition probabilities. Then we have
P (2) = P (1)P (1) = P 2.
In general (by induction) we have P (n) = P n. Let
p(n) =
P (Xn = 0) P (Xn = 1) .. .
(or whatever the outcomes are). Then
pj (n) = P (Xn = j) =
i
P (Xn = j|Xn− 1 = i)P (Xn− 1 = i) = pij pi(n − 1).
Stacking these up, we obtain the equation
p(n) = p(n − 1)P.
or p(n) = p(0)P n. We we run the Markov r.p. for a long time what happens to the probabilities? That is, what is pj (n) as n → ∞? Let us denote
πj = lim n→∞ pj (n).
If there is a limit, the probability vector π should satisfy
π = πP.
or P T^ πT^ = πT^.
This is an eigenvalue problem!
- Select a new state according to a discrete-time Markov chain with transition proba- bilities we will call q˜ij
- Repeat.
In discrete time we have the probability update p(k + 1) = p(k)P. We will develop an analogous result for continuous time. Instead of a set of couple difference equations, we will get a set of coupled differential equations. Let δ be a small time increment.
P (Ti > δ) = e−νiδ^ ≈ 1 − νiδ + o(δ).
The probability that we remain in the same state at time δ is
pii(δ) = P (Ti > δ) = 1 − νiδ + o(δ)
or 1 − pii(δ) = νiδ + o(δ). Now consider the transition. When leaving state i, we move to state j with probability q ˜ij : pij (δ) = (1 − pii(δ)) ︸ ︷︷ ︸ leave state
q ˜ij = νiδ q˜ij + o(δ).
Let γij = νi q˜ij : pij (δ) = γij δ + o(δ).
We say that γij is the rate at which X(t) enters state j from state i. Define γii = −νi, so that 1 − pii(δ) = −γiiδ + o(δ)
or pii(δ) − 1 = γiiδ + o(δ).
Summarizing what we have so far:
pii(δ) − 1 = γiiδ + o(δ) pij (δ) = γij δ + o(δ).
Divide by δ and take the limit:
lim δ→ 0
pii(δ) − 1 δ
= γii
lim δ→ 0
pij (δ) δ
= γij
Now define pj (t) = P (X(t) = j). Then we have
pj (t+δ) = P (X(t+δ) = j) =
i
P (X(t+δ) = j|X(t) = i)P (X(t) = i) =
i
pij (δ)pi(t).
and
pj (t + δ) − pj (t) =
i
pij (δ)pi(t) − pj (t) =
i 6 =j
pij (δ)pi(t) + pjj (t)pj (t) − pj (t)
i 6 =j
pij (δ)pi(t) + (pjj (t) − 1)pj (t).
Divide both sides by δ and take the limit:
p′ j (t) =
i 6 =j
γij pi(t) + γjj pj (t) =
i
γij pi(t)
Example 5 Let us model a two-state system, having an idle state and a busy state. In the idle state, the system is waiting for work to arrive. Assume that the waiting time is an exponential r.v. with mean 1 /α. In the busy state, the machine works for a random amount of time, with an exponential distribution having mean 1 /β. We can think of the rate of motion from idle to busy as α, and the rate of motion from busy to idle as β. The underlying discrete-time Markov chain has
Q^ ˜ =
[
]
(when the transition occurs, there is no ambiguity). We find that
γ 00 = −α γ 01 = αq 01 = α γ 10 = βq 10 = β γ 11 = −β.
We obtain the coupled differential equations
[ p 0 (t) p 1 (t)
]′
[
−α β α −β
] [
p 0 (t) p 1 (t)
]
or p′(t) = Ap(t).
We also have an auxiliary equation p 0 (t) + p 1 (t) = 1, or p(t)T^ 1 = 1. We can solve this as p(t) = exp[At]p(0). Somewhat more explicitly:
sp(s) − p(0) = Ap(s)
(sI − A)p(s) = p(0) p(s) = (sI − A)−^1 p(0).
(sI − A)−^1 =
[
s + α −β −α s + β
]− 1
(s + α)(s + β)
[
s + β β α s + α
]
so
p(s)
(s + α)(s + β)
[
(s + β)p 1 (0) + βp 2 (0) αp 1 (0) + (s + α)p 2 (0)
]
Now it is a matter of straightforward (but careful) computation to show that
p 0 (t) =
β α + β
β α + β
)e−(α+β)t
p 1 (t) = α α + β
)e−(α+β)t.
In the limit
p 0 (t) →
β α + β
p 1 (t) →
α α + β
What are the steady-state conditions in general?
0 sumiγij pi
- If in a recurrent state, then the state recurs an infinite number of times.
Let Xn denote the Markov chain with X 0 = i. Let
Ii(x) =
1 if X = i 0 otherwise
Then
E[number of returns to state i] = E
[
i
Ii(Xn)|X 0 = i
]
∑^ ∞
n=
pii(n).
We see that recurrent means that
n=1 pii(n) =^ ∞. Transient means that
n=1 pii(n)^ <^ ∞. Example 10
1 2 3
0
1/
β
α
1 − α
p 00 (n) = (1/2)n ∑^ ∞
n=
p 00 (n) = 1 < ∞
so state 0 is transient. What if we start in state 1?
p 11 (n) =
β + α(1 − α − β)n α + β
This sums to ∞. 2
Example 11
0 1 2 3
1 − p 1 − p 1 − p 1 − p
pii(n) = (1 − p)n. ∑
n
pii(n) < ∞
2
Example 12 Random walk.
0 1 2 3
p p (^) p (^) p
1 − p 1 − p 1 −^ p^1 −^ p
Start in state 0. We can return if we make as many right-hand moves (with probability p) as left-hand moves (with probability 1 − p). The total number of moves (left and right) must be an even number; take the total number as 2 n. There are
( 2 n n
ways of making n RH moves:
p 00 (n) =
2 n n
pn(1 − p)n
Summing (to see if transient):
∑^ ∞
n=
p 00 (n) =
∑^ ∞
n=
2 n n
pn(1 − p)n
How to sum this? We can get a good approximation using Stirling’s formula:
n! ≈
2 πnn
ne−n
Then (^) ( 2 n n
πn
4 n
and
p 00 (n) ≈
[4p(1 − p)]n √ πn Now sum: ∑∞
n=
p 00 (n) ≈
∑^ ∞
n=
[4p(1 − p)]n √ πν
Still a little hard. But take the particular case of p = 1/ 2. Then we get
∑^ ∞
n=
πn
= ∞ (why?)
If p 6 = 1/ 2 , then [4p(1 − p)] < 1 , and the sum converges. 2
Observation: The states of an irreducible, finite-state Markov chain are all recurrent.
Limiting probabilities
If all states are transient, then all the state probabilities approach 0 as n → ∞. If a M.C. has some transient classes and some recurrent classes, then eventually the process enters and remains in one of the recurrent classes. For limiting purposes, we can focus on individual recurrent classes. Suppose a M.C. starts in a recurrent state i at time 0. Let Ti(1), Ti(1) + Ti(2),... denote the times when the process returns to state i, where Ti(k) is the time that elapses between the (k − 1)th and kth returns. The Ti form an i.i.d. sequence. The proportion of time spent in state i after k returns is
k Ti(1) + Ti(2) + · · · + Ti(k)
In the limit,
proportion of time spent in state i →
E[Ti]
4 = πi
(by the law of large numbers), where E[Ti] is the mean recurrence time. If E[Ti] < ∞, we say that state i is positive recurrent : πi > 0. If E[Ti] = ∞, we say that state i is null recurrent : πi = 0.
Example 13
