Download Problem Set 5 for Stochastic Processes | MATH 285 and more Assignments Stochastic Processes in PDF only on Docsity!
Problem Set 5
Troy Kravitz
May 16, 2007
Problem 1
Exercise 1.9.1(b) and 1.9.1(e) (assuming S is finite).
From example 1.9.4, we know the transition matrix in part (b) will not be reversible. We’ll demonstrate this more formally, however. First, let
P =
0 p 1 − p 1 − p 0 p p 1 − p 0
A stationary distribution, Π′^ =
[
a b c
]
, is a probability distribution such that Π′P = Π′. This matrix expression plus the requirement that Π be a probability distribution yields four equations:
- b(1 − p) + cp = a
- ap + c(1 − p) = b
- a(1 − p) + bp = c
- a + b + c = 1
Solving these equations simultaneously, we find a = b = c = 13.
As in example 1.9.4, Π′^ =
[ 1
3
1 3
1 3
]
is an invariant distribution of P. But since the transition matrix for the time-reversed Markov chain is P ′^6 = P (since P is not symmetric), the Markov chain in problem 1.9.1(b) is not reversible.
For problem 1.9.1(e), since we assume S is finite, let the dimensionality of P be n × n where n ∈ N. Note that since P is symmetric, we have
P =
p 11 p 12 p 13 ... p 1 n p 21 p 22 p 23 ... p 2 n p 31 p 32 p 33 ... p 3 n ... ... ... ... ... pn 1 pn 2 pn 3 ... pnn
p 11 p 21 p 31 ... pn 1 p 12 p 22 p 32 ... pn 2 p 13 p 23 p 33 ... pn 3 ... ... ... ... ... p 1 n p 2 n p 3 n ... pnn
Now suppose (at this point it is only a guess) that Π′^ =
[ 1
n
1 n
1 n ...^
1 n
]
is a stationary distribution for P. In order for Π to be stationary, we require Π′P = Π′. This is a system of n equations:
- P n^11 + P n^12 + ... + P n^1 n = (^) n^1
- P n^21 + P n^22 + ... + P n^2 n = (^) n^1
- ...
- P nn^1 + P nn^2 + ... + Pnn n = (^) n^1
These equations are always satisfied since
j pij^ = 1^ ∀i^ ∈ S.^ Hence, our postulated Π′^ =
[ 1
n
1 n
1 n ...^
1 n
]
is invariant. Since P is symmetric, the Markov chain in 1.9.1(e) is reversible.
Problem 2
Exercise 1.10.
Let P =
1 3 0
2 3 1 0 0
(^) and Π′^ =
[
a b c
]
. Solving for an invariant distri-
bution yields the following four-equation system:
- b 13 + c = a
- a = b
- b 23 = c
- a + b + c = 1
which has the solution a = b = 38 and c = 14. Thus, Π′^ =
[ 3
8
3 8
1 4
]
is an invariant distribution. Now consider Theorem 1.10.2 and let f be the indicator function for being in state 2. Then (^1) n
∑n− 1 k=0 f^ (Xk) is the average value of the indicator function. This tells us the long-run time spent in state 2. By Theorem 1.10.2, the average of the indicator function converges to f¯ where f¯ =
i∈I πifi. In this expression, πi is the ith^ element of the unique invariant distribution Π – in our case, π 2 = 38 and f 2 = 1, so f¯ = 38. That is, the long-run probability of being in state 2 is 38.
Problem 3
Consider the Markov chain X with transition matrix given by P in example 1.2.2 on pg. 11 of the text. Prove that T = inf{n > 10 :
notes the probability that given the coin is currently in state i, it is in state j at the next time step. Assume that initially the coin is equally likely to be Coin 1 or Coin 2. (a) Suppose that you observe the sequence HHTTTH. What is a/the most likely state sequence to generate this output? (b) What is the probability that given the observation sequence it was generated entirely by Coin 2?
We will use the Viterbi algorithm to solve this problem. To this end, let δt(i) be defined (as in [Rabiner]) as the highest probability along a single path, at time t, which accounts for the first t observations. Also, let ψt(j) be used to keep track of the highest probability state. We label as state 1 the state in which the coin is fair; state 2 corresponds to the state in which the coin is biased.
We now initialize the δ expressions as δ 1 (i) = πibi(h) for i = 1, 2. Thus, δ 1 (1) = 1212 = 14 and δ 1 (2) = 1214 = 18. We also initialized ψ 1 (i) = 0 for both states.
Recursively, we solve for subsequent values using the formulas δt(j) = max 1 ≤i≤ 2 {δt− 1 (i)aij }bj (O?t ) and ψt(j) = arg max 1 ≤i≤ 2 {δt− 1 (i)aij }.
These formula yield:
δ 1 (1) =
, ψ 1 (1) = 0
δ 1 (2) =
, ψ 1 (2) = 0
δ 2 (1) = max{. 7
, ψ 2 (1) = 1
δ 2 (2) = max{. 3
, ψ 2 (2) = { 1 , 2 }
δ 3 (1) = max{. 7
, ψ 3 (1) = 1
δ 3 (2) = max{. 3
, ψ 3 (2) = 1
δ 4 (1) = max{. 7
, ψ 4 (1) = 1
δ 4 (2) = max{. 3
, ψ 4 (2) = 2
δ 5 (1) = max{. 7
, ψ 5 (1) = 1
δ 5 (2) = max{. 3
, ψ 5 (2) = 2
δ 6 (1) = max{. 7
, ψ 6 (1) = 1
δ 6 (2) = max{. 3
, ψ 6 (2) = 2
Our terminal condition is q? 6 = arg max 1 ≤i≤ 2 {δ 6 (i)}, so q 6? = 1. Backtracking, we define q?t = ψt+1(qt?+t) for t = T − 1 , T − 2 , ..., 1.
Applying the above, we find the most likely sequence of the hidden states is { 1 , 1 , 1 , 1 , 1 , 1 }.
The conditional probability of observing a string of all state 2’s given the output sequence is
P (q?|O?, λ) =
P (q?, O?|λ) P (O?|λ)
We first compute the numerator: the probability of observing a string of all state 2’s and the output sequence {HHT T T H}. This probability is governed by the probability of initializing the chain in state 2, observing a heads, staying in state 2, observing another heads, again staying in state 2, observing a tails, and so forth. Mathematically, letting π 2 = 12 be the probability of initializing the chain in state 2, P (22) = .6 be the probability of remaining in state two between steps, and b 2 (h) = 14 be the probability of observing output heads given that the chain is in state 2, the probability of the output and state sequence is
P (q?, O?|λ) = (π 2 b 2 (h)) (P (22)b 2 (h)) (P (22)b 2 (t)) (P (22)b 2 (t)) (P (22)b 2 (t)) (P (22)b 2 (h))
Inputting the values above, we have
P (q?, O?|λ) = (
Considering the denominator, P (O?|λ), we compute this using forward vari- ables, αt(i), initialized as α 1 (i) = πibi(O? 1 ). Our induction step is αt+1(j) = [
i=1 αt(i)aij^ ]bj^ (O
? t+1) and our terminal condition is^ P^ (O ?|λ) = ∑^2 i=1 α^6 (i).
for j = 1: for i=1: if state(j,i)== cumprob(j) = cumprob(j)* unfair(i); else cumprob(j) = cumprob(j)* fair(i); end
end
if maxcumprob(1) < cumprob(j) maxcumprob(1)=cumprob(j); maxcumprob(2)=j; end end
p11 =.7; p12 =.3; p21 =.4; p22 = .6;
for j = 1: for i=2: if state(j,i-1)== if state(j,i)== transitionProb(j)=transitionProb(j)p11; else transitionProb(j)=transitionProb(j)p12 ; end
else if state(j,i)== transitionProb(j)=transitionProb(j)p21; else transitionProb(j)=transitionProb(j)p22 ; end end end end
Likelihood = cumprob .* transitionProb; MLEState = find(Likelihood == max(Likelihood)); The MLE state sequence is 1 1 1 1 1 1
