STAT/ELEC 331 HW 11
Problems in addition to those from the book
Hint on Ch 7, Prob 13 Express the n+ 1 step transition probabilities in terms of the nstep
transition probabilities and the (one step) transition probabilities. Let n→ ∞. Verify the
expression immediately below Definition 7.2.5 on page 418.
1. In applications, the initial value X0of a stochastic process is typically provided, but it is
also valid to view it as a random quantity. Suppose that the initial observation X0in a
Markov chain has distribution q0(viewed as a row vector). Use the law of total probability
and properties of transition matrices to verify that the distribution of Xnis q0P(n). What
happens if q0is a stationary distribution?
2. Consider Problem 4, Chapter 7, in the book.
a. Describe the sequence of moves as a Markov chain by drawing a transition graph, and
write down the transition probability matrix P.
b. Using a computer, determine the limit (numerically) of P(n)as n→ ∞. Does a limit
distribution exist? If so, what is it, to four digits of accuracy?
c. Unfortunately, this numerical approach doesn’t give us exact expression for the limiting
distribution. However, by problem 13 in the book, limiting distributions are stationary
distributions. Perhaps we can find a stationary distribution and it will coincide with the
observed limit. Find a stationary distribution by solving a linear system of equations.
Don’t forget the equation P3
i=1 πi= 1. Does the result agree with b?
To rigorously conclude that the stationary distribution is indeed the limit distribution, we
need to verify that the Markov chain is irreducible, positive recurrent, and aperiodic. Recall
that positive recurrence is equivalent to the existence of a stationary distribution, which
we just established. Thus is remains to verify irreducibility and aperiodicity. Verify these
properties for yourself.
3. Random walk on a chessboard
Suppose a king takes a random walk on a 3 ×3 chessboard. Recall that a king can move one
step in any direction along rows, columns, or diagonals. What is the steady-state (limiting)
distribution? That is, if the random walk continues for a long, long time, and then we peek
in and observe the location of the king, what is the probability of it appearing on each of
the 9 squares? You may assume the fact that random walks on finite, connected graphs are
ergodic.
Repeat the problem where the king is replaced by a queen, a rook, and a bishop. Note that
there are two kinds of bishops.