Markov Processes
© Arizona State University, Department of Mathematics and S tatistics 1 of 3
Regular Markov Chains
There are some Markov chains with very interesting properties. The first we will look at are called regular.
Regularity comes from the properties of matrices. Recall that when you multiply matrices we can have non-zero
matrices create zero products, or perhaps one the original matrices even though we never used the identity or
other freaky occurrences.
Regular Markov Chains have transition matrices P such that for some n, has only positive entries (no zeros
n
P
allowed anywhere).
Thinking of what this means, it is always possible to move from one state to another, even if the first step
wouldn’t allow it.
Just to be clear about how we know where we are during a stage of transition, recall that
to find the probability of passing from state I to state j we need only look at the ij
th
entry.
In the matrix to the right, our probability in moving from s2 to s1 is zero since the 21
th
entry (p
21
) is zero. If you started in state two, you would be stuck there forever. We’ll
come back to that.
Look again at the transition matrix to the right. Clearly, it could not be regular. Taking
the second and then successive powers of the transition matrix, that zero value is stuck forever in the
21
th
entry (p
21
).
Now look at the 3 by 3 matrix to the right. Enter it into your calculator and take it to
successfully higher powers. You’ll see that the p
31
entry is non-zero after just the second
step. It is regular at n = 2. Even so, should you begin in the initial state of C, you have
no chance of moving to A immediately. If this matrix described product satisfaction
since, this could mean that if you bought C first you would never settle for less.
However, if we seek the fixed probability vector, t, we find that the matrix stabilizes at
. This is the long-term behavior of this regular Markov
20 50 91
, ,
261 87 261
chain. The long-term behavior is exactly that. It does not describe what will happen at the first step from a
given state.
As suggested, a regular Markov chain could demonstrate a level of satisfaction over some number of steps or
years. The impact of regularity is that . . . given a list of products, say A, B, C, . . . , ALL of the products will
eventually obtain some market share when the model is regular.
Example: Suppose the transition matrix above related to politicians A, B, C. Politician A is a scoundrel. He has
been caught with his hand in the cash box too many times. Let’s analyze what the Markov model
tells us about him.
From t, we would know that eventually 20 out of 261 voters will vote for that scoundrel A every time. This is
true even if we started with politician C in office. We establish this by setting the initial probability distribution
at . We could have started with A in office by setting the initial probability distribution at .
0,0,1
1,0,0
However , the gratifying part is that scoundrel A will probably never win the office two times. Surprisingly,
while C is the preferred candidate when A is initially in office, the eventual long-term candidate of choice is B.
We would characterize his eventual success as a “landslide.”
1 2
1 .95 .05
2 0 1
s s
s
s
.25 .35 .40
.10 .65 .25
0 .5 .5
ABC
A
B
C
