Download Continuous Time Markov Chains , Lecture Notes - Mathematics and more Study notes Operational Research in PDF only on Docsity!
IEOR 4106: Introduction to Operations Research: Stochastic Models
Spring 2011, Professor Whitt
Class Lecture Notes: Tuesday, March 1.
Continuous-Time Markov Chains, Ross Chapter 6
Problems for Discussion and Solutions
- Pooh Bear and the Three Honey Trees. A bear of little brain named Pooh is fond of honey. Bees producing honey are located in three trees: tree A, tree B and tree C. Tending to be somewhat forgetful, Pooh goes back and forth among these three honey trees randomly (in a Markovian manner) as follows: From A, Pooh goes next to B or C with probability 1/2 each; from B, Pooh goes next to A with probability 3/4, and to C with probability 1/4; from C, Pooh always goes next to A. Pooh stays a random time at each tree. (Assume that the travel times can be ignored.) Pooh stays at each tree an exponential length of time, with the mean being 5 hours at tree A or B, but with mean 4 hours at tree C.
(a) Construct a CTMC enabling you to find the limiting proportion of time that Pooh spends at each honey tree.
(b) What is the average number of trips per day Pooh makes from tree B to tree A? ANSWERS: (a) Find the limiting fraction of time that Pooh spends at each tree. ———————————————————————- These problems are part of a longer set of lecture notes, which have been posted on the web page. The focus here is on different ways to model. We are thus focusing on Section 3 of the notes.
For this problem formulation, it is natural to use the SMP (semi-Markov process) formu- lation of a CTMC (continuous-time Markov chain), involving the embedded DTMC and the mean holding times in each state. (See Sections 6.2 and 7.6 of Ross.) We thus define the embedded transition matrix P directly and the mean holding times 1/νi directly. Note that this problem is formulated directly in terms of the DTMC, describing the random motion at successive transitions, so it is natural to use this initial modelling approach. Here the transition matrix for the DTMC is
P =
A
B
C
In the displayed transition matrix P , we have only labelled the rows. The columns are assumed to be labelled in the same order.
In general, the steady-state probability is
αi = ∑πi(1/νi) j πj^ (1/νj^ )
In this case, the steady state probability vector of the discrete-time Markov chain is obtained by solving π = πP , yielding
π = (
Then the final steady-state distribution, accounting for the random holding times is
α = (
You could alternatively work with the infinitesimal transition rate matrix Q. If we want to define the infinitesimal transition matrix Q (Ross uses lower case q), then we can do so by setting Qi,j = νiPi,j for i 6 = j.
As usual, the diagonal elements Qi,i are set equal to minus the ith^ row sum; i.e.,
Qi,i = −
j:j 6 =i
Qi,j.
But we do not need the Q matrix to solve for the steady-state distribution. We can use the SMP representation. If we do define Q, then we can alternatively obtain the steady-state probability vector α above by solving the equation αQ = 0, where the elements αi are required to sum to 1. Note that this is just a system of linear equations, just like π = πP. You should work to understand why we here have 0 instead of α for the vector.
———————————————————————- (b) What is the average number of trips Pooh makes per day from tree B to tree A? ———————————————————————- The long-run fraction of time spent at B is 1/4, by part (a). Thus, on average, Pooh spends 6 hours per day at tree B. When at tree B, the rate of trips from B is 1/5 per hour (the reciprocal of 5 hours), and thus, on average, Pooh makes 6/5 = 1.2 trips per day from tree B. However, 3/4 of the trips from tree B are to tree A, so the average number of trips per day from B to A is (6/5) × (3/4) = (18/20) = 0.9.
———————————————————————-
Rate Diagram
(2,1)
(1,2)
2
0
1
J 1
J 2
J 2
J 1
(^1) E 1 E
E 2
E 2
J j = rate copier j fails, E j = rate copier j repaired
Figure 1: A rate diagram showing the transition rates among the 5 states in Problem 2, involving copier breakdown and repair.
We can thus define the transition-rate matrix Q. For this purpose, recall that the diagonal entries are minus the sum of the non-diagonal row elements, i.e.,
Qi,i = −
j,j 6 =i
Qi,j for all i.
That can be explained by the fact that the rate matrix Q is defined to be the derivative (from above) of the transition matrix P (t) at t = 0; see the CTMC notes. In other words, the rate matrix should be
Q =
−(γ 1 + γ 2 ) γ 1 γ 2 0 0 β 1 −(γ 2 + β 1 ) 0 γ 2 0 β 2 0 −(γ 1 + β 2 ) 0 γ 1 0 0 β 1 −β 1 0 0 β 2 0 0 −β 2
(c) Suppose that γ 1 = 1, β 1 = 2, γ 2 = 3 and β 2 = 4. Find the stationary distribution. ———————————————————————-
We first substitute the specified numbers for the rates γi and βi in the rate matrix Q above,
obtaining
Q =
Then we solve the system of linear equations αQ = 0 with αe = 1, which is easy to do with a computer and is not too hard by hand. Just as with DTMC’s, one of the equations in αQ = 0 is redundant, so that with the extra added equation αe = 1, there is a unique solution. Performing the calculation, we see that the limiting probability vector is
α ≡ (α 0 , α 1 , α 2 , α(1,2), α(2,1)) =
Thus, the long-run proportion of time that copier 1 is working is α 0 + α 2 = 80/ 129 ≈ 0 .62, while the long-run proportion of time that copier 2 is working is α 0 + α 1 = 60/ 129 ≈ 0 .47. The long-run proportion of time that the repairman is busy is α 1 + α 2 + α(1,2) + α(2,1) = 1 − α 0 = 85 / 129 ≈ 0 .659,
———————————————————————- (d) Now suppose, instead, that machine 1 is much more important than machine 2, so that the repairman will always service machine 1 if it is down, regardless of the state of machine 2. Formulate a CTMC for this modified problem and find the stationary distribution.
———————————————————————- With this alternative repair strategy, we can revise the state space. Now it does suffice to use 4 states, letting the state correspond to the set of failed copiers, because now we know what the repairman will do when both copiers are down; he will always work on copier 1. Thus it suffices to use the single state (1, 2) to indicate that both machines have failed. There now is only one possible transition from state (1, 2): Q(1,2), 2 = μ 1. We display the revised rate diagram in Figure 2 below. The associated rate matrix is now
Q =
−(γ 1 + γ 2 ) γ 1 γ 2 0 β 1 −(γ 2 + β 1 ) 0 γ 2 β 2 0 −(γ 1 + β 2 ) γ 1 0 0 β 1 −β 1
or, with the numbers assigned to the parameters,
Q =
Just as before, we obtain the limiting probabilities by solving αQ = 0 with αe = 1. Now we obtain
α ≡ (α 0 , α 1 , α 2 , α(1,2)) =
Thus, the long-run proportion of time that copier 1 is working is α 0 +α 2 = 38/57 = 2/ 3 ≈ 0 .67, while the long-run proportion of time that copier 2 is working is α 0 + α 1 = 24/ 57 ≈ 0 .42. The
