Continuous Time Markov Processes:
An Introduction
Thomas M. Liggett
Department of Mathematics, UCLA
Current address: Department of Mathematics, University of California,
Los Angeles CA 90095
Prepare for your exams
Study with the several resources on Docsity
Earn points to download
Earn points by helping other students or get them with a premium plan
Guidelines and tips
Prepare for your exams
Study with the several resources on Docsity
Earn points to download
Earn points by helping other students or get them with a premium plan
Continuous Time Markov Processes: An Introduction | Cryptography | MATH 0209A, Lecture notes of Cryptography and System Security
Typology: Lecture notes
2019/2020
1 / 24
This page cannot be seen from the preview
Don't miss anything!
Related documents
Partial preview of the text
Download Continuous Time Markov Processes: An Introduction | Cryptography | MATH 0209A and more Lecture notes Cryptography and System Security in PDF only on Docsity!
Continuous Time Markov Processes:
An Introduction
Thomas M. Liggett
Department of Mathematics, UCLA Current address: Department of Mathematics, University of California, Los Angeles CA 90095
E-mail address: [email protected]
1991 Mathematics Subject Classification. Primary 60J25, 60J27, 60J65; Secondary 35J05, 60J35, 60K
Key words and phrases. Probability theory, Brownian motion, Markov chains, Feller processes, the voter model, the contact process, exclusion processes, stochastic calculus, Dirichlet problem
This work was supported in part by NSF Grant #DMS-0301795.
Abstract. This is a textbook intended for use in the second semester of the basic graduate course in probability theory and/or in a semester topics course to follow the one year course.
- Chapter 1. One Dimensional Brownian Motion Preface xi
- §1.1. Some motivation
- §1.2. The multivariate Gaussian distribution
- §1.3. Processes with stationary independent increments
- §1.4. Definition of Brownian motion
- §1.5. The construction
- §1.6. Path properties
- §1.7. The Markov property
- §1.8. The strong Markov property and applications
- §1.9. Continuous time martingales and applications
- §1.10. The Skorokhod embedding
- §1.11. Donsker’s Theorem and applications
- Chapter 2. Continuous Time Markov Chains
- §2.1. The basic setup
- §2.2. Some examples
- §2.3. From Markov chain to infinitesimal description
- §2.4. Blackwell’s example
- §2.5. From infinitesimal description to Markov chain
- §2.6. Stationary measures, recurrence and transience
- §2.7. More examples
- Chapter 3. Feller Processes viii Contents
- §3.1. The basic setup
- §3.2. From Feller process to infinitesimal description
- §3.3. From infinitesimal description to Feller process
- §3.4. A few tools
- §3.5. Applications to Brownian motion and its relatives
- Chapter 4. Interacting Particle Systems
- §4.1. Some motivation
- §4.2. Spin systems
- §4.3. The voter model
- §4.4. The contact process
- §4.5. Exclusion processes
- Chapter 5. Stochastic Integration
- §5.1. Some motivation
- §5.2. The Itˆo integral
- §5.3. Itˆo’s formula and applications
- §5.4. Brownian local time
- §5.5. Connections to Feller processes on R^1
- Problem Chapter 6. Multidimensional Brownian Motion and the Dirichlet
- §6.1. Harmonic functions and the Dirichlet problem
- §6.2. Brownian motion on Rn
- §6.3. Back to the Dirichlet problem
- §6.4. The Poisson equation
- Appendix
- §A.1. Some measure theory
- §A.2. Some analysis
- §A.3. The Poisson distribution
- §A.4. Random series and laws of large numbers
- §A.5. The central limit theorem and related topics
- §A.6. Discrete time martingales
- §A.7. Discrete time Markov chains
- §A.8. The renewal theorem
- §A.9. Harmonic functions for discrete time Markov chains
Contents ix
§A.10. Subadditive functions 251
Bibliography 253
Index 255
Preface
Students are often surprised when they first hear the following definition: “A stochastic process is a collection of random variables indexed by time”. There seems to be no content here. There is no structure. How can anyone say anything of value about a stochastic process? The content and struc- ture are in fact provided by the definitions of the various classes of stochastic processes that are so important for both theory and applications. There are processes in discrete or continuous time. There are processes on countable or general state spaces. There are Markov processes, random walks, Gauss- ian processes, diffusion processes, martingales, stable processes, infinitely divisible processes, stationary processes, and many more. There are entire books written about each of these types of stochastic process.
The purpose of this book is to provide an introduction to a particularly important class of stochastic processes – continuous time Markov processes. My intention is that it be used as a text for the second half of a year-long course on measure theoretic probability theory. The first half of such a course typically deals with the classical limit theorems for sums of independent random variables (laws of large numbers, central limit theorems, random infinite series), and with some of the basic discrete time stochastic processes (martingales, random walks, stationary sequences). Alternatively, the book can be used in a semester-long special topics course for students who have completed the basic year-long course. In this case, students will probably already be familiar with the material in Chapter 1, so the course would start with Chapter 2.
The present book stresses the new issues that appear in continuous time. A difference that arises immediately is in the definition of the process. A discrete time Markov process is defined by specifying the law that leads from
xi
xii Preface
the state at one time to that at the next. This approach is not possible in continuous time. In most cases, it is necessary to describe the transition law infinitesimally in time, and then prove under appropriate conditions that this description leads to a well defined process for all time.
We begin with an introduction to Brownian motion, which is certainly the most important continuous time stochastic process. It is a special case of many of the types listed above – it is Markov, Gaussian, a diffusion, a martingale, stable, and infinitely divisible. It plays a fundamental role in stochastic calculus, and hence in financial mathematics. Through Donsker’s theorem, it provides a framework for far reaching generalizations of the clas- sical central limit theorem. While we will concentrate on this one process in Chapter 1, we will also discuss there the extent to which results and tech- niques apply (or do not apply) more generally. The infinitesimal definition discussed in the previous paragraph is not necessary in the case of Brownian motion; however it sets the stage for the set up that is required for processes that are defined in that way.
Next we discuss the construction problem for continuous time Markov chains. (The word “chain” here refers to the countability of the state space.) The main issue is to determine when the infinitesimal description of the process (given by the Q-matrix) uniquely determines the process via Kol- mogorov’s backward equations.
With an understanding of these two examples – Brownian motion and continuous time Markov chains – we will be in a position to consider the issue of defining the process in greater generality. Key here is the Hille- Yosida theorem, which links the infinitesimal description of the process (the generator) to the evolution of the process over time (the semigroup). Since usually only the generator is known explicitly, we will discuss how one de- duces properties of the process from information about the generator. The main examples at this point are variants of Brownian motion, in which the relative speed of the particle varies spatially, and/or there is a special be- havior at the boundary of the state space.
As an application of the theory of semigroups and generators, we then provide an introduction to a somewhat more recently developed area of prob- ability theory – interacting particle systems. This is a class of probabilistic models that come up in many areas of application – physics, biology, com- puter science, and even a bit in economics and sociology. Infinitely many agents evolve in time according to certain probabilistic rules that involve interactions among the agents. The nature of these rules is dictated by the area of application. The main issue here is the nature of the long time behavior of the process.
xiv Preface
typically covered most of the material in Chapters 1-3 and 6 in the third quarter of the graduate probability course, and Chapters 4 and 5 in special topics courses. There is more than enough material here for a semester course, even if Chapter 1 is skipped because students are already familiar with one dimensional Brownian motion.
As is usually the case with a text of this type, I have benefitted greatly from the work of previous authors, including those of [ 12 ], [ 18 ], [ 20 ], [ 21 ], [ 22 ], [ 27 ], [ 35 ], [ 37 ], [ 38 ], and [ 42 ]. I also appreciate the comments and corrections provided by P. Caputo, S. Roch, and A. Vandenberg-Rodes, and especially T. Richthammer, who read much of the book very carefully.
Thomas M. Liggett
Chapter 1
One Dimensional
Brownian Motion
1.1. Some motivation
The biologist Robert Brown noticed almost two hundred years ago that bits of pollen suspended in water undergo chaotic behavior. The bits of pollen are much more massive than the molecules of water, but of course there are many more of these molecules than there are bits of pollen. The chaotic motion of the pollen is the result of many infinitesimal jolts by the water molecules. By the central limit theorem, the law of the motion of the pollen should be closely related to the normal distribution. We now call this law Brownian motion.
During the past half century or so, Brownian motion has turned out to be a very versatile tool for both theory and applications. As we will see in Chapter 6, it provides a very elegant and general treatment of the Dirichlet problem, which asks for harmonic functions on a domain with prescribed boundary values. It is also the main building block for the theory of stochastic calculus, which is the subject of Chapter 5. Via stochastic calculus, it has played an important role in the development of financial mathematics.
As we will see later in this chapter, Brownian paths are quite rough – they are of unbounded variation in every time interval. Therefore, integrals with respect to them cannot be defined in the Stieltjes sense. A new type of integral must be defined, which carries the name of K. Itˆo, and more recently, of W. Doeblin. This new integral has some unexpected properties.
1.2. The multivariate Gaussian distribution 3
Definition 1.1. The real random vector (ξ 1 , ..., ξn) is said to be multivariate Gaussian if all linear combinations
∑^ n
k=
akξk
have univariate Gaussian distributions.
Remark 1.2. (a) If ξ 1 , ..., ξn are independent Gaussians, then (ξ 1 , ..., ξn) is multivariate Gaussian.
(b) Definition 1.1 is much stronger than the statement that each ξk is Gaussian. For example, suppose ζ is standard Gaussian, and
ξ =
+ζ if |ζ| ≤ 1; −ζ if |ζ| > 1.
Then ξ is also standard Gaussian. However, since |ζ + ξ| ≤ 2 and ζ + ξ is not constant, ζ + ξ is not Gaussian.
Remark 1.3. Definition 1.1 has a number of advantages over the alterna- tive, in which one specifies the joint density of (ξ 1 , ..., ξn):
(a) It does not require that (ξ 1 , ..., ξn) have a density. For example, (ξ, ξ) is bivariate Gaussian if ξ is Gaussian.
(b) It makes the next result immediate.
Proposition 1.4. Suppose ξ = (ξ 1 , ..., ξn) is Gaussian and A is an m × n matrix. Then the random vector ζ = Aξ is also Gaussian.
Proof. Any linear combination of ζ 1 , ..., ζm is some other linear combination of ξ 1 , ..., ξn.
An important property of a multivariate Gaussian vector ξ is that its distribution is determined by the mean vector Eξ and the covariance matrix, whose (i, j) entry is Cov(ξi, ξj ). To check this statement, we use character- istic functions. Recall that the characteristic function of a random variable with the N (m, σ^2 ) distribution is
exp
itm −
t^2 σ^2
Therefore, if ξ = (ξ 1 , ..., ξn) is multivariate Gaussian, its joint characteristic function is given by
φ(t 1 , ..., tn) = E exp
i
∑^ n
j=
tj ξj
= exp
im −
σ^2
4 1. One Dimensional Brownian Motion
where m and σ^2 are the mean and variance of
∑n j=1 tj^ ξj^ :
m =
∑^ n
j=
tj Eξj and σ^2 =
∑^ n
j,k=
tj tkCov(ξj , ξk).
Since φ(t 1 , ..., tn) depends on ξ only through its mean vector and covari- ance matrix, these determine the characteristic function of ξ, and hence its distribution by Proposition A.22. This observation has the following conse- quences:
Proposition 1.5. If ξ = (ξ 1 , ..., ξn) is multivariate Gaussian, then ξ 1 , ..., ξn are independent if and only if they are uncorrelated.
Proof. That independence implies uncorrelatedness is always true for ran- dom variables with finite second moments. For the converse, suppose that ξ 1 , ..., ξn are uncorrelated, i.e., that Cov(ξj , ξk) = 0 for j 6 = k. Take ζ 1 , ..., ζn to be independent, with ζi having the same distribution as ξi. Then ξ and ζ = (ζ 1 , ..., ζn) have the same characteristic function, and hence the same distribution, by Proposition A.22. It follows that ξ 1 , ..., ξn are indepen- dent.
Exercise 1.6. Show that if ξ = (ξ 1 , ..., ξn), where ξ 1 , ..., ξn are i.i.d. stan- dard Gaussian random variables, and O is an n × n orthogonal matrix, then Oξ has the same distribution as ξ.
Exercise 1.7. (a) Suppose that ξk ⇒ ξ and that ξk has the N (mk, σ^2 k) distribution for each k. Prove that ξ is N (m, σ^2 ) for some m and σ^2 , and that mk → m and σ^2 k → σ^2.
(b) State and prove an analogue of (a) for Gaussian random vectors.
The main topic of this book is a class of stochastic processes; in this chapter, they are Gaussian. We conclude this section with formal definitions of these concepts.
Definition 1.8. A stochastic process is a collection of random variables indexed by time. It is a discrete time process if the index set is a subset of { 0 , 1 , 2 , ...}, and a continuous time process if the index set is [0, ∞).
Definition 1.9. A stochastic process X(t) is Gaussian if for any n ≥ 1 and any choice of times t 1 , ..., tn, the random vector (X(t 1 ), ..., X(tn)) has a multivariate Gaussian distribution. Its mean and covariance functions are EX(t) and Cov(X(s), X(t)) respectively.
6 1. One Dimensional Brownian Motion
we may assume that 0 = t 0 < t 1 < · · · < tn. Summing by parts, and using X(0) = 0, we see that there are bk’s so that
∑^ n
k=
akX(tk) =
∑^ n
k=
bk
[
X(tk) − X(tk− 1 )
]
The right side is a sum of independent Gaussians, and is hence Gaussian. To check the covariance statement, take s < t and write
Cov(X(s), X(t)) = EX(s)X(t) = EX(s)
[
X(t) − X(s)
]
- EX^2 (s) = s = s ∧ t.
For the converse, assume (b). Then for s < t, X(t) − X(s) is Gaussian with mean zero and
V ar(X(t) − X(s)) = t − 2(s ∧ t) + s = t − s,
so the process has stationary increments and has the right marginal distri- butions.
If 0 ≤ t 1 < t 2 < · · · < tn, the vector of increments can be written in the form
(X(t 2 ) − X(t 1 ), ..., X(tn) − X(tn− 1 )) = A(X(t 1 ), ..., X(tn))
for an appropriately chosen matrix A. Therefore, by Proposition 1.4, the vector of increments is Gaussian. So, in order to check the independence of the increments, it is enough by Proposition 1.5 to show that the increments are uncorrelated. To do so, take u < v ≤ s < t. Then
Cov(X(v) − X(u), X(t) − X(s)) =v ∧ t − v ∧ s − u ∧ t + u ∧ s =v − v − u + u = 0.
Exercise 1.12. Suppose X(t) is a stochastic process with stationary inde- pendent increments that satisfies EX(1) = 0, EX^2 (1) = 1, and X(t) has the same distribution as
tX(1) for t ≥ 0. Show that X(t)−X(s) is N (0, |t−s|) for s, t ≥ 0.
Definition 1.13. A stochastic process (X(t), t ≥ 0), is said to have contin- uous paths if
(1.3) P ({ω : X(t, ω) is continuous in t}) = 1.
Definition 1.14. Standard Brownian motion B(t) is a stochastic process with continuous paths that satisfies the equivalent properties (a) and (b) in Proposition 1.11.
1.4. Definition of Brownian motion 7
Of course, it is not at all obvious that there exists a probability space on which one can construct a standard Brownian motion. Showing that this is the case is the objective of the next section. Taking this for granted for the time being, here are two exercises that provide some practice with the definition. In both cases, B(t) is standard Brownian motion.
Exercise 1.15. Let
X(t) =
∫ (^) t
0
B(s)ds.
(a) Explain why X(t) is a Gaussian process. (b) Compute the mean and covariance functions of X. (c) Compute E(X(t) − X(s))^2 , and compare its rate of decay as t ↓ s with that of E(B(t) − B(s))^2.
Exercise 1.16. Compute
P (B(s) > 0 , B(t) > 0), 0 < s < t.
(Recall Exercise 1.6.)
If it were not for the path continuity requirement in Definition 1.14, the existence of standard Brownian motion on some probability space would follow from Kolmogorov’s extension theorem – see Theorem A.1. Since every event in this probability space is determined by the process at only countably many times, and continuity of a path is not determined by its values at countably many times, this approach would lead to the awkward situation in which the set in question, C = {ω : B(t, ω) is continuous in t}, is not an event. Therefore, we would not even be able to discuss the issue of whether its probability is 1.
The situation is even more serious than this. Even if C were measurable, it would not be possible to prove that P (C) = 1 follows from properties (a) and (b). To see this, take a process B on some probability space satisfying properties (a) and (b), and let τ be a continuous random variable that is independent of B. Define a new process by
X(t, ω) =
B(t, ω) if t 6 = τ (ω); B(t, ω) + 1 if t = τ (ω).
Then not both B and X can have continuous paths. However, since
P (X(t) = B(t)) = P (τ 6 = t) = 1
for every t, it follows that X also satisfies properties (a) and (b) in Propo- sition 1.11. Therefore, there exist processes that satisfy properties (a) and (b) but do not have continuous paths. Note that
P (X(t) = B(t) for all t) = 0.
1.5. The construction 9
(ii) Using the Beta integral ∫ (^1)
0
uα−^1 (1 − u)β−^1 du =
Γ(α)Γ(β) Γ(α + β)
check that
I(α 1 , ..., αk) =
Γ(α 1 )Γ(α 2 ) Γ(α 1 + α 2 )
I(α 1 + α 2 , α 3 , ..., αk),
so that
I
[Γ(1/2)]k Γ(k/2)(k/2)
(iii) Check the recursion E|Z|k^ = (k − 1)E|Z|k−^2 , k ≥ 2 ,
and use it to compute E|Z|k.
1.5. The construction
In this section, we will give one construction of Brownian motion. Another construction is outlined in the exercises.
Theorem 1.20. There exists a probability space (Ω, F, P ) on which stan- dard Brownian motion B exists.
Proof. Properties (a) and (b) in Proposition 1.11 specify the finite dimen- sional distributions of B. By Kolmogorov’s extension theorem, Theorem A.1, there exists a probability space on which the random variables B(t) are defined for t ∈ Q+, the set of positive rationals. We will prove that for every N ≥ 1,
(1.4) B(t, ω) is uniformly continuous in t for t ∈ Q ∩ [0, N ] a.s.
Once this is done, B(t) can be extended to all t ≥ 0 by continuity. Note that the uniformity is important here. If b /∈ Q, the function
f (t) =
1 if t ≥ b; 0 if t < b
is continuous on Q, but not uniformly continuous on Q. It cannot be ex- tended from Q to R^1 by continuity.
Let ∆n = sup s,t∈Q∩[0,N ] |s−t|≤ (^1) n
|B(t) − B(s)|.
We need to prove that ∆n → 0 a.s. Since ∆n is decreasing in n, it is enough to prove convergence in probability. To see this, recall that convergence in
10 1. One Dimensional Brownian Motion
probability implies that a subsequence converges a.s. By the monotonicity, convergence along a subsequence implies convergence along the full sequence.
The next step is to reduce the number of arguments of B that need to be considered in the supremum. To do so, let
Yk,n = sup t∈Q k− n 1 ≤t≤ kn
∣B(t)^ −^ B
k − 1 n
Then
(1.5) ∆n ≤ 3 max 1 ≤k≤nN
Yk,n.
The factor of 3 above arises in the following way. For a given t, choose k so that k− n 1 ≤ t ≤ kn. If |s − t| ≤ (^1) n , then k− n 2 ≤ s ≤ k+1 n. If, for example, k n ≤^ s^ ≤^
k+ n ,^ bound^ |B(t)^ −^ B(s)|^ by ∣∣ ∣ ∣B(t)^ −^ B
k − 1 n
∣B
k n
− B
k − 1 n
∣B(s)^ −^ B
k n
which is at most the right side of (1.5).
Noting that the distribution of Yk,n does not depend on k, write for > 0
(1.6) P ( max 1 ≤k≤nN
Yk,n > ) ≤
∑^ nN
k=
P (Yk,n > ) = nN P (Y 1 ,n > ).
To check that the right side above tends to 0 as n → ∞, we will apply Doob’s inequality for discrete time submartingales – see Theorem A.33. If 0 < t 1 < · · · < tm are rational, (B(t 1 ), ..., B(tm)) is a martingale, since the successive differences are independent and have mean 0. Therefore, (B^4 (t 1 ), ..., B^4 (tm)) is a (nonnegative) submartingale. Doob’s inequality gives
P ( max 1 ≤k≤m
|B(tk)| > ) ≤
^4
EB^4 (tm).
Note that the bound on the right side depends on tm, but not on m – this is very important in the next step. Applying this to a sequence of subsets that exhausts Q ∩ [0, (^1) n ], and using the fact that EB^4 (t) is increasing in t, we see that
P (Y 1 ,n > ) ≤
^4
EB^4
n
Since B(t) has the same distribution as
tB(1) (the N (0, t) distribution), we conclude that
P (Y 1 ,n > ) ≤
EB^4 (1)
n^2 ^4
so that the right side of (1.6) tends to 0 as n → ∞. Therefore, ∆n → 0 in probability by (1.5) as required.