Stochastic Processes with Applications

The front matter includes the title page, series page, copyright page, dedication, TOC, preface to the classics edition, preface, and sample course outline

1 WHAT IS A STOCHASTIC PROCESS?

Denoting by Xn the value of a stock at an nth unit of time, one may represent its (erratic) evolution by a family of random variables { X0 , X1 ,…} indexed by the discrete-time parameter n∈ ℤ+ . The number Xt of car accidents in a city during the time interval [0, t] gives rise to a collection of random variables { Xt :t⩾0} indexed by the continuous-time parameter t. The velocity Xu at a point u in a turbulent wind field provides a family of random variables { Xu :u∈ ℝ3 } indexed by a multidimensional spatial parameter u. More generally we make the following definition.

Definition 1.1. Given an index set I, a stochastic process indexed by I is a collection of random variables { Xλ :λ∈I} on a probability space (Ω, ℱ, P) taking values in a set S. The set S is called the state space of the process.

In the above, one may take, respectively: (i) I= ℤ+ , S= ℝ+ ; (ii) I=[0,∞) , S= ℤ+ ; (iii) I= ℝ3 , S= ℝ3 . For the most part we shall study stochastic processes indexed by a one-dimensional set of real numbers (e.g., time). Here the natural ordering of numbers coincides with the sense of evolution of the process. This order is lost for stochastic processes indexed by a multidimensional parameter; such processes are usually referred to as random fields. The state space S will often be a set of real numbers, finite, countable, (i.e., discrete) or uncountable. However, we also allow for the possibility of vector-valued variables. As a matter of convenience in notation the index set is often suppressed when the context makes it clear. In particular, we often write { Xn } in place of { Xn :n=0,1,2,…} and { Xt } in place of { Xt :t⩾0} .

1 MARKOV DEPENDENCE

Consider a discrete-parameter stochastic process { Xn } . Think of X0 , X1 ,… , Xn−1 as “the past,” Xn as “the present,” and Xn+1 , Xn+2 ,… as “the future” of the process relative to time n. The law of evolution of a stochastic process is often thought of in terms of the conditional distribution of the future given the present and past states of the process. In the case of a sequence of independent random variables or of a simple random walk, for example, this conditional distribution does not depend on the past. This important property is expressed by Definition 1.1.

Definition 1.1. A stochastic process { X0 , X1 ,… , Xn ,…} has the Markov property if, for each n and m, the conditional distribution of Xn+1 ,… , Xn+m given X0 , X1 ,… , Xn is the same as its conditional distribution given Xn alone. A process having the Markov property is called a Markov process. If, in addition, the state space of the process is countable, then a Markov process is called a Markov chain.

In view of the next proposition, it is actually enough to take m=1 in the above definition.

Proposition 1.1. A stochastic process X0 , X1 , X2 ,… has the Markov property if and only if for each n the conditional distribution of Xn+1 given X0 , X1 ,… , Xn is a function only of Xn .

Proof. For simplicity, take the state space S to be countable. The necessity of the condition is obvious.

1 INTRODUCTION TO BIRTH-DEATH CHAINS

Each of the simple random walk examples described in Section 1.3 has the special property that it does not skip states in its evolution. In this vein, we shall study time-homogeneous Markov chains called birth—death chains whose transition law takes the form pij = βi if j=i+1 δi if j=i−1 αi if j=i0otherwise, 3.1 where αi + βi + δi =1 . In particular, the displacement probabilities may depend on the state in which the process is located.

Example 1. (The Bernoulli—Laplace Model). A simple model to describe the mixing of two incompressible liquids in possibly different proportions can be obtained by the following considerations. Consider two containers labeled box I and box II, respectively, each having N balls. Among the total of 2N balls, there are 2r red and 2w white balls, 1≤r≤w . At each instant of time, a ball is randomly selected from each of the boxes, and moved to the other box. The state at each instant is the number of red balls in box I.

1 INTRODUCTION TO CONTINUOUS-TIME MARKOV CHAINS

Suppose that { Xt :t⩾0} is a continuous-parameter stochastic process with a finite or denumerably infinite state space S. Just as in the discrete-parameter case, the Markov property here also refers to the property that the conditional distribution of the future, given past and present states of the process, does not depend on the past. In terms of finite-dimensional events, the Markov property requires that for arbitrary time points 0≤ s0 < s1 <⋯< sk <s<t< t1 <⋯< tn and states i0 ,… , ik ,i,j, j1 ,… , jn in S P ( Xt =j, X t1 = j1 ,… , X tn = jn | X s0 = i0 ,… , X sk = ik, Xs =i) =P ( Xt =j, X t1 = j1 ,… , X tn = jn | Xs =i). 4.1 In other words, for any sequence of time points 0≤ t0 < t1 <⋯ , the discrete parameter process Y0 ≔ X t0 , Y1 ≔ X t1 ,… is a Markov chain as described in Chapter II. The conditional probabilities pij (s,t)=P ( Xt =j | Xs =i) ,0≤s<t , are collectively referred to as the transition probability law for the process. In the case pij (s,t) is a function of t − s, the transition law is called time-homogeneous, and we write pij (s,t)= pij (t−s) .

Simple examples of continuous-parameter Markov chains are the continuous-time random walks, or processes with independent increments on countable state space. Some others are described in the examples below.

Example 1. (The Poisson Process).

1 INTRODUCTION AND DEFINITION

One-dimensional unrestricted diffusions are Markov processes in continuous time with state space S= (a,b) , −∞≤a<b≤∞ , having continuous sample paths. The simplest example is Brownian motion { Yt } with drift coefficient μ and diffusion coefficient σ2 >0 introduced in Chapter I as a limit of simple random walks. In this case, S= (−∞,∞) . The transition probability distribution p(t; x, dy) of Ys+t given Ys =x has a density given by p (t;x,y) = 1 (2π σ2 t) 1/2 e− 1 2 σ2 t (y−x−μt)2 . 5.1 Because { Yt } has independent increments, the Markov property follows directly. Notice that (1.1) does not depend on s; i.e., { Yt } is a time-homogeneous Markov process. As before, by a Markov process we mean one having a time-homogeneous transition law unless stated otherwise.

One may imagine a Markov process { Xt } that has continuous sample paths but that is not a process with independent increments. Suppose that, given Xs =x , for (infinitesimal) small times t, the displacement Xs+t − Xs = Xs+t −x has mean and variance approximately tμ(x) and t σ2 (x) , respectively. Here μ(x) and σ2 (x) are functions of the state of x, and not constants as in the case of { Yt } . The distinction between { Yt } and { Xt } is analogous to that between a simple random walk and a birth—death chain. More precisely, suppose E ( Xs+t − Xs | Xs =x)=tμ (x)+o (t), E ( ( Xs+t − Xs )2 | Xs =x)=t σ2 (x)+o (t), E ( | Xs+t − Xs |3 | Xs =x)= o (t), 5.2 hold, as t ↓ 0, for every x ∈ S.

1 FINITE-HORIZON OPTIMIZATION

Consider a finite set S, called the state space, a finite set A, called the action space, and, for each pair (x, a) ∈ S × A, a probability function p(y; x, a) on S: p (y;x,a)⩾0, ∑ y∈S p (y;x,a) =1. 6.1 The function p(y; x, a) denotes the probability that the state in the next period will be y, given that the present state is x and an action a has been taken.

A policy (or, a feasible policy) is a sequence of functions ( ƒ0 , ƒ1 ,…) defined as follows. ƒ0 is a function on S into A. If the state in period k=0 is x0 then an action ƒ0 ( x0 ) = a0 is taken. Given the state x0 and the action a0 = ƒ0 ( x0 ) , the state in period k=1 is x1 with probability p ( x1 ; x0 , ƒ0 ( x0 ) ) . Now ƒ1 is a function on S × A × S into A. Given the triple x0 , ƒ0 ( x0 ), x1 , an action ƒ1 ( x0 , ƒ0 ( x0 ), x1 )= a1 is taken. Given x0 , a0 and the state x1 and action a1 , the probability that the state in period k=2 is x2 is p ( x2 ; x1 , a1 ) . Similarly, ƒ2 is a function on S × A × S × A × S into A; given x0 , a0 , x1 , a1 , x2 an action a2 = ƒ2 ( x0 , a0 , x1 , a1 , x2 ) is taken. Given all the states and actions up to period k=2 (namely x0 , a0 = ƒ0 ( x0 ) , x1 , a1 = ƒ1 ( x0 , a0 , x1 ) , x2 , a2 = ƒ2 ( x0 , a0 , x1 , a1 , x2 ) ), the probability that a state x3 occurs during period k=3 is p ( x3 ; x2 , a2 ) , and so on. In general, a policy is a sequence of functions { ƒk :k=0,1,2,…} such that ƒk is a function on (S×A)k ×S into A (k=1,2,…) , with ƒ0 a function on S into A. The sequence may be finite { ƒk :k=0,1,… ,N} , called the finite-horizon case, or it may be infinite, the infinite-horizon case.

Consider first the case of a finite horizon.

1 THE STOCHASTIC INTEGRAL

A diffusion { Xt } on ℝ1 may be thought of as a Markov process that is locally like a Brownian motion. That is, in some sense the following relation holds, d Xt =μ ( Xt ) dt+σ ( Xt ) d Bt , 7.1 where μ(⋅), σ(⋅) are given functions on ℝ1 , and { Bt } is a standard one-dimensional Brownian motion. In other words, conditionally given { Xs :0≤s≤t} , in a small time interval (t, t + dt] the displacement d Xt ≔ Xt+dt − Xt is approximately the Gaussian random variable μ ( Xt )dt+σ ( Xt ) ( Bt+dt − Bt ) , having mean μ ( Xt )dt and variance σ2 ( Xt )dt . Observe, however, that (1.1) cannot be regarded as an ordinary differential equation such as d Xt /dt=μ ( Xt )+σ ( Xt ) d Bt /dt . For, outside a set of probability 0, a Brownian path is nowhere differentiable (Exercise 1; or see Exercises 7.8, 9.3 of Chapter I). The precise sense in which (1.1) is true, and may be solved to construct a diffusion with coefficients μ(⋅), σ2 (⋅) , is described in this section and the next.

The present section is devoted to the definition of the integral version of (1.1): Xt =x+ ∫ 0 t μ ( Xs ) ds+ ∫ 0 t σ ( Xs ) d Bs . 7.2 The first integral is an ordinary Riemann integral, which is defined if μ(⋅) is continuous and s→ Xs is continuous. But the second integral cannot be defined as a Riemann—Stieltjes integral, as the Brownian paths are of unbounded variation on [0, t] for every t>0 (Exercise 1).

1 PROBABILITY SPACES

Underlying the mathematical description of random variables and events is the notion of a probability space (Ω, ℱ, P). The sample space Ω is a nonempty set that represents the collection of all possible outcomes of an experiment. The elements of Ω are called sample points. The sigmafield ℱ is a collection of subsets of Ω that includes the empty set ∅ (the “impossible event”) as well as the set Ω (the “sure event”) and is closed under the set operations of complements and finite or denumerable unions and intersections. The elements of ℱ are called measurable events, or simply events. The probability measure P is an assignment of probabilities to events (sets) in ℱ that is subject to the conditions that (i) 0≤P (F)≤1 , for each F ∈ ℱ, (ii) P (∅)=0 , P (Ω)=1 , and (iii) P ( ∪i Fi ) = ∑i P ( Fi ) for any finite or denumerable sequence of mutually exclusive (pairwise disjoint) events Fi ,i=1,2,… , belonging to ℱ. The closure properties of ℱ ensure that the usual applications of set operations in representing events do not lead to nonmeasurable events for which no (consistent) assignment of probability is possible.

The required countable additivity property (iii) gives probabilities a sufficiently rich structure for doing calculations and approximations involving limits. Two immediate consequences of (iii) are the following so-called continuity properties: if A1 ⊂ A2 ⊂⋯ is a nondecreasing sequence of events in ℱ then, thinking of ∪ n=1 ∞ An as the “limiting event” for such sequences, P ∪ n=1 ∞ An = lim n P ( An ) . 8.1 To prove this, disjointify { An } by Bn = An − An−1 ,n⩾1 , A0 =∅ , and apply (iii) to ∪ n=1 ∞ Bn = ∪ n=1 ∞ An .

The back matter includes subject index, author index and errata