Markov Chains

One of the ways of generating a sequence of dependent random variables is to think of a system evolving in time. We have time points that are discrete say T = 0, 1, ⋯, N, · · ·. The state of the system is described by a point x in the state space X of the system. The state space X comes with a natural σ-field of subsets F. At time 0 the system is in a random state and its distribution is specified by a probability distribution μ_{0} on (X , F). At successive times T = 1, 2, ⋯, the system changes its state and given the past history (x_{0}, ⋯, x_{k}) of the states of the system at times T = 0, ⋯, k − 1 the probability that system finds itself at time k in a subset A ∈ F is given by π_{k}(x_{0}, ⋯, x_{k−1} ; A). For each (x_{0}, ⋯, x_{k−1}), π_{k} defines a probability measure on (X , F) and for each A ∈ F, π_{k}(x_{0}, ⋯, x_{k−1} ; A) is assumed to be a measurable function of (x_{0}, ⋯, x_{k−1}), on the space (X^{k}, F^{k}) which is the product of k copies of the space (X , F) with itself. We can inductively define measures μ_{k} on (X^{k+1}, F^{k+1}) that describe the probability distribution of the entire history (x_{0}, ⋯, x_{k}) of the system through time k. To go from μ_{k−1} to μ_{k} we think of (X^{k+1}, F^{k+1}) as the product of (X^{k}, F^{k}) with (X , F) and construct on (X^{k+1}, F^{k+1} ) a probability measure with marginal μ_{k−1} on (X^{k}, F^{k}) and conditionals π_{k}(x_{0}, ⋯, x_{k−1}; ·) on the fibers (x_{1}, ⋯, x_{k−1}) × X. This will define μ_{k} and the induction can proceed. We may stop at some finite terminal time N or go on indefinitely. If we do go on indefinitely, we will have a consitent family of finite dimensional distributions {μ_{k}} on (X^{k+1}, F^{k+1}) and we may try to use Kolmogorov’s theorem to construct a probability measure P on the space (X^{∞}, F^{∞}) of sequences {x_{j} : j ≥ 0} representing the total evolution of the system for all times. However Kolmogorov’s theorem requires some assumptions on (X , F) that are satisfied if X is a complete separable metric space and F are the Borel sets. However, in the present context, there is a result known as Tulcea’s theorem that proves the existence of a P on (X^{∞}, F^{∞}) for any choice of (X , F), exploiting the fact that the consistent family of finite dimensional distributions μ_{k} arise from well defined successive regular conditional probability distributions.

An important subclass is generated when the transition probability depends on the past history only through the current state. In other words

π_{k}(x_{0}, ⋯, x_{k−1}; ·) = π_{k−1,k} (x_{k−1}; ·).

In such a case the process is called a Markov Process with transition probabilities π_{k−1,k} (·, ·). An even smaller subclass arises when we demand that π_{k−1,k} (·, ·) be the same for different values of k. A single transition probability π(x, A) and the initial distribution μ_{0} determine the entire process i.e. the measure P on (X^{∞}, F^{∞}). Such processes are called time-homogeneous Markov Proceses or Markov Processes with stationary transition probabilities.

Chapman-Kolmogorov Equations:. If we have the transition probabilities π_{k,k+1} of transition from time k to k + 1 of a Markov Chain it is possible to obtain directly the transition probabilities from time k to k + ℓ for any l ≥ 2. We do it by induction on ℓ. Define

π_{k,k+ℓ+1}(x, A) = ∫_{X} π_{k,k+ℓ}(x, dy) π_{k+ℓ,k+ℓ+1}(y, A)  (3.1)

or equivalently, in a more direct fashion

π_{k,k+ℓ+1}(x, A) = ∫_{X} ⋯ ∫_{X} π_{k,k+1}(x, dy_{k+1}) ⋯ π_{k+ℓ,k+ℓ+1}(y_{k+ℓ}, A)

Theorem 3.1. The transition probabilities π_{k,m}(·, ·) satisfy the relations

π_{k,n}(x, A) = ∫_{X} π_{k,m}(x, dy) π_{m,n}(y, A)  (3.2)

for any k < m < n and for the Markov Process defined by the one step transition probabilities π_{k,k+1}(·, ·), for any n > m

P [x_{n} ∈ A|Σ_{m}] = π_{m,n}(x_{m}, A) a.e.

where Σ_{m} is the σ-field of past history upto time m generated by the coordinates x_{0}, x_{1}, ⋯, x_{m}.

Proof: The identity is basically algebra. The multiple integral can be carried out by iteration in any order and after enough variables are integrated we get our identity. To prove that the conditional probabilities are given by the right formula we need to establish

P [{x_{n} ∈ A} ∩ B] = ∫_{B} π_{m,n}(x_{m}, A) dP

for all B ∈ Σ_{m} and A ∈ F. We write

P [{x_{n} ∈ A} ∩ B] = ∫_{{x_{n}∈A}∩B} dP
= ∫ ⋯ ∫_{{x_{n}∈A}∩B} dμ(x_{0}) π_{0,1}(x_{0}, dx_{1}) ⋯ π_{m−1,m}(x_{m−1}, dx_{m})
π_{m,m+1}(x_{m}, dx_{m−1}) ⋯ π_{n−1,n}(x_{n−1}, dx_{n})
= ∫ ⋯ ∫_{B} dμ(x_{0}) π_{0,1}(x_{0}, dx_{1}) ⋯ π_{m−1,m}(x_{m−1}, dx_{m})
π_{m,m+1}(x_{m}, dx_{m−1}) ⋯ π_{n−1,n}(x_{n−1}, A)

= ∫ ⋯ ∫_{B} dμ(x_{0}) π_{0,1}(x_{0}, dx_{1}) ⋯ π_{m−1,m}(x_{m−1}, dx_{m}) π_{m,n}(x_{m}, A)

= ∫_{B} π_{m,n}(x_{m}, A) dP

and we are done.

Remark: If the chain has stationary transition probabilities then the transition probabilities π_{m,n}(x, dy) from time m to time n depend only on the difference k = n − m and are given by what are usually called the k step transition probabilities. They are defined inductively by

π^{(k+1)}(x, A) = ∫_{X} π^{(k)}(x, dy) π(y, A)

and satisfy the Chapman-Kolmogorov equations

π^{(k+ℓ)}(x, A) = ∫_{X} π^{(k)}(x, dy)π^{(ℓ)}(x, A) = ∫_{§} π^{(ℓ)}(x, dy)π^{(k)}(y, A)

Let us look at some examples.

Suppose we have an urn containg a certain number of balls (nonzero) some red and others green. A ball is drawn at random and its color is noted. Then it is returned to the urn along with an extra ball of the same color. Then a new ball is drawn at random and the process continues ad infinitum. The current state of the system can be characterized by two integers r, g such that r + g ≥ 1. The initial state if the system is some r_{0}, g_{0} with r_{0} + g_{0} ≥ 1. The system can go from (r, g) to either (r + 1, g) with probability \frac{r}{r+g} or to (r, g + 1) with probability \frac{g}{r+g}. This is clearly an example of a Markov Chain with stationary transition probabilities.

Consider a queue for service in a store. Suppose at each of the times 1, 2, ⋯, a random number of new customers arrive and and join the queue. If the queue is non empty at some time, then exactly one customer will be served and will leave the queue at the next time point. The distribution of the number of new arrivals is specified by {p_{j} : j ≥ 0} where p_{j} is the probability that exactly j new customers arrive at a given time. The number of new arrivals at distinct times are assumed to be independent. The queue length is a Markov Chain on the state space X = {0, 1, ⋯, } of nonegative integers. The transition probabilities π(i, j) are given by π(0, j) = p_{j} because there is no service and nobody in the queue to begin with and all the new arrivals join the queue. On the other hand π(i, j) = p_{j−i+1} if j + 1 ≥ i ≥ 1 because one person leaves the queue after being served.

Consider a reservoir into which water flows. The amount of additional water flowing into the reservoir on any given day is random, and has a distribution α on [0, ∞). The demand is also random for any given day, with a probability distribution β on [0, ∞). We may also assume that the inflows and demands on successive days are random variables ξ_{n} and η_{n}, that have α and β for their common distributions and are all mutually independent. We may wish to assume a percentage loss due to evaporation. In any case the storage level at successive days have a recurrence relation

S_{n+1} = [(1 − p)S_{n} + ξ_{n} − η_{n}]^{+}

p is the loss and we have put the condition that the outflow is the demand unless the stored amount is less than the demand in which case the outflow is the available quantity. The current amount in storage is a Markov Process with Stationary transition probabilities.

Let X_{1}, ⋯, X_{n}, ⋯ be a sequence of independent random variables with a common distribution α. Let S_{n} = Y + X_{1} + · + X_{n} for n ≥ 1 with S_{0} = Y where Y is a random variable independent of X_{1}, …, X_{n}, … with distribution μ. Then S_{n} is a Markov chain on R with one step transition probaility π(x, A) = α(A − x) and initial distribution μ. The n step transition probability is α^{n}(A − x) where α^{n} is the n-fold convolution of α.

These are examples that can described by models of the type

x_{n} = f(x_{n−1}, ξ_{n})

where x_{n} is the current state and ξ_{n} is some random external disturbance. ξ_{n} are assumed to be independent and identically distributed. They could have two components like inflow and demand. The new state is a deterministic function of the old state and the noise.

Exercise 3.10. Verify that our previous two examples can be cast in the above form. In fact there is no loss of generality in assuming that ξ_{j} are mutually independent random variables having as common distribution the uniform distribution on the interval [0, 1].

Given a Markov Chain with stationary transition probabilities π(x, dy) on a state space (X , F), the behavior of π^{(n)}(x, dy) for large n is an important and natural question. In the best situation of independent random variables π^{(n)}(x, A) = μ(A) are independent of x as well as n. Hopefully after a long time the Chain will ”forget” its origins and π^{(n)}(x, ·) → μ(·), in some suitable sense, for some μ that does not depend on x . If that happens, then from the relation

π^{(n+1)}(x, A) = ∫ π^{(n)}(x, dy)π(y, A),

we conclude

μ(A) = ∫ π(y, A) dμ(y) for all A ∈ F

Measures that satisfy the above property abbreviated as μπ = μ are called invariant measures for the Markov Chain. If we start with the initial distribution μ which is inavariant then the probability measure P has μ as marginal at every time. In fact P is stationary i.e., invariant with respect to time translation, and can be extended to a stationary process where time runs from −∞ to +∞.

Stopping Times and Renewal Times

One of the important notions in the analysis of Markov Chains is the idea of stopping times and renewal times. A function

τ(ω) : Ω → {n : n ≥ 0}

is a random variable defined on the set Ω = X^{∞} such that for every n ≥ 0 the set {ω : τ(ω) = n} (or equivalently for each n ≥ 0 the set {ω : τ(ω) ≤ n}) is measurable with respect to the σ-field F_{n} generated by X_{j} : 0 ≤ j ≤ n. It is not necessary that τ(ω) < ∞ for every ω. Such random variable τ are called stopping times. Examples of stopping times are, constant times n ≥ 0, the first visit to a state x, or the second visit to a state x. The important thing is that in order to decide if τ ≤ n i.e. to know if what ever is supposed to happen did happen before time n the chain need be observed only up to time n. Examples of τ that are not stopping times are easy to find. The last time a site is visited is not a stopping time nor is is the first time such that at the next time one is in a state x. An important fact is that the Markov property extends to stopping times. Just as we have σ-fields F_{n} associated with constant times, we do have a σ field F_{τ} associated to any stopping time. This is the information we have when we observe the chain upto time τ. Formally

F_{τ} = {A : A ∈ F^{∞} and A ∩ {τ ≤ n} ∈ F_{n} for each n}

One can check from the definition that τ is F_{τ} measurable and so is X_{τ} on the set τ < ∞. If τ is the time of first visit to y then τ is a stopping time and the event that the chain visits a state z before visiting y is F_{τ} measurable.

Lemma 3.2. At any stopping time τ the Markov Property holds in the sense that the conditional distribution of X_{τ+1}, ⋯, X_{τ+n}, ⋯ conditioned on F_{τ} is the same as the original chain starting from the state x = X_{τ} on the set τ < ∞. In other words

P_{x}{X_{τ+1} ∈ A_{1}, ⋯, X_{τ+n} ∈ A_{n}|F_{τ}}
= ∫_{A_{1}} ⋯ ∫_{A_{n}} π(X_{τ}, dx_{1}) ⋯ π(x_{n−1}, dx_{n})

a.e. on {τ < ∞}.

Proof: Let A ∈ F_{τ} be given with A ⊂ {τ < ∞}. Then

P_{x}{A ∩ {X_{τ+1} ∈ A_{1}, ⋯, X_{τ+n} ∈ A_{n}}}

= ∑_{k} P_{x}{A ∩ {τ = k} ∩ {X_{k+1} ∈ A_{1}, ⋯, X_{k+n} ∈ A_{n}}}

= ∑_{k} ∫_{A∩{τ=k}} ∫_{A_{1}} ⋯ ∫_{A_{n}} π(X_{k}, dx_{k+1}) ⋯ π(x_{k+n−1}, dx_{k+n}) dP_{x}

= ∫_{A} ∫_{A_{1}} ⋯ ∫_{A_{n}} π(X_{τ}, dx_{1}) ⋯ π(x_{n−1}, dx_{n}) dP_{x}

We have used the fact that if A ∈ F_{τ} then A ∩ {τ = k} ∈ F_{k} for every k ≥ 0.

Remark: If X_{τ} = y a.e. with respect to P_{x} on the set τ < ∞, then at time τ, when it is finite, the process starts afresh with no memory of the past and will have conditionally the same probabilities in the future as P_{y}. At such times the process renews itself and these times are called renewal times.

Countable State Space

From the point of view of analysis a particularly simple situation is when the state space X is a countable set. It can be taken as the integers {x : x ≥ 1}. Many applications fall in this category and an understanding of what happens in this situation will tell us what to expect in general.

The one step transition probability is a matrix π(x, y) with nonnegative entries such that ∑_{y} π(x, y) = 1 for each x. Such matrices are called stochastic matrices. The n step transition matrix is just the n-th power of the matrix defined inductively by

π^{(n+1)}(x, y) = ∑_{z} π^{(n)}(x, z)π(z, y)

To be consistent one defines π^{(0)}(x, y) = δ_{x,y} which is 1 if x = y and 0 otherwise. The problem is to analyse the behaviour for large n of π^{(n)}(x, y). A state x is said to communicate with a state y if π^{(n)}(x, y) > 0 for some n ≥ 1. We will assume for simplicity that every state communicates with every other state. Such Markov Chains are called irreducible. Let us first limit ourselves to the study of irreducible chains. Given an irreducible Markov chain with transition probabilities π(x, y) we define f_{n}(x) as the probability of returning to x for the first time at the n-th step assuming that the chain starts from the state x.. Using the convention that P_{x} refers to the measure on sequences for the chain starting from x and {X_{j}} are the successive positions of the chain

f_{n}(x) = P_{x}{X_{j} ≠ x for 1 ≤ j ≤ n − 1 and X_{n} = x}
= ∑_{y_{1}≠x ; {··· \atop y_{n−1}≠x}} π(x, y_{1}) π(y_{1}, y_{2}) ⋯ π(y_{n−1}, x)

Since f_{n}(x) are probailities of disjoint events ∑_{n} f_{n}(x) ≤ 1. The state x is called transient if ∑_{n} f_{n}(x) < 1 and recurrent if ∑_{n} f_{n}(x) = 1. The recurrent case is divided into two situations. If we denote by τ_{x} = inf{n ≥ 1 : X_{n} = x}, the time of first visit to x, then recurrence is P_{x}{τ_{x} < ∞} = 1. A recurrent state x is called positive recurrent if

E^{P_{x}}{τ_{x}} = ∑_{n≥1} n f_{n}(x) < ∞

and null recurrent if

E^{P_{x}}{τ_{x}} = ∑_{n≥1} n f_{n}(x) = ∞

Lemma 3.3. If for a (not necessarily irreducible) chain starting from x, the probability of ever visiting y is positive then so is the probability of visiting y before returning to x.

Proof: Assume that for the chain starting from x the probability of visiting y before returning to x is zero. But when it returns to x it starts afresh and so will not visit y until it returns again. This reasoning can be repeated and so the chain will have to visit x infinitely often before visiting y. But this will use up all the time and so it cannot visit y at all.

Lemma 3.4. For an irreducible chain all states x are of the same type.

Proof: Let x be recurrent and y be given. Since the chain is irreducible, for some k, π^{(k)}(x, y) > 0. By the previous lemma, for the chain starting from x, there is a positive probability of visiting y before returning to x. After each successive return to x, the chain starts afresh and there is a fixed positive probability of visiting y before the next return to x. Since there are infinitely many returns to x, y will be visited infinitely many times as well. Or y is also a recurrent state.

We now prove that if x is positive recurrent then so is y. We saw already that the probability p = P_{x}{τ_{y} < τ_{x}} of visiting y before returning to x is positive. Clearly

E^{P_{x}}{τ_{x}} ≥ P_{x}{τ_{y} < τ_{x}} E^{P_{y}}{τ_{x}}

and therefore

E^{P_{y}}\{τ_{x}\} ≤ \frac{1}{p}E^{P_{x}}\{τ_{x}\} < ∞

On the other hand we can write

E^{P_{x}}{τ_{y} } ≤ ∫_{τ_{y}<τ_{x}} τ_{x} dP_{x} + ∫_{τ_{x}<τ_{y}} τ_{y} dP_{x}
= ∫_{τ_{y}<τ_{x}} τ_{x} dP_{x} + ∫_{τ_{x}<τ_{y}}{τ_{x} + E^{P_{x}}{τ_{y} }} dP_{x}
= ∫_{τ_{y}<τ_{x}} τ_{x} dP_{x} + ∫_{τ_{x}<τ_{y}} τ_{x} dP_{x} + (1 − p) E^{P_{x}}{τ_{y} }
= ∫ τ_{x} dP_{x} + (1 − p) E^{P_{x}}{τ_{y} }

by the renewal property at the stopping time τ_{x}. Therefore

E^{P_{x}}\{τ_{y} \} ≤ \frac{1}{p}E^{P_{x}}\{τ_{x} \}

We also have

E^{P_{y}}\{τ_{y} \} ≤ E^{P_{y}}\{τ_{x} \} + E^{P_{x}}\{τ_{y} \} ≤ \frac{2}{p}E^{P_{x}}\{τ_{x} \}

proving that y is positive recurrent.

Transient Case: We have the following theorem regarding transience.

Theorem 3.5. An irreducible chain is transient if and only if

G(x, y) = ∑_{n=0}^{∞} π^{(n)}(x, y) < ∞ for all x, y

Moreover for any two states x and y,

G(x, y) = f(x, y)G(y, y)

and

G(x, x) = \frac{1}{1 − f(x, x)}

where f(x, y) = P_{x}{τ_{y} < ∞}.

Proof: Each time the chain returns to x there is a probability 1−f(x, x) of never returning. The number of returns has then the geometric distribution

P_{x}{ exactly n returns to x} = (1 − f(x, x))f(x, x)^{n}

and the expected number of returns is given by

∑_{k=1}^{∞} π^{(k)}(x, x) = \frac{f(x, x)}{1 − f(x, x)}

The left hand side comes from the calculation

E^{P_{x}} ∑_{k=1}^{∞} χ_{{x}}(X_{k}) = ∑_{k=1}^{∞} π^{(k)}(x, x)

and the right hand side from the calculation of the mean of a Geometric distribution. Since we count the visit at time 0 as a visit to x we add 1 to both sides to get our formula. If we want to calculate the expected number of visits to y when we start from x, first we have to get to y and the probability of that is f(x, y). Then by the renewal property it is exactly the same as the expected number of visits to y starting from y, including the visit at time 0 and that equals G(y, y).

Before we study the recurrent behavior we need the notion of periodicity. For each state x let us define D_{x} = {n : π^{(n)}(x, x) > 0} to be the set of times at which a return to x is possible if one starts from x. We define d_{x} to be the greatest common divisor of D_{x}.

Lemma 3.6. For any irreducible chain d_{x} = d for all x ∈ X and for each x, D_{x} contains all sufficiently large multiples of d.

Proof: Let us define

D_{x,y} = {n : π^{(n)}(x, y) > 0}

so that D_{x} = D_{x,x}. By the Chapman-Kolmogorov equations

π^{(m+n)}(x, y) ≥ π^{(m)}(x, z)π^{(n)}(z, y)

for every z, so that if m ∈ D_{x,z} and n ∈ D_{z,y}, then m + n ∈ D_{x,y}. In particular if m, n ∈ D_{x} it follows that m + n ∈ D_{x}. Since any pair of states communicate with each other, given x, y ∈ X, there are positive integers n_{1} and n_{2} such that n_{1} ∈ D_{x,y} and n_{2} ∈ D_{y,x}. This implies that with the choice of ℓ = n_{1} + n_{2}, n + ℓ ∈ D_{x} whenever n ∈ D_{y}; similarly n + ℓ ∈ D_{y} whenever n ∈ D_{x}. Since ℓ itself belongs to both D_{x} and D_{y} both d_{x} and d_{y} divide ℓ. Suppose n ∈ D_{x}. Then n + ℓ ∈ D_{y} and therefore d_{y} divides n + ℓ. Since d_{y} divides ℓ, d_{y} must divide n. Since this is true for every n ∈ D_{x} and d_{x} is the greatest common divisor of D_{x}, d_{y} must divide d_{x}. Similarly d_{x} must divide d_{y}. Hence d_{x} = d_{y}. We now complete the proof of the lemma. Let d be the greatest common divisor of D_{x}. Then it is the greatest common divisor of a finite subset n_{1}, n_{2}, ⋯, n_{q} of D_{x} and there will exist integers a_{1}, a_{2}, ⋯, a_{q} such that

a_{1}n_{1} + a_{2}n_{2} + ⋯ + a_{q}n_{q} = d

Some of the a’s will be positive and others negative. Seperating them out, and remembering that all the n_{i} are divisible by d, we find two integers md, (m + 1)d such that they both belong to D_{x}. If now n = kd with k > m^{2} we can write k = ℓm + r with a large ℓ ≥ m and the remainder r is less than m.

kd = (ℓm+r)d = ℓmd+r(m+1)d−rmd = (ℓ−r)md+r(m+1)d ∈ D_{x} since ℓ ≥ m > r.

Remark: For an irreducible chain the common value d is called the period of the chain and an irreducible chain with period d = 1 is called aperiodic.

The simplest example of a periodic chain is one with 2 states and the chain shuttles back and forth between the two. π(x, y) = 1 if x ≠ y and 0 if x = y. A simple calculation yields π^{(n)}(x, x) = 1 if n is even and 0 otherwise. There is oscillatory behavior in n that persists. The main theorem for irreducible, aperiodic, recurrent chains is the following.

Theorem 3.7. Let π(x, y) be the one step transition probability for a recurrent aperiodic Markov chain and let π^{(n)}(x, y) be the n-step transition probabilities. If the chain is null recurrent then

lim_{n→∞} π^{(n)}(x, y) = 0 for all x, y

If the chain is positive recuurrent then of course E^{P_{x}}{τ_{x}} = m(x) < ∞ for all x, and in that case

lim_{n→∞} π^{(n)}(x, y) = q(y) = \frac{1}{m(y)}

exist for all x and y is independent of the starting point x and ∑_{y} q(y) = 1.

The proof is based on

Theorem 3.8. (Renewal Theorem.) Let {f_{n} : n ≥ 1} be a sequence of nonnegative numbers such that

∑_{n} f_{n} = 1,  ∑_{n} n f_{n} = m ≤ ∞

and the greatest common divisor of {n : f_{n} > 0} is 1. Suppose that {p_{n} : n ≥ 0} are defined by p_{0} = 1 and recursively

p_{n} = ∑_{j=1}^{n} f_{j} p_{n−j}  (3.3)

Then

lim_{n→∞} p_{n} = \frac{1}{m}

where if m = ∞ the right hand side is taken as 0.

The proof is based on several steps.

Step 1. We have inductively p_{n} ≤ 1. Let a = lim sup_{n→∞} p_{n}. We can choose a subsequence n_{k} such that p_{n_{k}} → a. We can assume without loss of generality that p_{n_{k}+j} → q_{j} as k → ∞ for all positive and negative integers j as well. Of course the limit q_{0} for j = 0 is a. In relation 3.3 we can pass to the limit along the subsequence and use the dominated convergence theorem to obtain

q_{n} = ∑_{j=1}^{∞} f_{j} q_{n−j}  (3.4)

valid for −∞ < n < ∞. In particular

q_{0} = ∑_{j=1}^{∞} f_{j} q_{−j}  (3.5)

Step 2: Because a = lim sup p_{n} we can conclude that q_{j} ≤ a for all j. If we denote by S = {n : f_{n} > 0} then q_{−k} = a for k ∈ S. We can then deduce from equation 3.4 that q_{−k} = a for k = k_{1} + k_{2} with k_{1}, k_{2} ∈ S. By repeating the same reasoning q_{−k} = a for k = k_{1} + k_{2} + ⋯ + k_{ℓ}. By lemma 3.6 because the greatest common factor of the integers in S is 1, there is a k_{0} such that for k ≥ k_{0},we have q_{−k} = a. We now apply the relation 3.4 again to conclude that q_{j} = a for all positve as well as negative j.

Step 3: If we add up equation 3.3 for n = 1, ⋯, N we get

p_{1} + p_{2} + ⋯ + p_{N} = (f_{1} + f_{2} + ⋯ + f_{N}) + (f_{1} + f_{2} + ⋯ + f_{N−1})p_{1}
+ ⋯ + (f_{1} + f_{2} + ⋯ + f_{N−k})p_{k} + ⋯ + f_{1}p_{N−1}

If we denote by T_{j} = ∑_{i=j}^{∞} f_{i}, we have T_{1} = 1 and ∑_{j=0}^{∞} T_{j} = m. We can now rewrite

∑_{j=1}^{N} T_{j}p_{N−j+1} = ∑_{j=1}^{N} f_{j}

Step 4: Because p_{N−j} → a for every j along the subsequence N = n_{k}, if ∑_{j} T_{j} = m < ∞, we can deduce from the dominated convergence theorem that m a = 1 and we conclude that

lim sup_{n→∞} p_{n} = \frac{1}{m}

If ∑_{j} T_{j} = ∞, by Fatou’s Lemma a = 0. Exactly the same argument applies to liminf and we conclude that

lim inf_{n→∞} p_{n} = \frac{1}{m}

This concludes the proof of the theorem.

We now turn to the proof of our main theorem.

If we take a fixed x ∈ X and consider f_{n} = P_{x}{τ_{x} = n}, then f_{n} and p_{n} = π^{(n)}(x, x) are related by (1) and m = E^{P_{x}}{τ_{x}}. In order to apply the renewal theorem we need to establish that the greatest common divisor of S = {n : f_{n} > 0} is 1. In general if f_{n} > 0 so is p_{n}. So the greatest common divisor of S is always larger than that of {n : p_{n} > 0}. That does not help us because the greatest common divisor of {n : p_{n} > 0} is 1. On the other hand if f_{n} = 0 unless n = k d for some k, the relation 3.3 can be used inductively to conclude that the same is true of p_{n}. Hence both sets have the same greatest common divisor. We can now conclude that

lim_{n→∞} π^{(n)}(x, x) = q(x) = \frac{1}{m(x)}

On the other hand if f_{n}(x, y) = P_{x}{τ_{y} = n}, then

π^{(n)}(x, y) = ∑_{k=1}^{n} f_{k}(x, y) π^{(n−k)}(y, y)

and recurrence implies ∑_{k+1}^{∞} f_{k}(x, y) = 1 for all x and y. Therefore

lim_{n→∞} π^{(n)}(x, y) = q(y) = \frac{1}{m(y)}

and is independent of x, the starting point. In order to complete the proof we have to establish that

Q = ∑_{y} q(y) = 1

It is clear by Fatou’s lemma that

∑_{y} q(y) = Q ≤ 1

By letting n → ∞ in the Chapman-Kolmogorov equation

π^{(n+1)}(x, y) = ∑_{z} π^{n}(x, z)π(z, y)

and using Fatou’s lemma we get

q(y) ≥ ∑_{z} π(z, y)q(z)

Summing with repect to y we obtain

Q ≥ ∑_{z,y} π(z, y)q(z) = Q

and equality holds in this relation. Therefore

q(y) = ∑_{z} π(z, y)q(z)

for every y or q(·) is an invariant measure. By iteration

q(y) = ∑_{z} π^{n}(z, y)q(z)

and if we let n → ∞ again an application of the bounded convergence theorem yields

q(y) = Q q(y)

implying Q = 1 and we are done.