QUASISTATIONARITY IN CONTINUOUS-TIME MARKOV CHAINS

WITH POSITIVE DRIFT

by

Pauline Coolen-Schrijner, The University of Durham

Andrew Hart, The University of Queensland Phil Pollett, The University of Queensland

MARKOV CHAINS WITH POSITIVE DRIFT

Let (X_{t}, t ≥ 0) be a continuous time Markov chain over a denumerable state space S. For simplicity take S = {0, 1, … }.

Let Q = (q_{ij}, i, j ∈ S) be the matrix of transition rates (q-matrix), assumed to be stable and conservative, so that q_{ij} (≥ 0), for j ≠ i, represents the transition rate from state i to state j and q_{ii} = −q_{i}, where q_{i} = ∑_{j≠i} q_{ij} (< ∞) represents the transition rate out of state i.

Let P (t) = (p_{ij}(t), i, j ∈ S), where

p_{ij}(t) = Pr(X_{t} = j|X_{0} = i),

and suppose that p_{ij}′(0+) = q_{ij}. It will not be necessary to assume that the transition function P is determined uniquely by Q.

We assume only that P is irreducible, that is, p_{ij}(t) > 0 for all i, j ∈ S, and transient, that is,

∫_{0}^{∞} p_{ij}(t) dt < ∞, i, j ∈ S.

LIMITING CONDITIONAL DISTRIBUTIONS

We shall be concerned with the existence of a limiting conditional distribution (LCD) for the process:

lim_{t→∞} Pr(X_{t} = j|X_{0} = i, X_{t+s} = 0 for some s > 0).

The conditional probability can be evaluated as follows:

Pr(X_{t} = j|X_{0} = i, X_{t+s} = 0 for some s > 0)

= \frac{p_{ij}(t)b_{j}}{∑_{k∈S} p_{ik}(t)b_{k}} ,

where

b_{i} = Pr(X_{t} = 0 for some t > 0|X_{0} = i).

Note that b_{0} = 1 and (by irreducibility and transience) 0 < b_{i} < 1, i ∈ S.

LCDs for ABSORBING CHAINS

Compare the above definition with the widely studied LCDs for absorbing chains:

Append to S an absorbing state (or, more generally, an absorbing set) θ such that p_{iθ}(t) > 0 for some (and then all) i ∈ S and t > 0.

In this context we study the limit (as t → ∞) of

\begin{aligned}Pr(X_{t} = j|X_{0} & = i, X_{t} ∈ S) \\ & = \frac{p_{ij}(t)}{∑_{k∈S} p_{ik}(t)} , i, j ∈ S, \end{aligned}

or, more generally (when a_{j}, the probability of absorption starting in j, is less than 1),

Pr(X_{t} = j|X_{0} = i, X_{t} ∈ S, X_{t+s} = θ for some s)

= \frac{p_{ij}(t)a_{j}}{∑_{k∈S} p_{ik}(t)a_{k}} , i, j ∈ S.

HITTING PROBABILITIES

Let b_{i}(t) = Pr(X_{s} = 0 for some s > t|X_{0} = i), so that b_{i}(0) = b_{i}. This can be evaluated as

b_{i}(t) = \frac{∫_{t}^{∞} p_{i0}(u)du}{∫_{0}^{∞} p_{00} (u)du} , i ∈ S, t ≥ 0.

Claim. b_{i}(t) = ∑_{j∈S} p_{ij}(t)b_{j}, i ∈ S, t ≥ 0. It follows that b = (b_{i}, i ∈ S) is a subinvariant vector for P; in particular,

_{j∈S}^{∑} p_{ij}(t)b_{j} = b_{i} − \frac{1}{u_{0}} ∫_{0}^{t} p_{i0}(u)du (< b_{i} for t > 0),

where u_{0} = ∫_{0}^{∞} p_{00} (u)du (< ∞). Claim. b is a subinvariant vector for Q; in fact, b is ‘almost invariant’:

_{j∈S}^{∑} q_{ij}b_{j} = \begin{cases}0, & if i ≥ 1, \\ −\frac{1}{u_{0}}, & if i = 0. \end{cases}

CONSTRUCTION OF THE DUAL

Define P̄ (t) = (p̄_{ij}(t), i, j ∈ S) and Q̄ = (q̄_{ij}, i, j ∈ S) by

p̄_{ij}(t) = \frac{p_{ij}(t)b_{j}}{b_{i}} , i, j ∈ S,

and

q̄_{ij} = \frac{q_{ij}b_{j}}{b_{i}} , i, j ∈ S.

Clearly P̄ is a (standard) transition function and Q̄ is a q-matrix such that p̄_{ij}′(0+) = q̄_{ij}. However, P is dishonest with

_{j∈S}^{∑} p̄_{ij}(t) = \frac{b_{i}(t)}{b_{i}} (< 1 for t > 0),

and Q̄ is non-conservative with

_{j∈S}^{∑} q̄_{ij} = \begin{cases}0, & if i ≥ 1, \\ −\frac{1}{u_{0}}, & if i = 0. \end{cases}

(Recall that b_{0} = 1.)

CONSTRUCTION OF THE DUAL Append to S an absorbing state θ and extend the definition of P̄ and Q̄ to S̄ = S ∪ {θ} as follows:

p̄_{θj}(t) = δ_{θj}, j ∈ S̄,

p̄_{iθ}(t) = 1 − \frac{b_{i}(t)}{b_{i}} , i ∈ S,

and q̄_{θj} = 0, j ∈ S̄,

q̄_{iθ} = \frac{δ_{i0}}{u_{0}}, i ∈ S.

In this way Q̄ will be conservative and P̄ will be honest. P̄ is a Q̄-transition function since, for i ∈ S,

lim_{t↓0} \frac{p̄_{iθ}(t)}{t} = lim_{t↓0} \frac{1}{t} (1 − \frac{b_{i}(t)}{b_{i}} )

= \frac{1}{b_{i}u_{0}} lim_{t↓0} \frac{1}{t} ∫_{0}^{t} p_{i0}(u)du

= \frac{1}{b_{i}u_{0}} lim_{t↓0} p_{i0}(t) = \frac{δ_{i0}}{b_{i}u_{0}} = q̄_{iθ}.

INTERPRETATION OF THE DUAL

Let L be the last exit time of state 0, that is,

L(ω) = sup {t ≥ 0 : X_{t}(ω) = 0},

and define (X̄_{t}, t ≥ 0) by

X̄_{t}(ω) = \begin{cases}X_{t}(ω), & if t < L(ω), \\ θ, & if t ≥ L(ω); \end{cases}

X̄_{t} is X_{t} killed at time L. Then^{∗}, since b_{i} = Pr(L > 0|X_{0} = i), X̄_{t} is Markovian with transition function P̄, and, if X_{0} = i, its initial distribution is given by

Pr(X̄_{0} = j) = \begin{cases}b_{i}, & if j = i, \\ 1 − b_{i}, & if j = θ, \\ 0, & otherwise . \end{cases}

Notice that X̄_{t} is absorbed with probability 1:

ā_{i} := lim_{t→∞} p̄_{iθ}(t) = lim_{t→∞} (1 − \frac{b_{i}(t)}{b_{i}} ) = 1, i ∈ S.

∗Meyer, P.A., Smithe, R.T., and Walsh, J.B. (1971) Birth and death of Markov processes, Proc. 6th Berkeley Symposium, III, pp. 295–305, University of California Press.

EXISTENCE OF LCDs

By the definition of P̄, we see immediately that, for every i, j ∈ S,

Pr(X_{t} = j|X_{0} = i, X_{t+s} = 0 for some s > 0)

= \frac{p_{ij}(t)b_{j}}{∑_{k∈S} p_{ik}(t)b_{k}} = \frac{p̄_{ij}(t)}{∑_{k∈S} p̄_{ik}(t)} .

Thus, the existence or otherwise of an LCD can be established via the dual transition function using results for absorbing chains.

Here is one such result for absorbing chains:

Theorem. Suppose S is λ-positive recurrent and let m̄ = (m̄_{j}, j ∈ S) be the (essentially unique) λ-invariant measure for P̄. Then, if ∑_{j∈S} m̄_{j} < ∞,

lim_{t→∞} \frac{p̄_{ij}(t)}{∑_{k∈S} p̄_{ik}(t)} = \frac{m̄_{j}}{∑_{k∈S} m̄_{k}} , i, j ∈ S.

λ-CLASSIFICATION

P̄ and P have the same λ-classification. The decay parameter:

λ̄ = lim_{t→∞} −\frac{1}{t} log p̄_{ij}(t) = lim_{t→∞} −\frac{1}{t} log p_{ij}(t) = λ.

Note that λ (≥ 0) is the same for each i, j ∈ S and p_{ij}(t) = O(e^{−λt}), p̄_{ij}(t) = O(e^{−λt}). Geometric ergodicity: λ̄ > 0 ⇔ λ > 0. λ-transience/recurrence: for i, j ∈ S,

∫_{0}^{∞} e^{λt}p̄_{ij}(t)dt \begin{cases}< ∞ \\ = ∞ \end{cases} ⇔ ∫_{0}^{∞} e^{λt}p_{ij}(t)dt \begin{cases}< ∞ \\ = ∞ \end{cases}

λ-positive/null recurrence: for i, j ∈ S,

lim_{t→∞} e^{λt}p̄_{ij}(t) \begin{cases}> 0 \\ = 0 \end{cases} ⇔ lim_{t→∞} e^{λt}p_{ij}(t) \begin{cases}> 0 \\ = 0 \end{cases}

λ-INVARIANT MEASURES FOR P

Definition. A collection of positive numbers m = (m_{j}, j ∈ S) is called a λ-subinvariant measure for P if, for all t ≥ 0,

_{i∈S}^{∑} m_{i}p_{ij}(t) ≤ e^{−λt}m_{j}, j ∈ S,

and λ-invariant if, for all t ≥ 0,

_{i∈S}^{∑} m_{i}p_{ij}(t) = e^{−λt}m_{j}, j ∈ S.

Now, for any given collection m = (m_{j}, j ∈ S), define m̄ = (m̄_{j}, j ∈ S) by m̄_{j} = m_{j}b_{j}.

Claim. m is λ-(sub)invariant for P if and only if m̄ is λ-(sub)invariant for P̄.

Claim. If m is a λ-subinvariant measure for P, then ∑_{i∈S} m_{i}b_{i} < ∞ if λ > 0. If m is λ-invariant for P, then ∑_{i∈S} m_{i}b_{i} < ∞ only if λ > 0.

Thus, the usual premise, ∑_{i∈S} m̄_{i} < ∞, that m̄ be a finite λ-invariant measure is always satisfied in the present context of irreducible transient chains.

λ-INVARIANT MEASURES FOR Q

Definition. A collection of positive numbers m = (m_{j}, j ∈ S) is called a λ-invariant measure for Q if, for all t ≥ 0,

_{i∈S}^{∑} m_{i}q_{ij} = −λm_{j}, j ∈ S.

Again we have that m is λ-invariant for Q if and only if m̄ is λ-invariant for Q̄.

Now suppose that Q is regular, so that P, now being the unique Q-transition function, is honest.

We can then apply to Q̄ the following result for absorbing chains (with a transient class S and absorbing state θ):

Theorem. If m̄ = (m̄_{j}, j ∈ S) is a finite λinvariant measure for Q̄, then it is λ-invariant for P̄ if and only if

λ _{i∈S}^{∑} m̄_{i} = _{i∈S}^{∑} m̄_{i}q̄_{iθ}.

LCDs WHEN Q IS REGULAR

Theorem. Let λ > 0 and suppose that m = (m_{j}, j ∈ S) is a λ-invariant measure for Q. Then, ∑_{j∈S} m_{j}b_{j} < ∞, and m is λ-invariant for P if (and only if)

_{j∈S}^{∑} m_{j}b_{j} = \frac{m_{0}}{λu_{0}},

whence if S is λ-positive the LCD exists:

lim_{t→∞} Pr(X_{t} = j|X_{0} = i, X_{t+s} = 0 for some s > 0)

= \frac{m_{j}b_{j}}{∑_{k∈S} m_{k}b_{k}}, i, j ∈ S.

Other examples:

1. Kesten’s bounded jump conditions^{∗}.

2. Asymptotic remoteness^{†}.

3. Results for special processes: birth-death processes, branching processes, catastrophe processes.

∗Kesten, H. (1995) A ratio limit theorem for sub Markov chains on {1, 2, 3 … } with bounded jumps, Adv. Appl. Probab. 27, 652–691. †Ferrari, P.A., Kesten, H., Martínez, S. and Picco, P. (1995) Existence of quasi-stationary distributions. A renewal dynamic approach, Ann. Probab. 23, 501–521.

BIRTH-DEATH PROCESSES

Consider an irreducible birth-death process on S = {0, 1, … }. Its q-matrix is given by

q_{ij} = \{ \begin{matrix}λ_{i}, & if j = i + 1, \\ −(λ_{i} + μ_{i}), & if j = i, \\ μ_{i}, & if j = i − 1, \\ 0, & otherwise , \end{matrix}

where λ_{i} > 0 (i ≥ 0), μ_{i} > 0 (i ≥ 1) and μ_{0} = 0.

Define series A and C by

A = ∞ ; ∑ ; i=0 \frac{1}{λ_{i}π_{i}} and C = ∞ ; ∑ ; i=0 \frac{1}{λ_{i}π_{i}} i ; ∑ ; j=0 π_{j} ,

where π_{0} = 1 and, for i ≥ 1,

π_{i} = i ; ∏ ; k=1 \frac{λ_{k−1}}{μ_{k}} .

Assume that C = ∞ (Q regular) and that A < ∞ (S transient). Then, the hitting probabilities are given by

b_{i} = \frac{1}{A} ∞ ; ∑ ; k=i \frac{1}{λ_{k}π_{k}} .

Define polynomials (ϕ_{i}(·), i ∈ S), where ϕ_{i} : R → R, by ϕ_{0}(x) = 0, λ_{0}ϕ_{1}(x) = λ_{0} + μ_{0} − x and, for i ≥ 1, λ_{i}ϕ_{i+1} (x) = (λ_{i} + μ_{i} − x)ϕ_{i}(x) − μ_{i}ϕ_{i−1}(x), and let m_{i} = π_{i}ϕ_{i}(λ), i ∈ S, where λ is the decay parameter of S. Then, m is the essentially unique λ-invariant measure.

The dual is an absorbing birth-death process; we get (in an obvious notation):

λ̄_{i} = λ_{i}b_{i+1} /b_{i}, i ≥ 0, λ_{θ} = 0,

μ̄_{i} = μ_{i}b_{i−1}/b_{i}, i ≥ 1, μ̄_{0} (= q̄_{0θ}) = λ_{0}(1 − b_{1}),

π̄_{i} = b_{i}^{2}π_{i}, i ≥ 0, π̄_{θ} = 1,

ϕ̄_{i}(x) = ϕ_{i}(x)/b_{i}, i ≥ 0, ϕ_{θ}(x) = 0,
m̄_{i} = π̄_{i}ϕ̄_{i}(λ) = b_{i}π_{i}ϕ_{i}(λ), i ≥ 0.

A direct application of van Doorn’s^{∗} result for absorbing birth-death processes shows that the LCD exists if and only if λ > 0:

lim_{t→∞} Pr(X_{t} = j|X_{0} = i, X_{t+s} = 0 for some s > 0)

= \frac{b_{i}π_{i}ϕ_{i}(λ)}{∑_{k∈S} b_{k}π_{k}ϕ_{k}(λ)} , i, j ∈ S.

∗Van Doorn, E.A (1991) Quasi-stationary distributions and convergence to quasi-stationarity of birth-death processes, Adv. Appl. Probab. 23, 683–700.

AN EXAMPLE

Let f : [0, ∞) → [0, ∞), V > 0 and 0 < α < 1, and define λ_{0} = q_{0},

λ_{n} = αq_{n}, μ_{n} = (1 − α)q_{n}, n ≥ 1,

where q_{n} = V f(n/V).

This defines a density-dependent birth-death process with constant jump probabilities: think of V as “area” and n/V as “population density”. The idea is that, as V gets large, the density process x_{t} = X_{t}/V becomes “more deterministic”:

\frac{dx}{dt} = αf(x) − (1 − α)f(x).

Clearly the process is irreducible if f(n) > 0 for all n ≥ 0, and then transient if (and only if) α > 1/2, in which case

b_{i} = (\frac{1 − α}{α} )^{i} , i ≥ 0.

Two choices for f:

f(x) = \frac{(π − x)^{2}}{(π + x)^{3}} , f(x) = \frac{(π − x)^{2}(2π − x)^{2}}{(π + x)^{5}} .

Each has f(x) → 0 and x → ∞.