Identifying Markov chains with a given invariant measure

by

Phil Pollett

Department of Mathematics
The University of Queensland


PRELIMINARIES

State-space: tex2html_wrap_inline1164

Transition functions: A set of real-valued functions tex2html_wrap_inline1166, tex2html_wrap_inline1168 defined on tex2html_wrap_inline1170 is called a process (or transition function) if

tex2html_wrap_inline1172
tex2html_wrap_inline1174,
tex2html_wrap_inline1172
tex2html_wrap_inline1178, and
tex2html_wrap_inline1172
tex2html_wrap_inline1182.

It is called standard

tex2html_wrap_inline1172
tex2html_wrap_inline1186

and honest if

tex2html_wrap_inline1172
tex2html_wrap_inline1190,

for some (and then for all) t>0.

THE Q-MATRIX

For a standard process P the right-hand derivative tex2html_wrap_inline1196 exists and defines a q-matrix tex2html_wrap_inline1200 whose entries satisfy

tex2html_wrap_inline1172
tex2html_wrap_inline1204, tex2html_wrap_inline1206, and
tex2html_wrap_inline1172
tex2html_wrap_inline1210.

We set tex2html_wrap_inline1212, tex2html_wrap_inline1214.

Suppose that Q is given. Assume that Q is stable, that is tex2html_wrap_inline1220 for all i in S. A standard process P will then be called a Q-process if its q-matrix is Q.

THE KOLMOGOROV DIFFERENTIAL EQUATIONS

For simplicity, we shall assume that Q is conservative, that is

displaymath1236

for all tex2html_wrap_inline1214. Under this condition, every Q-process P satisfies the backward equations,

displaymath1244

but might not satisfy the forward equations,

displaymath1246

STATIONARY DISTRIBUTIONS

A collection of positive numbers tex2html_wrap_inline1248 is a stationary distribution if tex2html_wrap_inline1250 and

  equation215

Recipe: Find a collection of strictly positive numbers tex2html_wrap_inline1252 such that

displaymath1254

Such an m is called an invariant measure for Q. If tex2html_wrap_inline1260, then we set

displaymath1262

and hope that tex2html_wrap_inline1264 satisfies (2).

BIRTH-DEATH PROCESSES

Transition rates:

tex2html_wrap_inline1266 ( tex2html_wrap_inline1268 - birth rates)

tex2html_wrap_inline1270 ( tex2html_wrap_inline1272 - death rates) ( tex2html_wrap_inline1274 )

Solve: tex2html_wrap_inline1276, tex2html_wrap_inline1278, that is,

displaymath1280

and, for tex2html_wrap_inline1282,

displaymath1284

Solution: tex2html_wrap_inline1286 and

displaymath1288

MILLER'S EXAMPLE

Rates: fix r>0 and set

tex2html_wrap_inline1292,

tex2html_wrap_inline1294.

Solution: tex2html_wrap_inline1286 and

displaymath1288

So,

tex2html_wrap_inline1300, where tex2html_wrap_inline1302,

and hence if r>1,

displaymath1306

Simulation (minimal process)

[Download a postscript version of the above image]

WHAT IS GOING WRONG?

Rates:

tex2html_wrap_inline1308,

tex2html_wrap_inline1310.

The relative proportion of births to deaths is r and so, if r>1, the ``process'' is clearly transient.

In fact, the ``process'' is explosive: Q is not regular. R.G. Millergif showed that Q needs to be regular for the recipe to work.

THE MOTIVATING QUESTION

If Q is regular, then there exists uniquely a Q-process, namely the minimal process: the minimal solution tex2html_wrap_inline1324, tex2html_wrap_inline1168 to BE.

If Q is not regular there are infinitely many Q-processes, infinitely many of which are honest.

Question: Suppose that there exists a collection of strictly positive numbers tex2html_wrap_inline1248 such that

displaymath1336

Does  tex2html_wrap_inline1264 admit an interpretation as a stationary distribution for any of these processes?

Simulation (exit process)

[Download a postscript version of the above image]

Simulation (entrance process)

[Download a postscript version of the above image]

AN INVARIANCE RESULT

Let tex2html_wrap_inline1340 be a collection of strictly positive numbers. We call m a subinvariant measure for Q if

displaymath1346

and an invariant measure for Q if

displaymath1254

It is called an invariant measure for P if

displaymath1354

Theorem: Let P be an arbitrary Q-process. If m is invariant for P, then m is subinvariant for Q, and invariant for Q if and only if P satisfies the forward equations FE over S.

Corollary: If m is invariant for the minimal process F, then m is invariant for Q.

A CONSTRUCTION PROBLEM

Suppose that Q is a stable and conservative q-matrix, and that m is subinvariant for Q.

Problem 1: Does there exist a Q-process for which m is invariant?

Problem 2: Does there exist an honest Q-process for which m is invariant?

Problem 3: When such a Q-process exists, is it unique?

Problem 4: In the case of non-uniqueness, can one identify all Q-processes (or perhaps all honest Q-processes) for which m is invariant?

THE RESOLVENT

Let P be a transition function. If we write

displaymath1410

for the Laplace transform of tex2html_wrap_inline1412, then tex2html_wrap_inline1414 enjoys the following (analogous) properties:

tex2html_wrap_inline1172
tex2html_wrap_inline1418,
tex2html_wrap_inline1172
tex2html_wrap_inline1422, and
tex2html_wrap_inline1172
tex2html_wrap_inline1426.

tex2html_wrap_inline1428 is called the resolvent of P. Indeed, if tex2html_wrap_inline1428 is a given resolvent, in that it satisfies these properties, then there exists a standard (!) process P with  tex2html_wrap_inline1428 as its resolventgif.

IDENTIFYING Q-PROCESSES USING RESOLVENTS

Now, if one is given a stable and conservative q-matrix Q, together with a resolvent tex2html_wrap_inline1428 which satisfies the backward equations,

displaymath1444

then tex2html_wrap_inline1428 determines a standard Q-process. In particular, as tex2html_wrap_inline1450,

tex2html_wrap_inline1172
tex2html_wrap_inline1454, and
tex2html_wrap_inline1172
tex2html_wrap_inline1458.

One can also use the resolvent to determine whether or not the Q-process is honest, for this happens when and only when

displaymath1462

IDENTIFYING INVARIANT MEASURES USING RESOLVENTS

Theorem: Let P be an arbitrary process and let tex2html_wrap_inline1428 be its resolvent. Then, m is invariant for P if and only if it is invariant for  tex2html_wrap_inline1428, that is,

displaymath1474

if and only if

displaymath1476

THE EXISTENCE OF A Q-PROCESS WITH A GIVEN INVARIANT MEASURE

Theorem: Let Q be a stable and conservative q-matrix, and suppose that m is a subinvariant measure for Q. Let tex2html_wrap_inline1486 be the resolvent of the minimal Q-process and define tex2html_wrap_inline1490 and tex2html_wrap_inline1492 by

tex2html_wrap_inline1494,

and

tex2html_wrap_inline1496.

Then, if d=0, m is invariant for the minimal Q-process. Otherwise, if

tex2html_wrap_inline1504,

for all tex2html_wrap_inline1506, there exists a Q-process P for which m is invariant.

THE EXISTENCE OF A Q-PROCESS WITH A GIVEN INVARIANT MEASURE

Theorem continued: The resolvent of one such process is given by

  equation399

and this process is honest if and only if

displaymath1514

for all tex2html_wrap_inline1506. A sufficient condition for there to exist an honest Q-process for which m is invariant is that m satisfies

displaymath1524

for all tex2html_wrap_inline1506.

Corollary: If m is a subinvariant probability distribution for Q, then there exists an honest Q-process with stationary distribution m. The resolvent of one such process is given by (4).

THE SINGLE-EXIT CASE

Suppose that Q is a single-exit q-matrix, that is, the space of bounded, non-negative vectors tex2html_wrap_inline1540 which satisfy

displaymath1542

has dimension 1. (The minimal process has only one available ``escape route'' to infinity). Then, the condition

displaymath1544

is necessary for the existence of a Q-process for which the specified measure is invariant; the Q-process is then determined uniquely by

displaymath1550

NON-UNIQUENESS

Consider a pure-birth process with strictly positive birth rates tex2html_wrap_inline1552, but imagine that we have two distinct sets of birth rates, tex2html_wrap_inline1554 and tex2html_wrap_inline1556, which satisfy

displaymath1558

Let tex2html_wrap_inline1560 and define tex2html_wrap_inline1562 by

displaymath1564

for r=0,1 and tex2html_wrap_inline1568. The measure tex2html_wrap_inline1570, given by

displaymath1572

is subinvariant for Q.

Double water slide

NON-UNIQUENESS

The resolvents of two distinct Q-processes for which m is invariant are given by

displaymath1580

and

displaymath1582

Interpretation: The first process chooses between (0,0) and (1,0) with equal probability as the starting point following an explosion, no matter which was the most recently traversed path, and the second process traverses alternate paths following successive explosions.

THE REVERSIBLE CASE

Suppose that Q is symmetrically reversible with respect to m, that is, tex2html_wrap_inline1592, tex2html_wrap_inline1594. Then, tex2html_wrap_inline1596, and so we arrive at the following corollary due to Hou and Chengif.

Corollary: If Q is reversible with respect to m, then there exists uniquely a Q-function P for which m is invariant if and only if tex2html_wrap_inline1608, for all tex2html_wrap_inline1506. It is honest and its resolvent is given by

displaymath1612

Moreover, P is reversible with respect to m in that tex2html_wrap_inline1618 (or, equivalently, tex2html_wrap_inline1620 tex2html_wrap_inline1622 ).

BIRTH-DEATH PROCESSES

Suppose that the birth rates tex2html_wrap_inline1624 and death rates tex2html_wrap_inline1626 are strictly positive. The q-matrix Q is then regular if and only if

  equation509

Proposition: Let tex2html_wrap_inline1340 be the essentially unique invariant measure for Q.

(i) m is invariant for the minimal Q-process if and only if (6) holds.

(ii) When (6) fails, there exists uniquely a Q-process P for which m is invariant if and only if m is finite, in which case P is the unique, honest Q-process which satisfies FE; P is positive recurrent and its stationary distribution is obtained by normalizing m.

tex2html_wrap_inline1658-INVARIANCE

Suppose that tex2html_wrap_inline1660, where 0 is an absorbing state and C is irreducible (for F). Let tex2html_wrap_inline1668. A collection tex2html_wrap_inline1670 of strictly positive numbers is called a tex2html_wrap_inline1672-subinvariant measure for Q if

displaymath1676

and a tex2html_wrap_inline1672-invariant measure for Q if

displaymath1682

It is called a tex2html_wrap_inline1672-invariant measure for P, where P is any transition function, if

displaymath1690

Proposition: A probability distribution tex2html_wrap_inline1692 is a tex2html_wrap_inline1672-invariant measure for some tex2html_wrap_inline1696 if and only if it is a quasistationary distribution: for tex2html_wrap_inline1698,

displaymath1700

tex2html_wrap_inline1658-INVARIANCE FOR F

Theorem: If m is tex2html_wrap_inline1672-invariant for P, then m is tex2html_wrap_inline1672-subinvariant for Q, and tex2html_wrap_inline1672-invariant for Q if and only if P satisfies the forward equations over C. For example, if m is tex2html_wrap_inline1672-invariant for the minimal process, then it is tex2html_wrap_inline1672-invariant for Q.

Theorem: If m is tex2html_wrap_inline1672-invariant for Q, then it is tex2html_wrap_inline1672-invariant for F if and only if the equations

displaymath1742

have no non-trivial solution for some (and then all) tex2html_wrap_inline1744.

Theorem: If m is a finite tex2html_wrap_inline1672-invariant measure for Q, then

  equation572

where tex2html_wrap_inline1752, and m is tex2html_wrap_inline1672-invariant for F if and only if equality holds in (8).

Q-PROCESSES WITH A GIVEN tex2html_wrap_inline1658-INVARIANT MEASURE

Theorem: Suppose that Q is single-exit and that m is a finite tex2html_wrap_inline1672-subinvariant measure for Q. Then, there exists a Q-process for which m is tex2html_wrap_inline1672-invariant if and only if

displaymath1776

The resolvent tex2html_wrap_inline1428 of any Q-process for which m is tex2html_wrap_inline1672-invariant must be of the form

displaymath1786

where

displaymath1788

displaymath1790

and e satisfies tex2html_wrap_inline1794.

Q-PROCESSES WITH A GIVEN tex2html_wrap_inline1658-INVARIANT MEASURE

Theorem continued: Conversely, if

displaymath1798

then all Q-processes for which m is tex2html_wrap_inline1672-invariant can be constructed in this way by varying e in the range

displaymath1808

Exactly one of these is honest; this is obtained by setting tex2html_wrap_inline1810. And, exactly one satisfies the forward equations FE tex2html_wrap_inline1812 over tex2html_wrap_inline1814 ; this is obtained by setting tex2html_wrap_inline1816.