Path integrals for continuous-time Markov chains

Phil Pollett

Abstract: Suppose that we are using a random process (X(t), 0<t<T) to model a stochastic system, and that there is a cost f(x) associated with it being in state x. By integrating f(X(t)) from t=0 to T, we obtain the total cost C over the lifetime T of the process. These "path integrals" have been used extensively in managing stochastic systems. For example, f(x) might represent the cost of an epidemic when there are x individuals infected. It could be the amount of nutrient consumed when a population is in state x, or, it could be the storage cost associated with an inventory consisting of x items. This paper is concerned with evaluating the distribution and the expected value of C when the underlying process is a Markov chain in continuous time. The literature is abound with results for particular classes of process and particular choices of f, usually linear functions of the state. We will describe a method which assumes only that f is non-negative, and which characterizes both the distribution and the expected value of C as extremal solutions of systems of linear equations. Of particular interest in biological applications is the case where there is a single absorbing state, corresponding to population extinction, where we are usually interested in evaluating the cost of the process up to the time of extinction. We will illustrate our results with reference to three important Markovian models: the pure-birth process, the birth-death process, and the linear birth-death and catastrophe process.

Acknowledgement: This work is supported by the ARC Centre of Excellence for Mathematics and Statistics of Complex Systems

The speaker:

Last modified: 15th September 2004