We present closed-form solutions to some discounted optimal stopping problems for the running maximum of a geometric Brownian motion with payoffs switching according to the dynamics of a continuous-time Markov chain with two states. The proof is based on the reduction of the original problems to the equivalent free-boundary problems and the solution of the latter problems by means of the smooth-fit and normal-reflection conditions. We show that the optimal stopping boundaries are determined as the maximal solutions of the associated two-dimensional systems of first-order nonlinear ordinary differential equations. The obtained results are related to the valuation of real switching lookback options with fixed and floating sunk costs in the Black–Merton–Scholes model.

In their 1960 book on finite Markov chains, Kemeny and Snell established that a certain sum is invariant. The value of this sum has become known as Kemeny’s constant. Various proofs have been given over time, some more technical than others. We give here a very simple physical justification, which extends without a hitch to continuous-time Markov chains on a finite state space. For Markov chains with denumerably infinite state space, the constant may be infinite and even if it is finite, there is no guarantee that the physical argument will hold. We show that the physical interpretation does go through for the special case of a birth-and-death process with a finite value of Kemeny’s constant.

Many complex systems can be modeled via Markov jump processes. Applications include chemical reactions, population dynamics, and telecommunication networks. Rare-event estimation for such models can be difficult and is often computationally expensive, because typically many (or very long) paths of the Markov jump process need to be simulated in order to observe the rare event. We present a state-dependent importance sampling approach to this problem that is adaptive and uses Markov chain Monte Carlo to sample from the zero-variance importance sampling distribution. The method is applicable to a wide range of Markov jump processes and achieves high accuracy, while requiring only a small sample to obtain the importance parameters. We demonstrate its efficiency through benchmark examples in queueing theory and stochastic chemical kinetics.

In this article we investigate the minimal entropy martingale measure for continuous-time Markov chains. The conditions for absence of arbitrage and existence of the minimal entropy martingale measure are discussed. Under this measure, expressions for the transition intensities are obtained. Differential equations for the arbitrage-free price are derived.

Continuous-time Markov chains are a widely used modelling tool. Applications include DNA sequence evolution, ion channel gating behaviour, and mathematical finance. We consider the problem of calculating properties of summary statistics (e.g. mean time spent in a state, mean number of jumps between two states, and the distribution of the total number of jumps) for discretely observed continuous-time Markov chains. Three alternative methods for calculating properties of summary statistics are described and the pros and cons of the methods are discussed. The methods are based on (i) an eigenvalue decomposition of the rate matrix, (ii) the uniformization method, and (iii) integrals of matrix exponentials. In particular, we develop a framework that allows for analyses of rather general summary statistics using the uniformization method.

In this paper we introduce conditionally independent increment point processes, that is, processes that are conditionally independent inside and outside a bounded set A given N(A), the number of points in A. We show that these point processes can be characterized by means of the avoidance function of a multinomial ‘support process’, the solution of a suitably defined linear system of equations, and, finally, the infinitesimal matrix of a continuous-time Markov chain.

A simple, widely applicable method is described for determining factorial moments of N̂t, the number of occurrences in (0,t] of some event defined in terms of an underlying Markov renewal process, and asymptotic expressions for these moments as t → ∞. The factorial moment formulae combine to yield an expression for the probability generating function of N̂t, and thereby further properties of such counts. The method is developed by considering counting processes associated with events that are determined by the states at two successive renewals of a Markov renewal process, for which it both simplifies and generalises existing results. More explicit results are given in the case of an underlying continuous-time Markov chain. The method is used to provide novel, probabilistically illuminating solutions to some problems arising in the stochastic modelling of ion channels.

The conditional tail expectation in risk analysis describes the expected amount of risk that can be experienced given that a potential risk exceeds a threshold value, and provides an important measure of right-tail risk. In this paper, we study the convolution and extreme values of dependent risks that follow a multivariate phase-type distribution, and derive explicit formulae for several conditional tail expectations of the convolution and extreme values for such dependent risks. Utilizing the underlying Markovian property of these distributions, our method not only provides structural insight, but also yields some new distributional properties of multivariate phase-type distributions.

Consider a continuous-time Markov chain evolving in a random environment. We study certain forms of interaction between the process of interest and the environmental process, under which the stationary joint distribution is tractable. Moreover, we obtain necessary and sufficient conditions for a product-form stationary distribution. A number of examples that illustrate the applicability of our results in queueing and population growth models are also included.

As proposed by Irle and Gani in 2001, a process X is said to be slower in level crossing than a process Y if it takes X stochastically longer to exceed any given level than it does Y. In this paper, we extend a result of Irle (2003), relative to the level crossing ordering of uniformizable skip-free-to-the-right continuous-time Markov chains, to derive a new set of sufficient conditions for the level crossing ordering of these processes. We apply our findings to birth-death processes with and without catastrophes, and M/M/s/c systems.

We consider the problem of estimating the rate of convergence to stationarity of a continuous-time, finite-state Markov chain. This is done via an estimator of the second-largest eigenvalue of the transition matrix, which in turn is based on conventional inference in a parametric model. We obtain a limiting distribution for the eigenvalue estimator. As an example we treat an M/M/c/c queue, and show that the method allows us to estimate the time to stationarity τ within a time comparable to τ.

This work presents an estimate of the error on a cumulative reward function until the entrance time of a continuous-time Markov chain into a set, when the infinitesimal generator of this chain is perturbed. The derivation of an error bound constitutes the first part of the paper while the second part deals with an application where the time until saturation is considered for a circuit switched network which starts from an empty state and which is also subject to possible failures.

An open hierarchical (manpower) system divided into a totally ordered set of k grades is discussed. The transitions occur only from one grade to the next or to an additional (k+1)th grade representing the external environment of the system. The model used to describe the dynamics of the system is a continuous-time homogeneous Markov chain with k+1 states and infinitesimal generator R = (rij) satisfying rij = 0 if i > j or i + 1 < j ≤ k (i, j = 1,…,k+1), the transition matrix P between times 0 and 1 being P = expR. In this paper, two-wave panel data about the hierarchical system are considered and the resulting fact that, in general, the maximum-likelihood estimated transition matrix cannot be written as an exponential of an infinitesimal generator R having the form described above. The purpose of this paper is to investigate when this can be ascribed to the effect of sampling variability.

The gating mechanism of a single ion channel is usually modelled by a continuous-time Markov chain with a finite state space, partitioned into two classes termed ‘open’ and ‘closed’. It is possible to observe only which class the process is in. A burst of channel openings is defined to be a succession of open sojourns separated by closed sojourns all having duration less than t0. Let N(t) be the number of bursts commencing in (0, t]. Then are measures of the degree of temporal clustering of bursts. We develop two methods for determining the above measures. The first method uses an embedded Markov renewal process and remains valid when the underlying channel process is semi-Markov and/or brief sojourns in either the open or closed classes of state are undetected. The second method uses a ‘backward’ differential-difference equation.

The observed channel process when brief sojourns are undetected can be modelled by an embedded Markov renewal process, whose kernel is shown, by exploiting connections with bursts when all sojourns are detected, to satisfy a differential-difference equation. This permits a unified derivation of both exact and approximate expressions for the kernel, and leads to a thorough asymptotic analysis of the kernel as the length of undetected sojourns tends to zero.

Consider a system of interacting finite Markov chains in continuous time, where each subsystem is aggregated by a common partitioning of the state space. The interaction is assumed to arise from dependence of some of the transition rates for a given subsystem at a specified time on the states of the other subsystems at that time. With two subsystem classes, labelled 0 and 1, the superposition process arising from a system counts the number of subsystems in the latter class. Key structure and results from the theory of aggregated Markov processes are summarized. These are then applied also to superposition processes. In particular, we consider invariant distributions for the level m entry process, marginal and joint distributions for sojourn-times of the superposition process at its various levels, and moments and correlation functions associated with these distributions. The distributions are obtained mainly by using matrix methods, though an approach based on point process methods and conditional probability arguments is outlined. Conditions under which an interacting aggregated Markov chain is reversible are established. The ideas are illustrated with simple examples for which numerical results are obtained using Matlab. Motivation for this study has come from stochastic modelling of the behaviour of ion channels; another application is in reliability modelling.

The gating mechanism of a single ion channel is usually modelled by a continuous-time Markov chain with a finite state space. The state space is partitioned into two classes, termed ‘open’ and ‘closed’, and it is possible to observe only which class the process is in. In many experiments channel openings occur in bursts. This can be modelled by partitioning the closed states further into ‘short-lived’ and ‘long-lived’ closed states, and defining a burst of openings to be a succession of open sojourns separated by closed sojourns that are entirely within the short-lived closed states. There is also evidence that bursts of openings are themselves grouped together into clusters. This clustering of bursts can be described by the ratio of the variance Var (N(t)) to the mean [N(t)] of the number of bursts of openings commencing in (0, t]. In this paper two methods of determining Var (N(t))/[N(t)] and limt→∝ Var (N(t))/[N(t)] are developed, the first via an embedded Markov renewal process and the second via an augmented continuous-time Markov chain. The theory is illustrated by a numerical study of a molecular stochastic model of the nicotinic acetylcholine receptor. Extensions to semi-Markov models of ion channel gating and the incorporation of time interval omission are briefly discussed.

We describe a computational procedure for evaluating the quasi-stationary distributions of a continuous-time Markov chain. Our method, which is an ‘iterative version' of Arnoldi's algorithm, is appropriate for dealing with cases where the matrix of transition rates is large and sparse, but does not exhibit a banded structure which might otherwise be usefully exploited. We illustrate the method with reference to an epidemic model and we compare the computed quasi-stationary distribution with an appropriate diffusion approximation.

Continuous-time Markov chain models have been widely considered for the gating behaviour of a single ion channel. In such models the state space is usually partitioned into two classes, designated ‘open' and ‘closed', and there is ‘aggregation' in that it is possible to observe only which class the process is in at any given time. Hawkes et al. (1990) have derived an expression for the density function of the exact distribution of an observed open time in such an aggregated Markov model, where brief sojourns in either the open or the closed class are unobservable. This paper extends their result to single ion channel models based on aggregated semi-Markov processes, giving a more direct derivation which is probabilistic and exhibits clearly the combinatorial content.

We show that for a large class of one-dimensional interacting particle systems, with a finite initial configuration, any limit measure , for a sequence of times tending to infinity, must be invariant. This result is used to show that the one-dimensional biased annihilating branching process with parameter > 1/3 converges in distribution to the upper invariant measure provided its initial configuration is almost surely finite and non-null.

We consider a semi-Markov process with finite state space, partitioned into two classes termed ‘open' and ‘closed'. It is possible to observe only which class the process is in. We show that complete information concerning the aggregated process is contained in an embedded Markov renewal process, whose parameters, moments and equilibrium behaviour are determined. Such processes have found considerable application in stochastic modelling of single ion channels. In that setting there is time interval omission, i.e. brief sojourns in either class failed to be detected. Complete information on the aggregated process incorporating time interval omission is contained in a Markov renewal process, whose properties are derived, obtained from the above Markov renewal process by a further embedding. The embedded Markov renewal framework is natural, and its invariance to time interval omission leads to considerable economy in the derivation of properties of the observed process. The results are specialised to the case when the underlying process is a continuous-time Markov chain.