In this paper we study discrete-time Markov decision processes with Borel state and action spaces. The criterion is to minimize average expected costs, and the costs may have neither upper nor lower bounds. We first provide two average optimality inequalities of opposing directions and give conditions for the existence of solutions to them. Then, using the two inequalities, we ensure the existence of an average optimal (deterministic) stationary policy under additional continuity-compactness assumptions. Our conditions are slightly weaker than those in the previous literature. Also, some new sufficient conditions for the existence of an average optimal stationary policy are imposed on the primitive data of the model. Moreover, our approach is slightly different from the well-known ‘optimality inequality approach’ widely used in Markov decision processes. Finally, we illustrate our results in two examples.

In this paper we present a general framework for the modelling of the process of corrective and condition-based preventive maintenance actions for complex repairable systems. A new class of models is proposed, the generalized virtual age models. On the one hand, these models generalize Kijima's virtual age models to the case where both preventive and corrective maintenances are present. On the other hand, they generalize the usual competing risks models to imperfect maintenance actions which do not renew the system. A generalized virtual age model is defined by both a sequence of effective ages which characterizes the effects of both types of maintenance according to a classical virtual age model, and a usual competing risks model which characterizes the dependency between the two types of maintenance. Several particular cases of the general model are derived.

For the directed edge preferential attachment network growth model studied by Bollobás et al. (2003) and Krapivsky and Redner (2001), we prove that the joint distribution of in-degree and out-degree has jointly regularly varying tails. Typically, the marginal tails of the in-degree distribution and the out-degree distribution have different regular variation indices and so the joint regular variation is nonstandard. Only marginal regular variation has been previously established for this distribution in the cases where the marginal tail indices are different.

In this note we point out an inherent difficulty in numerical identification of bacteria. The problem is that of uniqueness of the taxonomic structure or, in mathematical terms, the lack of statistical identifiability of finite mixtures of multivariate Bernoulli probability distributions shown here.

The arrival rate of customers to a service facility is assumed to have the form λ(t) = λ(0) — βt2 for some constant β. Diffusion approximations show that for λ(0) sufficiently close to the service rate μ, the mean queue length at time 0 is proportional to β–1/5. A dimensionless form of the diffusion equation is evaluated numerically from which queue lengths can be evaluated as a function of time for all λ(0) and β. Particular attention is given to those situations in which neither deterministic queueing theory nor equilibrium stochastic queueing theory apply.

The general coalescent process with simultaneous multiple mergers of ancestral lines was initially characterized by Möhle and Sagitov (2001) in terms of a sequence of measures defined on the finite-dimensional simplices. A more compact characterization of the general coalescent requiring a single probability measure Ξ defined on the infinite simplex Δ was suggested by Schweinsberg (2000). This paper presents a simple criterion of weak convergence to the Ξ-coalescent. In contrast to the earlier criterion of Möhle and Sagitov (2001) based on the moment conditions, the key condition here is expressed in terms of the joint distribution of the ranked offspring sizes. This criterion interprets a vector in Δ as the ranked fractions of the total population size assigned to sibling groups constituting a (rare) generation, where a merger might occur. An example of the general coalescent is developed on the basis of the Poisson–Dirichlet distribution. It suggests a simple algorithm of simulating the Kingman coalescent with occasional (simultaneous) multiple mergers.

This paper is concerned with the asymptotic theory of estimates of an unknown parameter in continuous-time Markov processes, which are described by non-linear stochastic differential equations. The maximum likelihood estimate and the minimum contrast estimate are investigated. For these estimates strong consistency and asymptotic normality are proved. The unknown parameter is assumed to take its values either in an open or in a compact set of real numbers.

A population process is considered where particles reproduce according to an age-dependent branching process, and are subjected to disasters which occur at the epochs of an independent renewal process. Each particle alive at the time of a disaster, survives it with probability p and the survival of any particle is assumed independent of the survival of any other particle. The asymptotic behavior of the mean of the process is determined and as a consequence, necessary and sufficient conditions are given for extinction.

Zipf's laws are probability distributions on the positive integers which decay algebraically. Such laws have been shown empirically to describe a large class of phenomena, including frequency of words usage, populations of cities, distributions of personal incomes, and distributions of biological genera and species, to mention only a few. In this paper we present a Dirichlet–multinomial urn model for describing the above phenomena from a stochastic point of view.

We derive the Zipf's law under certain regularity conditions; some limit theorems are also obtained for the urn model under consideration.

Stochastic comparison results for replacement policies are shown in this paper using the formalism of point processes theory. At each failure moment a repair is allowed. It is performed with a random degree of repair including (as special cases) perfect, minimal and imperfect repair models. Results for such repairable systems with schemes of planned replacements are also shown. The results are obtained by coupling methods for point processes.

Corrections are made to formulations and proofs of some theorems about convolution equivalence closure for random sum distributions. These arise because of the falsity of a much used asymptotic equivalence lemma, and they impinge on the convolution equivalence closure theorem for general infinitely divisible laws.

A well-known maintenance procedure for stochastically failing units is the age replacement policy (ARP). Under the ARP a unit is replaced on failure, or when it reaches a predetermined critical age.

In previous papers concerned with this procedure the authors searched for the optimal critical age to minimize the objective function under discussion. The purpose of this paper is to prove that the ARP itself is the optimal decision rule amongst all reasonable policies.

The paper studies the formally defined stochastic process where {tj} is a homogeneous Poisson process in Euclidean n-space En and the a.e. finite Em-valued function f(·) satisfies |f(t)| = g(t) (all |t | = t), g(t) ↓ 0 for all sufficiently large t → ∞, and with either m = 1, or m = n and f(t)/g(t) =t/t. The convergence of the sum at (*) is shown to depend on

1. (i)

2. (ii)

3. (iii)

We generalize Jackson's theorem to the non-ergodic case. Here, despite the fact that the entire Jackson network will not achieve steady state, it is still possible to determine the maximal subnetwork that does. We do so by formulating and algorithmically solving a new non-linear throughput equation. These results, together with the ergodic results and the ones for closed networks, completely characterize the large-time behavior of any Jackson network.

Let P be the transition matrix of a positive recurrent Markov chain on the integers, with invariant distribution π. If (n)P denotes the n x n ‘northwest truncation’ of P, it is known that approximations to π(j)/π(0) can be constructed from (n)P, but these are known to converge to the probability distribution itself in special cases only. We show that such convergence always occurs for three further general classes of chains, geometrically ergodic chains, stochastically monotone chains, and those dominated by stochastically monotone chains. We show that all ‘finite’ perturbations of stochastically monotone chains can be considered to be dominated by such chains, and thus the results hold for a much wider class than is first apparent. In the cases of uniformly ergodic chains, and chains dominated by irreducible stochastically monotone chains, we find practical bounds on the accuracy of the approximations.

A random graph is a collection of n points and n directed arcs: a directed arc goes equiprobably from each point to one of (n – 1) other points. m points are initially ‘infected'. We consider several schemes of epidemic process, e.g. when the infection is delivered according to arc direction. We prove that the probability of infecting all the n points with m = 1 is ∼ e/n, when n → ∞; another result is that m = o(√ n) cannot infect an essential part of the graph (having the size of O(n)). Possible applications of the models to real world phenomena are briefly discussed.

Imagine a man owning a commodity, e.g., a house, which is for sale. Offers at varying amounts are coming in every now and then. The longer he postpones selling the more he loses because of deterioration, interest losses, or the like. At each offer he must decide whether to accept it or wait for a better one. (A more picturesque example would be that of a girl scrutinizing successive suitors.)

Self-exciting point processes (SEPPs), or Hawkes processes, have found applications in a wide range of fields, such as epidemiology, seismology, neuroscience, engineering, and more recently financial econometrics and social interactions. In the traditional SEPP models, the baseline intensity is assumed to be a constant. This has restricted the application of SEPPs to situations where there is clearly a self-exciting phenomenon, but a constant baseline intensity is inappropriate. In this paper, to model point processes with varying baseline intensity, we introduce SEPP models with time-varying background intensities (SEPPVB, for short). We show that SEPPVB models are competitive with autoregressive conditional SEPP models (Engle and Russell 1998) for modeling ultra-high frequency data. We also develop asymptotic theory for maximum likelihood estimation based inference of parametric SEPP models, including SEPPVB. We illustrate applications to ultra-high frequency financial data analysis, and we compare performance with the autoregressive conditional duration models.

By considering k-out-of-n systems with independent and nonidentically distributed components, we discuss stochastic monotone properties of the residual life and the inactivity time. We then present some stochastic comparisons of two systems based on the residual life and inactivity time.

In many applications of queueing theory assumptions of either Poisson arrivals or exponential service times are made. The implicit assumption is that if the actual arrival process approximates a Poisson process and the service times are close to exponential, then the quantities of interest in the real queueing system (viz. the virtual waiting time, queue length, idle times, etc.), will approximate those of the idealized model. The continuity of the single server queue acting as functionals of the arrival and service processes is established. The proof involves an application of the theory of weak convergence of probability measures on metric spaces.