The multiarmed-bandit problem is often taken as a basic model for the trade-off between the exploration and utilization required for efficient optimization under uncertainty. In this article, we study the situation in which the unknown performance of a new bandit is to be evaluated and compared with that of a known one over a finite horizon. We assume that the bandits represent random variables with distributions from the one-parameter exponential family. When the objective is to maximize the Bayes expected sum of outcomes over a finite horizon, it is shown that optimal policies tend to simple limits when the length of the horizon is large.

We derive single integral representations for the exact distribution of the sum of independent generalized Pareto random variables. The integrands involve the incomplete and complementary incomplete gamma functions. Applications to insurance and catastrophe bonds are described.

We consider a mixture of one exponential distribution and one gamma distribution with increasing failure rate. For the right choice of parameters, it is shown that its failure rate has an upsidedown bathtub shape failure rate. We also consider a mixture of a family of exponentials and a family of gamma distributions and obtain a similar result.

The discrete-time G/GI/∞ queue model is explored. Jobs arrive to an infinite-server queuing system following an arbitrary input process X; job sizes are general independent and identically distributed random variables. The system's output process Y (of job departures) and queue process N (tracking the number of jobs present in the system) are analyzed. Various statistics of the stochastic maps X↦ Y and X↦ N are explicitly obtained, including means, variances, autocovariances, cross-covariances, and multidimensional probability generating functions. In the case of stationary inputs, we further compute the spectral densities of the stochastic maps, characterize the fixed points (in the L2 sense) of the input–output map, precisely determine when the output and queue processes display either short-ranged or long-ranged temporal dependencies, and prove a decomposition result regarding the intrinsic L2 structure of general stationary G/GI/∞ systems.

We analyze a construction for optimal nested group-testing procedures, and show that, when individuals are independently positive with probability p, the expected number of tests per positive individual, F*(p), has, as p→0, the asymptotic behavior

$$F^{\ast}(p) = \log_2 {1\over p} + \log_2 \log 2 + 2 + f\left(\log_2 {1\over p} + \log_2 \log 2\right) + O(p),$$

$$f(z) = 4\times 2^{-2^{1-\{z\}}} - \{z\} - 1,$$

In this paper, we study a two-queue polling model with zero switchover times and k-limited service (serve at most ki customers during one visit period to queue i, i=1, 2) in each queue. The arrival processes at the two queues are Poisson, and the service times are exponentially distributed. By increasing the arrival intensities until one of the queues becomes critically loaded, we derive exact heavy-traffic limits for the joint queue-length distribution using a singular-perturbation technique. It turns out that the number of customers in the stable queue has the same distribution as the number of customers in a vacation system with Erlang-k2 distributed vacations. The queue-length distribution of the critically loaded queue, after applying an appropriate scaling, is exponentially distributed. Finally, we show that the two queue-length processes are independent in heavy traffic.

We propose a particular class of transition probability matrices for discrete-time Markov chains with a closed form to compute the stationary distribution. The stochastic monotonicity properties of this class are established. We give algorithms to construct monotone, bounding matrices belonging to the proposed class for the variability orders. The accuracy of bounds with respect to the underlying matrix structure is discussed through an example.

This paper studies a multiple-recipe predictive maintenance problem with M/G/1 queueing effects. The server degrades according to a discrete-time Markov chain and we assume that the controller knows both the machine status and the current number of jobs in the system. The controller's objective is to minimize total discounted costs or long-run average costs which include preventative and corrective maintenance costs, holdings costs, and possibly production costs. An optimal policy determines both when to perform maintenance and which type of job to process. Since the policy takes into account the machine's degradation status, such control decisions are known as predictive maintenance policies. In the single-recipe case, we prove that the optimal policy is monotone in the machine status, but not in the number of jobs in the system. A similar monotonicity result holds in the two-recipe case. Finally, we provide computational results indicating that significant savings can be realized when implementing a predictive maintenance policies instead of a traditional job-based threshold policy for preventive maintenances.

The existence of a moment function satisfying a drift function condition is well known to guarantee non-explosiveness of the associated minimal Markov process (cf. [1,9]), under standard technical conditions. Surprisingly, the reverse is true as well for a countable space Markov process. We prove this result by showing that recurrence of an associated jump process, that we call the α-jump process, is equivalent to non-explosiveness. Non-explosiveness corresponds in a natural way to the validity of the Kolmogorov integral relation for the function identically equal to 1. In particular, we show that positive recurrence of the α-jump chain implies that all bounded functions satisfy the Kolmogorov integral relation. We present a drift function criterion characterizing positive recurrence of this α-jump chain.

Suppose that to a drift function V there corresponds another drift function W, which is a moment with respect to V. Via a transformation argument, the above relations hold for the transformed process with respect to V. Transferring the results back to the original process, allows to characterize the V-bounded functions that satisfy the Kolmogorov forward equation.

Analysis of the initial transient problem of Monte Carlo steady-state simulation motivates the following question for Markov chains: when does there exist a deterministic Tsuch that P{X(T) = y|(0) = x} = ®(y), where ρ is the stationary distribution of X? We show that this can essentially never happen for a continuous-time Markov chain; in discrete time, such processes are i.i.d. provided the transition matrix is diagonalizable.

We consider a class of stochastic models for systems subject to random regulation. We derive expressions for the distribution of the intervals between regulating instants and for the transient and equilibrium properties of the process. Some of these are evaluated explicitly for some models of interest.

This paper focuses on a discrete-time risk model in which both insurance risk and financial risk are taken into account. We study the asymptotic behavior of the ruin probability and the tail probability of the aggregate risk amount. Precise asymptotic formulas are derived under weak moment conditions of involved risks. The main novelty of our results lies in the quantification of the impact of the financial risk.

In this paper, we consider exponentiated location-scale model and obtain several ordering results between extreme order statistics in various senses. Under majorization type partial order-based conditions, the comparisons are established according to the usual stochastic order, hazard rate order and reversed hazard rate order. Multiple-outlier models are considered. When the number of components are equal, the results are obtained based on the ageing faster order in terms of the hazard rate and likelihood ratio orders. For unequal number of components, we develop comparisons according to the usual stochastic order, hazard rate order, and likelihood ratio order. Numerical examples are considered to illustrate the results.

We consider a stochastic process with an embedded point process in a stationary and ergodic context. Under a “lack of anticipation” assumption for the evolution of the process vis-à-vis the point process, a new better (worse) than used expectation property for the point process, and a monotonicity assumption for the behavior of the process between points, inequalities between event and time averages are obtained. Sample path monotonicity between points is not required (as is the case with existing approaches) and can be replaced with a simple monotonicity requirement for the expected value of the process between points. Inequalities between conditional event and time averages are also examined via a novel argument involving a conditional version of the Palm inversion formula.

In this paper, we give the definition of tree-indexed Markov chains in random environment with countable state space, and then study the realization of Markov chain indexed by a tree in random environment. Finally, we prove the strong law of large numbers and Shannon–McMillan theorem for Markov chains indexed by a Cayley tree in a Markovian environment with countable state space.

To find zeros or locate maximum values of a regression function with noisy measurements, a commonly used algorithm is the RM or KW procedure. In various applications, the dimensionality of the problems involved might be quite large. As a result, enormous memory space and extensive computation time may be needed. Motivated by the recent progress in stochastic approximation methods for decentralized and distributed computing, a parallel stochastic approximation algorithm is developed in this paper. The essence is to take advantage of state-space decompositions, and to exploit the opportunities provided by parallel processing and asynchronous communication. In lieu of utilizing a single processor as in the classical cases, a number of parallel processors are employed to solve the underlying problem in a cooperative way. First, the large dimensional vector is partitioned into a number of subvectors with relatively small dimension, then each of the subvectors is assigned to one of the processors. The processors compute and communicate in an asynchronous manner and at random times. Under rather weak conditions, the global convergence of the parallel algorithm is obtained via the methods of randomly varying truncations.

We derive a simple recursion to compute moments of arbitrary order of the duration of busy periods of MX/G/1/n systems starting with an arbitrary number of customers in the system.

We studied several group testing models with and without processing times. The objective was to choose an optimal group size for pooled screening of a contaminated population so as to collect a prespecified number of good items from it with minimum testing expenditures. The tested groups that were found to be contaminated were used as a new sampling population in later stages of the procedures. Since testing may be time-consuming, we also considered deadlines to be met for the testing process. We derived algorithms and exact results for the underlying distributions, enabling us to find optimal procedures. Several numerical examples are given.

Some monotonicity properties and bounds on expectations of certain functions of minimal repair times and interminimal repair times are obtained under assumptions on the aging characteristics of the equipment under maintenance.

Let [NA(t), t≥0] and [NB(t), t≥0] be two counting processes with independent interarrivals Ai and Bi, respectively, i ∈ + ≡ [1,2,… ], and let [Xi, i ∈ +] and [Yi, i ∈ +] be two sequences of independent non-negative random variables. Consider the two compound processes [U(t), t ≥ 0] and [V(t), t ≥ 0] defined as U(t) = Xi, t ≥ 0, and V(t) = Yi, t ≥ 0. Then it is well known (and not hard to verify) that if the Ai's are independent of the Xi's, and if the Bi's are independent of the yi's, and if Ai ≥st Bi and Xi ≤st Yi, for all i ∈ +, then [U(t), t ≥ 0] ≤st [V(t), t ≥ 0]. In this paper we provide conditions under which this stochastic order relation holds even if Ai ≤st Bi and/or Xi ≥st Yi for all i ∈ +. In fact, we derive the stochastic order relationships among processes that are much more general than the cumulative ones described above. For example, it is not necessarily assumed that the Ai's are independent of the Xi's or that the Bi's are independent of the Yi's. Examples that illustrate the applications of the theory are included.