Given the full shift over a countable state space on a countable amenable group, we develop its thermodynamic formalism. First, we introduce the concept of pressure and, using tiling techniques, prove its existence and further properties, such as an infimum rule. Next, we extend the definitions of different notions of Gibbs measures and prove their existence and equivalence, given some regularity and normalization criteria on the potential. Finally, we provide a family of potentials that nontrivially satisfy the conditions for having this equivalence and a nonempty range of inverse temperatures where uniqueness holds.

Motivated by various applications in queueing systems, this work is devoted to continuous-time Markov chains with countable state spaces that involve both fast-time scale and slow-time scale with the aim of approximating the time-varying queueing systems by their quasistationary counterparts. Under smoothness conditions on the generators, asymptotic expansions of probability vectors and transition probability matrices are constructed. Uniform error bounds are obtained, and then sequences of occupation measures and their functionals are examined. Mean square error estimates of a sequence of occupation measures are obtained; a scaled sequence of functionals of occupation measures is shown to converge to a Gaussian process with zero mean. The representation of the variance of the limit process is also explicitly given. The results obtained are then applied to treat Mt/Mt/1 queues and Markov-modulated fluid buffer models.

This paper establishes the existence of an optimal stationary strategy in a leavable Markov decision process with countable state space and undiscounted total reward criterion.

Besides assumptions of boundedness and continuity, an assumption is imposed on the model which demands the continuity of the mean recurrence times on a subset of the stationary strategies, the so-called ‘good strategies'. For practical applications it is important that this assumption is implied by an assumption about the cost structure and the transition probabilities. In the last part we point out that our results in general cannot be deduced from related works on bias-optimality by Dekker and Hordijk, Wijngaard or Mann.