We prove that every infinite-state stochastic matrix P say, that is irreducible and consists of positive-recurrrent states can be represented in the form I – P=(A – I)(B – S), where A is strictly upper-triangular, B is strictly lower-triangular, and S is diagonal. Moreover, the elements of A are expected values of random variables that we will specify, and the elements of B and S are probabilities of events that we will specify. The decomposition can be used to obtain steady-state probabilities, mean first-passage-times and the fundamental matrix.

Given a family of Markov chains with a single recurrent class, we present a potential application of Schweitzer's exact formula relating the steady-state probability and fundamental matrices of any two chains in the family. We propose a new policy iteration scheme for Markov decision processes where in contrast to policy iteration, the new criterion for selecting an action ensures the maximal one-step average cost improvement. Its computational complexity and storage requirement are analysed.

An elegant time-dependent solution for the number in the M/M/c queueing system is derived in a direct way.

For the multi-server queue with Poisson arrivals and general service times we present various approximations for the steady-state probabilities of the queue size. These approximations are computed from numerically stable recursion schemes which can be easily applied in practice. Numerical experience reveals that the approximations are very accurate with errors typically below 5%. For the delay probability the various approximations result either into the widely used Erlang delay probability or into a new approximation which improves in many cases the Erlang delay probability approximation. Also for the mean queue size we find a new approximation that turns out to be a good approximation for all values of the queueing parameters including the coefficient of variation of the service time.