Evolutionary game theory is a method of analysing the evolution of phenotypes (including types of behaviour) when the fitness of a particular phenotype depends onits frequency in the population. It was first applied to pairwise contests between animals. Such contests usually have some associated asymmetry, in size, prior residence, or age or sex status; the theory predicts that the asymmetry will be used as a cue to settlethe contest, and this is found to be the case. The theory can also be applied when individuals are competing against the population as a whole, or some part of it. In such cases, the evolution of variable behaviour - so-called mixed strategies - is predicted; actual examples of this are given. Game theory can be applied to the evolution of cooperative as well as of antagonistic behaviour. An analysis of the evolution of learning leads to testable predictions about learning behaviour.

We consider a model in which players must divide a fixed amount of resource between a number of trials of an underlying contest, with the underlying contest based on the War of Attrition. We are able to find the unique ES set (a simple generalisation of the idea of an evolutionarily stable strategy) in certain circumstances: in particular we find the conditions under which the ES set may contain a pure strategy.

This paper considers a version of the Hawk–Dove game of Maynard Smith and Price (1973) in which animals compete for a sequence of food items. Actions may depend on an animal's energy reserves. Costs and transition probabilities under a given policy depend on the mean level of aggressiveness, p, of the rest of the population. We find the optimal policy for a single animal under an average cost criterion when ρ is constant over time. We then consider the whole interacting population when individual members follow the same stationary policy. It is shown that the mean aggressiveness, p, asymptotically approaches a limiting value in this population. We then consider the existence of evolutionarily stable strategies for the population. It is shown that such strategies always exist but may not be unique.

A class of Lyapunov functions is used to demonstrate that strategy stability occurs in complex randomly mating diploid populations. Strategies close to the evolutionarily stable strategy tend to fare better than more remote strategies. If convergence in mean strategy to an evolutionarily stable strategy is not possible, evolution will continue until all strategies in use lie on a unique face of the convex hull of available strategies.

The results obtained are also relevant to the haploid parthenogenetic case.

The changes in diversity of competitive strategies in a Maynard Smith population model with mixed strategies are related to the changes in population mean strategy. The effects of slight mutations in strategy frequencies, and of slight perturbations of the contest payoff rules are then investigated, and found to increase and decrease diversity respectively (to a third-order approximation). A relation among mutational effects, payoff perturbation effects and stable population diversity is suggested.

The possibility that interspecific contest tactics might be able to stabilize two species predator-prey populations is investigated using a matrix model of the population. It is found that if the interspecific tactics have no influence on intraspecific encounters, the non-trivial equilibrium population composition is necessarily unstable. The result provides some insight into a reported instability of a predator-prey population with ‘stable' strategies.

In biological populations, several strategies for engaging in intraspecific contests may be present. Investigations of stability show that the mean strategy in use in the population can prove stable but that neutrally stable and gradually changing polymorphisms in strategy are to be expected when the contest affects reproductive success.