The habenula, which in humans is a small nuclear complex within the epithalamus, plays an essential role in regulating the intensity of reward-seeking and adversity-avoiding behavior in all vertebrate ancestors by regulating the activity of ascending midbrain monoaminergic tracts. In lampreys, considered to possess a brain comparable to humans’ earliest evolutionary vertebrate ancestor, the activity of the lateral habenula is controlled by a subset of glutamatergic neurons of the animal’s pallidum (habenula-projecting globus pallidus) that inhibit reward-seeking behavior when this conduct is not successful enough. The pathophysiological roles of the habenula and habenula-projecting globus pallidus in humans have hardly been studied, which is probably due to insufficient resolution of common neuroimaging techniques. Their dysregulation may, however, play an essential role in the pathogenesis of mood and stress disorders and addiction.

Taking the evolutionary development of the forebrain as a starting point, the authors developed a biological framework for the subcortical regulation of human emotional behaviour which may offer an explanation for the pathogenesis of the principle symptoms of mental disorders. Appetitive-searching (reward-seeking) and distress-avoiding (misery-fleeing) behaviour are essential for all free-moving animals to stay alive and to have offspring. Even the oldest ocean-dwelling animal creatures, living about 560 million years ago and human ancestors, must therefore have been capable of generating these behaviours. Our earliest vertebrate ancestors, with a brain comparable with the modern lamprey, had a sophisticated extrapyramidal system generating and controlling all motions as well as a circuit including the habenula for the evaluation of the benefits of their actions. Almost the complete endbrain of the first land animals with a brain comparable with that of amphibians became assimilated into the human amygdaloid and hippocampal complex, whereas only a small part of the dorsal pallium and striatum developed into the ventral extrapyramidal circuits and the later insular cortex. The entire neocortex covering the hemispheres is of recent evolutionary origin, appearing first in early mammals. During the entire evolution of vertebrates, the habenular system was well conserved and maintained its function in regulating the intensity of reward-seeking (pleasure-related) and misery-fleeing (happiness-related) behaviour. The authors propose that the same is true in humans. Symptomatology of human mental disorders can be considered to result from maladaptation within a similar amygdalo/hippocampal–habenular–mesencephalic–ventral striatal system.

The epithalamus is a major subdivision of the diencephalon constituted by the habenular nuclei and pineal complex. Structural asymmetries in this region are widespread amongst vertebrates and involve differences in size, neuronal organisation, neurochemistry and connectivity. In species that possess a photoreceptive parapineal organ, this structure projects asymmetrically to the left habenula, and in teleosts it is also situated on the left side of the brain. Asymmetries in size between the left and right sides of the habenula are often associated with asymmetries in neuronal organisation, although these two types of asymmetry follow different evolutionary courses. While the former is more conspicuous in fishes (with the exception of teleosts), asymmetries in neuronal organisation are more robust in amphibia and reptiles. Connectivity of the parapineal organ with the left habenula is not always coupled with asymmetries in habenular size and/or neuronal organisation suggesting that, at least in some species, assignment of parapineal and habenular asymmetries may be independent events.

The evolutionary origins of epithalamic structures are uncertain but asymmetry in this region is likely to have existed at the origin of the vertebrate, perhaps even the chordate, lineage. In at least some extant vertebrate species, epithalamic asymmetries are established early in development, suggesting a genetic regulation of asymmetry. In some cases, epigenetic factors such as hormones also influence the development of sexually dimorphic habenular asymmetries. Although the genetic and developmental mechanisms by which neuroanatomical asymmetries are established remain obscure, some clues regarding the mechanisms underlying laterality decisions have recently come from studies in zebrafish. The Nodal signalling pathway regulates laterality by biasing an otherwise stochastic laterality decision to the left side of the epithalamus. This genetic mechanism ensures a consistency of epithalamic laterality within the population. Between species, the laterality of asymmetry is variable and a clear evolutionary picture is missing. We propose that epithalamic structural asymmetries per se and not the laterality of these asymmetries are important for the behaviour of individuals within a species. A consistency of the laterality within a population may play a role in social behaviours between individuals of the species.