In most social ungulate species, males are larger than females and the sexes live in separate groups outside the breeding season. It is important for our understanding of the evolution of sociality to find out why sexual segregation is so widespread not only in ungulates but also in other mammals. Sexual body size dimorphism was proposed as a central factor in the evolution of sexual segregation in ungulates. We tested three hypotheses put forward to explain sexual segregation: the predation-risk, the forage-selection, and the activity budget hypothesis. We included in our analyses ungulate species ranging from non-dimorphic to extremely dimorphic in body size. We observed oryx, zebra, bighorn sheep and ibex in the field and relied on literature data for 31 additional species. The predation-risk hypothesis predicts that females will use relatively predator-safe habitats, while males are predicted to use habitats with higher predation risk but better food quality. Out of 24 studies on different species of ungulates, females and their offspring chose poorer quality but safer habitat in only eight cases. The forage-selection hypothesis predicts that females would select habitat based on food quality, while males should prefer high forage biomass. In fact, females selected higher quality food in only six out of 18 studies where males and females segregated, in eight studies there was no difference in forage quality and in four studies males were in better quality habitat. The activity budget hypothesis predicts that with increasing dimorphism in body size males and females will increasingly differ in the time spent in different activities. Differences in activity budgets would make it difficult for males and females to stay in mixed-sex groups due to increased costs of synchrony to maintain group cohesion. The predictions of the activity budget hypothesis were confirmed in most cases (22 out of 23 studies). The heavier males were compared to females, the more time females spent foraging compared to males. The bigger the dimorphism in body mass, the more males spent time walking compared to females. Lactating females spent more time foraging than did non-lactating females or males. Whether species were mainly bulk or intermediate feeders did not affect sexual differences in time spent foraging. We conclude that sexual differences in activity budgets are most likely driving sexual segregation and that sexual differences in predation risk or forage selection are additive factors.