There are surprisingly few experimental studies directly comparing the cognition of primate species representing distinct phylogenetic groupings, specialized foraging ecologies, or unique social structures. Although researchers have focused on the role of foraging and social ecology in predicting cognition, they have examined social and foraging strategies in a nuanced fashion that would permit an understanding of how specific aspects of a species’ natural environment might sculpt the evolution of specific forms of cognition. In the absence of such studies, and a clear consensus as to whether cognition should best be viewed as domain-general or domain-specific suites of abilities, it is challenging to draw conclusions as to (1) cognitive differences between primate families or (2) selection pressures responsible for shaping differences. We conclude, based on paltry but accumulating evidence, that there is little utility in postulating separate physical and social domains. In addition, we see little evidence that group-living species are cognitively advantaged compared to primates that exhibit other social structures. Lastly, we advocate for greater attention to reproductive and parental strategies and individual differences in ontogenetic experiences that may color species-level comparisons.