Cognitive abilities are heritable and influenced by socioeconomic status (SES). It is critical to understand the association between SES and cognition beyond genetic propensity to inform potential benefits of SES-based interventions and to determine if such associations vary across (i) cognitive domains, (ii) facets of SES, and/or (iii) genetic propensity for different aspects of cognition. We examined the contributions of neighborhood socioeconomic advantage, family income, and polygenic scores (PGS) for domains of cognition (i.e., general cognitive ability, executive function, learning and memory, fluid reasoning) in a sample of children (ages 9–10; n = 5549) most genetically similar to reference populations from Europe. With some variability across cognitive outcomes, family income and PGS were independently significantly associated with cognitive performance. Within-sibling analyses revealed that cognitive PGS associations were predominantly driven by between-family effects suggestive of non-direct genetic mechanisms. These findings provide evidence that SES and genetic propensity to cognition have unique associations with cognitive performance in middle childhood. These results underscore the importance of environmental factors and genetic influences in the development of cognitive abilities and caution against overinterpreting associations with PGS of cognitive and educational outcomes as predominantly direct genetic effects.