Our spatial analysis indicates that in 2000 over one third of the rural population in developing countries was located on less favored agricultural land and areas, which are constrained by biophysical conditions or poor market access. We examine whether these spatial distributions of rural population in 2000 influence subsequent changes in the rate of poverty from 2000 to 2012 in 83 developing countries. We find no evidence of a direct impact on changes in poverty, but there is a significant indirect impact via the elasticity of poverty reduction with respect to growth. If climate change leads to more people concentrated in these areas, or an increase in unfavorable agricultural regions, then the poverty-reducing impact of overall per capita income growth could be further weakened. Reducing poverty will require targeting rural populations in less favored lands and remote areas and encouraging out-migration.