Many mammalian populations conform spatially and temporally to Bergmann's rule. This ecogeographic pattern is driven by selection for larger body masses by cooler temperatures and smaller ones by warming temperatures. However, it is unclear whether the response to warming or cooling temperatures is (a)symmetrical. Studies of the evolutionary record suggest that mammals evolve smaller body sizes more rapidly than larger ones, suggesting that it may be “easier” to adapt to warming climates than cooling ones. Here, we examine the potential asymmetrical response of mammals to past temperature fluctuations. We use the fossil midden record of the bushy-tailed woodrat, Neotoma cinerea, a well-studied animal that generally conforms to Bergmann's rule, to test the ability of populations to respond to warming versus cooling climate throughout its modern range in western North America over the late Quaternary. We quantified the response to temperature change, as characterized by the Greenland Ice Sheet Project 2 temperature record, using N. cinerea presence/absence and “darwins.” Our results show that populations within the modern range of N. cinerea show little difference between warming and cooling events. However, northern, peripheral populations are absent during older, cooler periods, possibly due to climate or taphonomy. Our study suggests adaptation in situ may be an underestimated response to future climate change.