In the hinterland of the desert, valuable archives of paleoenvironmental evolution are scarce. Calcareous root tubes (CRTs) have a strong potential for reconstructing paleoenvironmental conditions. It is still unclear, however, whether chemical elements in the CRTs can provide insights into paleoenvironmental conditions. In this study, the major- and trace-element composition of 32 CRT samples from the Alashan Desert were analyzed by X-ray fluorescence spectrometry. Results showed that the elemental composition and content change were controlled by the parent material and climatic conditions at the time of CRT formation. Ca, Mg, and Sr were significantly affected by climate, whereas the enrichment of P is likely related to the growth of plants. Higher (lower) Mg/Ca and Sr/Ca ratios corresponded to higher (lower) effective moisture and a higher (lower) Mg/Sr ratio indicated a higher (lower) temperature during the middle Holocene (8–5 cal ka BP). The reconstruction results for effective moisture were consistent with those in the Asian monsoon margin of northwestern China, which were caused by higher monsoon precipitation and lower evaporation. Therefore, chemical elements in the CRTs can reflect changes in paleo-effective moisture and paleotemperature at a millennial resolution in this area.