Cognitive decline is a public health problem for the world’s ageing population. This study was to evaluate the relationships between serum Fe, blood Pb, Cd, Hg, Se and Mn and cognitive decline in elderly Americans. Data of this cross-sectional study were extracted from the National Health and Nutritional Examination Survey (NHANES 2011–2014). Cognitive performance was measured by the Consortium to Establish a Registry for Alzheimer’s Disease (CERAD), Animal Fluency and Digit Symbol Substitution Test (DSST) tests. Weighted univariable and multivariate logistic regression analyses were used to assess the associations between six trace elements and low cognitive performance. Subgroup analyses based on diabetes and hypertension history were further assessed the associations. A total of 2002 adults over 60 years old were included. After adjusting covariates, elevated serum Fe levels were associated with the decreased risk of low cognitive performance, especially in the elderly without diabetes history and with hypertension history. High blood Cd levels were associated with the high odds of low cognitive performance in old adults with diabetes and hypertension history. Elevated blood Mn levels were connected with low cognitive performance in old hypertensive people. High blood Pb levels were related to the high odds of low cognitive performance, especially in the elderly without diabetes and hypertension history. High blood Se levels were linked to the decreased risk of low cognitive performance in all the elderly. Appropriate Fe, Se supplementation and Fe-, Se-rich foods intake, while reducing exposure to Pb, Cd and Mn may be beneficial for cognitive function in the elderly.