Healthy diet has been linked to better age-related functioning, but evidence on the relationship of diet quality in late midlife and measures of physical capability in later life is limited. Research on potential sex differences in this relationship is scarce. The aim was to investigate the prospective association between overall diet quality, as assessed by the Healthy Eating Index-2015 (HEI-2015) at 60–64 years and measures of walking speed 7 years later, among men and women from the Insight 46, a neuroscience sub-study of the Medical Research Council National Survey of Health and Development. Diet was assessed at 60–64 years using 5-d food diaries, from which total HEI-2015 was calculated. At 69–71 years, walking speed was estimated during four 10-m walks at self-selected pace, using inertial measurement units. Multivariable linear regression models with sex as a modifier, controlling for age, follow-up, lifestyle, health/social variables and physical performance, were used. The final sample consists of 164 women and 167 men (n 331). Women had higher HEI-2015 and slower walking speed than men. A 10-point increase in HEI-2015 was associated with faster walking speed among women (B 0·024, 95 % CI 0·006, 0·043), but not men. The association remained significant in the multivariable model (B 0·021, 95 % CI 0·003, 0·040). In women, higher diet quality in late midlife is associated with faster walking speed. A healthy diet in late midlife is likely to contribute towards better age-related physical capability, and sex differences are likely to affect this relationship.