Background: Cerebral small vessel disease (CSVD) is associated with stroke, cognitive decline, and dementia. In this study, we examined how SVD is longitudinally related to cognitive performance in transient ischemic attack (TIA) patients compared to controls. Methods: We rated CSVD at baseline on MRI in TIA patients (n=197) and controls (n=113) for microbleeds (CMB), lacunes, white matter hyperintensities (WMH), and perivascular spaces (EPVS). Neuropsychological testing was administered across 5 years using the following assessments: BVMT, RAVLT, TMTA, TMTB, WAIS-R. Results: Periventricular WMH≥2 yielded slower performance on TMTB across all timepoints (adjusted difference 20.3 seconds, 95%CI [8.4,32.2]), as did deep WMH≥2 (20.1 sec, 95%CI [7.6,32.6]). Basal ganglia EPVS>20 performed slower on TMTA (10.1 sec, 95%CI [4.7,15.5]) and TMTB (21.2 sec, 95%CI [3.4,39.1]). Centrum semiovale EPVS>20 performed slower on TMTB (27.2 sec, 95%CI [10.6,43.8]) and worse on WAIS-R at 5-years (-18.6, 95%CI [-35.0,-2.2]). Lacunes≥3 performed slower on TMTA across all timepoints (4.0 sec, 95%CI [0.1,7.9]). Total CSVD≥2 performed slower on TMTA (3.7 sec, 95%CI [0.4,7.0]) and TMTB (13.9 sec, 95%CI [2.9,24.9]) across 5 years. When stratifying results, associations were generally found in TIA, not controls. Conclusions: Findings demonstrate that CSVD is associated with poorer cognitive performance longitudinally, and is more pronounced in TIA compared to control.