Gilbert's interactions with Coleridge and Southey at Bristol in 1795–8 can be traced through letters and publications. In the case of Wordsworth the traces of his friendship with Gilbert are all retrospective, and there is no means of pinpointing exactly when Gilbert and Wordsworth met. Wordsworth came to Bristol in August 1795, staying with his friends John and Azariah Pinney at their father's town house for about five weeks. He was introduced to Coleridge, Southey, and Cottle at that time so it is likely that Gilbert, being part of that circle, would have appeared at some stage. Another possible connection is through the Pinneys, given that they were – like Gilbert – members of a family whose wealth derived from West Indian sugar estates.

After Wordsworth moved with Dorothy to Racedown, a property owned by John Pinney senior in Dorset, he continued to visit Bristol at intervals and, during his collaboration with Coleridge when he moved to Alfoxden in the Quantock Hills, the two poets looked to Bristol as the city where their publisher was based. Wordsworth's connection with Bristol continued up to the time he wrote out ‘Tintern Abbey’ there after his Wye Valley tour in July 1798. Later in life he wrote fondly to Cottle, ‘I often think with lively remembrance of the days I passed at Bristol—not setting the least value on those passed under the roof of your good Father and Mother’ (WL, V: p. 10).

Wordsworth was in a similar vein of reminiscence in 1839, when trying to acquire of a copy of The Hurricane for a visitor, the wealthy art connoisseur Richard Monckton Milnes. Milnes had found Wordsworth's copy and wanted one of his own for his extensive library. Wordsworth's thoughts immediately turned to Bristol. He wrote to his friend John Peace, an admirer of his poetry, who ran the Bristol City Library and was in touch with members of the old Cottle circle. In the course of this letter Wordsworth left a brief description of Gilbert: ‘he was a Barrister and had practised in the West Indies, and lived some time at Bristol, between the year –95 and –98, at which time I often conversed with him, and admired his genius though he was in fact insane’ (WL, VI: p. 726).