James Croll's Physical Theory of Secular Changes of Climate emerged during an age of revolution in geology that included the rise of the glacial theory and the search for its underlying causes. According to Croll, periods of high eccentricity are associated with the persistence of long glacial epochs, within which glaciations occur in alternate hemispheres when winter is at aphelion every ~11,000 years; however, astronomical forcing is only able to produce glaciation by means of physical agencies (climate feedbacks) that amplify the small effects of varying seasonal irradiation. Croll understood the importance of interglacial deposits because they provided evidence for the occurrence of multiple glaciations within his long glacial epochs. He was aware of the limitations of the terrestrial record and suggested that deep-sea sediments would contain a continuous succession of glacial-interglacial cycles. Contrary to a widespread view, however, Croll was not envisaging the advent of palaeoceanographic exploration avant la lettre, but instead was drawing attention to the inadequacy of the land record as a testbed of his astronomical theory. Yet, the marine record did eventually deliver a test of astronomical theories almost exactly 100 years after the publication of his 1875 book Climate and Time in their Geological Relations. Here, we provide an historical account of the technological and scientific developments that led to this and a summary of insights on astronomically paced climate changes from marine, terrestrial and ice core records. We finally assess Croll's ideas in the context of our current understanding of the theory of ice ages.