This chapter uses notions of language and civilisation to understand how disabled people were understood in the context of the British empire.The chapter begins with a discussion of the long-standing association in western European thought between language and humanity. Debates about whether animals had access to language, which gained renewed importance during the Enlightenment and again with evolutionism, identified language acquisition as the ‘true marker’ of what it meant to be human. This left those with aphasia, intellectual disabilities that affected speech or deaf muteness in a highly vulnerable position. Whilst the development of deaf education helped mitigate fears around disability, during the nineteenth century, it became deeply contested and the question as to whether deaf children should be educated using sign language or spoken English became of surprisingly wide importance. Drawing on the work of Douglas Baynton, who argues that in nineteenth-century America sign language was increasingly linked with an early stage of evolution and ‘foreignness’, I resituate these trends, found also in Britain, in the context of colonialism and imperialism. I argue that the intolerance of sign language was linked to a wider intolerance of difference, both the difference of disability and the difference of ‘race’.