The built form of asylums in Britain and the United States in the late eighteenth and early nineteenth centuries can be better understood when they are seen in relation to prison architecture, despite repeated statements in the contemporary sources as well as in the scholarly literature that asylums are non-prisons. Taking a wide view of the institutions and the people involved in designing and promoting them, this article revisits the insights of Erving Goffman and Michel Foucault, who proposed the carceral institution as a category encompassing both asylums and prisons — a category usually rejected by scholars of the asylum. Working across textual and architectural sources, I argue that the category does indeed have validity in the cases of Britain and the US between 1790 and 1850. From the beginning, however, there was resistance to the conflation of asylums and prisons, and some asylum designers deployed space, style and nomenclature to construct differentiation. Class is key: the wider the gap between an institution’s founders and its intended inmates, the more carceral it was in architecture and regime. I show how comparing modern building types requires close attention to their changing character; both asylums and prisons were undergoing radical reforms in the period, and it is crucial to determine precisely what version of the prison is being invoked when an asylum is deemed to be prison-like or not prison-like. I examine the floor plan as a rhetorical device, and interpret the significance of pristine geometrical abstraction on the one hand and familiar domestic conventions on the other. Even decisions about whether to name the architect involved, and whether to include floor plans in the institution’s publicity, have significance, as does the naming of things: when to call a room a cell or a chamber.