Archaeologies of colonialism have have called for exploring the culturally dynamic entanglements of people and objects while acknowledging the violence that accompanied these entanglements. Heeding these calls requires attention to how the state and state power were materialized, particularly in settler colonies where state apparatuses advanced unevenly, insidiously and clumsily. Here, I explore how the (mis)understandings and (mis)apprehensions of people and places that accompanied the halting expansion of colonial frontiers were materialized. Focusing on southern Africa's Highveld and Maloti-Drakensberg Mountains, I offer anxiety as a framework for conceiving of colonialisms as epistemic encounters: processes of ‘making sense’ of new people, things and places based on material practices, empirical experience and desire. Through a narrative of the nineteenth-century Maloti-Drakensberg told with archival, archaeological and ethnographic materials, I revisit a longstanding trope of southern African archaeology and historiography: refugia from social distress. I argue that refuge can be taken as a sense-making practice rather than as reaction to stress. I close with thoughts on what an anxiety framework can offer the still-developing field of African historical archaeology.