In this paper I argue that we should attend to why and how forgetting happens in concert with the construction of social memory, history, identity and heritage. Through a focus on processes of forgetting, this discussion offers a new set of interpretations of early colonial Sylvester Manor, a 17th-century plantation site in coastal New York. More specifically, the construction of racial categories over several centuries implicates social memory and forgetting, and introduces issues to the manner in which we remember the site where people of European, African and Native American ancestry met. This analysis views memory and forgetting not only as historical vectors in racialization, but also as factors in current identity politics.