Ecological impacts from invasive plants that have been identified include reductions in biodiversity, changes in resource cycling, and disruptions of ecosystem function. To mitigate these negative ecological impacts, managers work to remove invasive plants. However, removal does not necessarily immediately lead to a return to the uninvaded ecological state. Similarly, the accumulation rate of ecological impacts following invader establishment is almost entirely unknown for most species, hindering identification of optimal management times. The accumulation and loss (so-called legacy effects) of impacts following invader establishment and removal represent an “invasion shadow.” To begin to understand invasion shadows, we measured the changes in biotic and abiotic ecological impacts during establishment and following removal of the forest understory invader Japanese stiltgrass. We found that when the abiotic metrics were considered, seeded areas became more functionally similar to the invaded landscape and removed areas became more similar to the uninvaded landscape. However, while the plant community did not change in a 3-yr period during a new invasion, following invader removal, it became less similar to both the invaded and uninvaded landscape altogether, suggesting legacies. Surprisingly, all changes occurred almost immediately and persisted following invader establishment and removal. Our results show, at least in a 3-yr period, that ecosystems can respond to changes in invader abundance, and in some cases simply removing the invader could result in long-term changes to the resident plant community.