Two sequential experiments examined the effects of abrasive grit on seedlings of grass weeds and young shoots of perennial weeds. First, four types of grit derived from agricultural residues (bone meal, eggshell, hazelnut shell, and sugar beet pulp) were tested under high air pressure in a controlled environment for their abilities to abrade seedlings of an annual grass, Setaria faberi Herrm., and the perennials Festuca arundinacea Schreb., Poa pratensis L., and Elymus repens (L.) Gould. Differing grit particle sizes and amounts, as well as weed seedling stages, were examined for efficacy after abrasion by each type of grit. Second, hazelnut shell grit was used to control P. pratensis and Taraxicum officinale Weber in field trials with aronia (Aronia melanocarpa [Michx.] Elliott), which is a new, shrubby, berry crop in the midwestern USA. Grit weeding was compared to two other treatments: manual weeding (hand-hoeing + hand-pulling) and no weed control (weedy check) over two years. In the grit comparison experiment, control of S. faberi was highest for egg-shell grit (63–100% across grit particle sizes, rates, and seedling stages) and least for sugar beet pulp (17–97%). The former grit had the highest bulk density of all grits, and the latter had the lowest bulk density. For damage to perennial weeds, egg-shell grit performed best (17–80% control) and bone meal least (10–47% control). Elymus repens was controlled better than other perennial grasses, especially by eggshell grit (up to 73% control) and hazelnut shell grit (up to 67% control) with particle sizes of 1–2 mm. In the aronia experiment, both grit abrasion and manual weeding achieved comparable levels of weed suppression (≥87%) and required similar amounts of cumulative seasonal time spent weeding (3–4 min per shrub). Thus, applications of abrasive grit derived from agricultural residues are potential alternatives for non-chemical management of weeds in aronia and, perhaps, in other high-value perennial crops.