The ad hominem appears to be the simplest fallacy form—one criticizes speakers instead of their statements or arguments. It is regularly taken to be a fallacy of irrelevance, in that who is speaking does not bear on the truth of what is said. But three puzzles attend this analysis. (1) Given that the fallacy is simple and seemingly obvious, how could it be effective in practice? (2) Are there not cases when who is speaking is relevant? How do we sort those cases from those where it is irrelevant? (3) Isn’t there another level to the ad hominem, one where we observe it, know it is a bad argumentative move, and make inferences about the argumentative circumstances and arguers in light of it? Accusing another of committing the ad hominem, on this line of thought, has broader implications about the reasoner and their reasons. This article is an attempt to tell a coherent story of the ad hominem that makes sense of these three puzzles and shows how the observed preponderance of the argument form should trouble us as reasoners.