If Wittgenstein's later account of language is applied to music, what seems to follow is a version of musical formalism. This is to say that the meaning of music is constituted by the rules of a given system of music, and the understanding of music is the ability to follow these rules. I argue that, while this view may seem unattractive at the outset, Wittgenstein actually held this view. Moreover, his later notion of a rule gives us resources to answer some of the traditional criticisms directed against formalism.