The general concept of symmetry is realized in manifold ways in different realms of reality, such as plants, animals, minerals, mathematical objects or human artefacts in literature, fine arts and society. In order to arrive at a common ground for this variedness a very general conceptualization of symmetry is proposed: the existence of substitutions, which, in the given context, do not lead to an essential change. This simple definition has multiple consequences. (i) The context dependence of the notion of symmetry is evident in the humanities but is by no means irrelevant, yet is often neglected, in science. By taking this context dependence into account, the subtle problematics of concept formation and of the ontological status of ‘similarities’ become evident. In general, the substitutions underlying the concept of symmetry are not really performed but remain in a state of virtuality. Counterfactuality, freedom and creativity come into focus. The detection of previously hidden symmetries may provide deep and surprising insights. (ii) Related to this, due attention is devoted to the aesthetic dimension of symmetry and the breaking of it. (iii) Finally, we point out to what extent life is based on the interplay between order and freedom, between full and broken symmetry.