The flow in a stationary open cylinder driven by the constant rotation of the bottom endwall is unstable to three-dimensional perturbations for sufficiently large rotation rates. The bifurcated state takes the form of a rotating wave. Two distinct physical mechanisms responsible for the symmetry breaking are identified, which depend on whether the fluid depth is sufficiently greater or less than the cylinder radius. For deep systems, the rotating wave results from the instability of the near-wall jet that forms as the boundary layer on the rotating bottom endwall is turned into the interior. In this case the three-dimensional perturbations vanish at the air/water interface. On the other hand, for shallow systems, the fluid at radii less than about half the cylinder radius is in solid-body rotation whereas the fluid at larger radii has a strong meridional circulation. The interface between these two regions of flow is unstable to azimuthal disturbances and the resulting rotating wave persists all the way to the air/water interface. The flow dynamics are explored using three-dimensional Navier–Stokes computations and experimental results obtained via digital particle image velocimetry. The use of a flat stress-free model for the air/water interface reproduces the experimental results in the deep system but fails to capture the primary instability in the shallow system, even though the experimental imperfections, i.e. departures from a perfectly flat and clean air/water interface, are about the same for the deep and the shallow systems. The flat stress-free model boundary conditions impose a parity condition on the numerical solutions, and the consideration of an extended problem which reveals this hidden symmetry provides insight into the symmetry-breaking instabilities.