The stability of elliptically perturbed circular vortices is investigated in a two-layer shallow-water model, with constant background rotation. The fluid is bounded above and below by rigid and at surfaces. The linear stability analysis shows that elliptical perturbations are most unstable for moderate Burger numbers and vorticity shears. Shorter waves dominate for more sheared vortices. Shallow-water and quasi-geostrophic growth rates exhibit a striking similarity, except at each end of the Burger number domain. There, cyclones (anticyclones) with finite Rossby numbers are more (less) unstable than their quasi-geostrophic counterparts. A simple model gives a first-order trend for this bias.

Nonlinear model runs with initially perturbed vortices also show the similarity between the two dynamics. In these runs, elliptically deformed vortices stabilize as stationary rotating tripoles for moderate linear instability; on the other hand, strongly unstable vortices break as dipoles. During these nonlinear processes, energy transfers indicate that barotropic instability is at least as active as the baroclinic one. For tripole formation, the modal analysis of the perturbation exhibits a dominant contribution of the original wave and of the mean flow correction. The ageostrophic and divergent parts of the flow are respectively weak and negligible. The Lighthill equation proves that few internal gravity waves are generated during tripole formation or dipolar breaking. Finally, the effects of triangular perturbations on circular vortices and the formation of quadrupoles are briefly addressed.