In this article, the multipolar vortex instability of the flow in a finite cylinder is addressed. The experimental study uses a rotating elastic deformable tube filled with water which is elliptically or triangularly deformed by two or three rollers. The experimental control parameters are the cylinder aspect ratio and the Reynolds number based on the angular frequency.

For Reynolds numbers close to threshold, different instability modes are visualized using anisotropic particles, according to the value of the aspect ratio. These modes are compared with those predicted by an asymptotic stability theory in the limit of small deformations and large Reynolds numbers. A very good agreement is obtained which confirms the instability mechanism; for both elliptic and triangular configurations, the instability is due to the resonance of two normal modes (Kelvin modes) of the underlying rotating flow with the deformation field. At least four different elliptic instability modes, including combinations of Kelvin modes with azimuthal wavenumbers m = 0 and m = 2 and Kelvin modes m = 1 and m = 3 are visualized. Two different triangular instability modes which are a combination of Kelvin modes m = −1 and m = 2 and a combination of Kelvin modes m = 0 and m = 3 are also evidenced.

The nonlinear dynamics of a particular elliptic instability mode, which corresponds to the combination of two stationary Kelvin modes m = −1 and m = 1, is examined in more detail using particle image velocimetry (PIV). The dynamics of the phase and amplitude of the instability mode is shown to be predicted well by the weakly nonlinear analysis for moderate Reynolds numbers. For larger Reynolds number, a secondary instability is observed. Below a Reynolds number threshold, the amplitude of this instability mode saturates and its frequency is shown to agree with the predictions of Kerswell (1999). Above this threshold, a more complex dynamic develops which is only sustained during a finite time. Eventually, the two-dimensional stationary elliptic flow is reestablished and the destabilization process starts again.