Long-surface-wave instability in dense granular flows down inclined planes is analysed using recently proposed three-dimensional constitutive equations. A full linear stability analysis of the local governing equations is performed and compared to previous experimental results obtained with glass beads. We show that the proposed rheology is able to capture all the features of the instability quantitatively. In particular, it predicts well the behaviour and scaling for the cutoff frequency of the instability observed in the experiments. This result gives strong support for the three-dimensional rheology proposed and suggests new terms in the Saint-Venant equations used to describe free-surface granular flows.