This paper presents an extension of the formulation of wave propagation in transverse electric (TE) and transverse magnetic (TM) modes in the case of metallic circular waveguides filled with anisotropic metamaterials. The determined higher-order modes have been analyzed and exploited to the design of filters. Among the particularities of anisotropic material, the backward waves can propagate below the cut-off frequency. The numerical results for TE and TM modes have been compared with theoretical predictions. Good agreements were obtained. We analyzed a periodic structure containing waveguides filled with anisotropic metamaterial using the mode-matching technique. By using modal analysis, our approach reduced considerably the computation time compared to HFSS.