Wind forcing injects energy into mesoscale eddies and near-inertial waves (NIWs) in the ocean, and the NIWs are believed to solve the puzzle of mesoscale energy budget by absorbing energy from mesoscale eddies. This work studies the turbulent energy transfer in the NIW–quasigeostrophic mean mesoscale eddy coupled system based on a previously derived two-dimensional model which inherits conserved quantities in Boussinesq equations (Xie & Vanneste, J. Fluid Mech., vol. 774, 2015, pp. 147–169). The conservation of energy, potential enstrophy and wave action implies the existence of phase transition with a change of the relative strength between NIW and mean-flow, quantified by a parameter $R$. By running forced-dissipative numerical simulations, we justify the existence of second-order phase transition around a critical value $R_c$. When $0<R<R_c$, energy transfers bidirectionally, wave action transfers downscale and vorticity forms strong cyclones. When $R>R_c$, energy transfers downscale, wave action transfers bidirectionally and vortex filaments are dominant. We find the catalytic wave induction mechanism where the NIW induces a downscale energy flux of the mean flow, which differs from the stimulated loss of balance mechanism observed in inertial value problems. In the parameter regime $0<R<R_c$, catalytic wave induction is similar to the stimulated loss of balance as the downscale energy transfer is proportional to the NIW energy injection, however, catalytic wave induction has a distinct feature of the absence of energy conversion from mesoscale eddies to NIWs. The new mechanism is effective in the toy-model study, making it potentially important for ocean energetics.