Numerous laboratory and field experiments on nonlinear surface wave trains propagating in deep water (Lake & Yuen 1978; Ramamonjiarisoa & Mollo-Christensen 1979; Mollo-Christensen & Ramamonjiarisoa 1982; Melville 1983) have showed a specific wave modulation that so far has not been explained by nonlinear theories. Typical effects were the so-called wave phase reversals, negative frequencies, and crest pairing, experimentally observed in some portions of the modulated wave train. In the present paper, in order to explain these modulation manifestations, the equations for wavenumber, frequency, and velocity potential amplitude are derived consistently in the third-order approximation related to the wave steepness. The resulting model generalizes, for instance, the well-known nonlinear Schrödinger equation theory, to which it transforms at certain values of the governing parameters.

The stationary solutions to the derived set of equations are found in quadrature and then analysed. Within well-defined ranges of the model parameters, these solutions explicitly manifest the above-mentioned wave modulation effects. In particular, they show the wave phase kinks to arise on areas of relatively small free-surface displacement in complete accordance with the experiments.

The model with deeply modulated wavenumber and frequency permits one also to analyse the appropriately short surface wavepackets and modulation periods. In this case, a variety of new interesting wave solutions arises revealing complicated alteration of smooth and rough portions of the free surface. Of special importance are solitary waves, naturally generalizing envelope solitons of the nonlinear Schrödinger equation, but having a varying frequency (as a principle of the proposed theory) and a non-zero wave ‘pedestal’ at infinity. These new types of modulated surface waves should be also observable in laboratory tanks and under field conditions, because the relevant free parameters of theory are not extreme.