Starting from the full set of governing equations including the weak relativistic effect, three coupled nonlinear equations with higher order nonlinear and dispersive terms are derived, which describe the nonlinear evolution of a three-dimensional electromagnetic wave packet propagating in a plasma. Two particular cases are considered, where the space dependence of the dependent variables is either on the space coordinate along the direction of propagation of the wave or on that along an arbitrary fixed direction. In both cases the three coupled equations reduce to a single equation. This single equation is used to study the modulational instability of a uniform electromagnetic wave train. It is found that due to the inclusion of the weak relativistic effect there is always modulational instability, and a higher order nonlinear term in the evolution equation has a stabilizing influence.