Two nonlinear diffusion equations for thin film epitaxy, with or without slope selection, are studied in this work. The nonlinearity models the Ehrlich–Schwoebel effect – the kinetic asymmetry in attachment and detachment of adatoms to and from terrace boundaries. Both perturbation analysis and numerical simulation are presented to show that such an atomistic effect is the origin of a nonlinear morphological instability, in a rough-smooth-rough pattern, that has been experimentally observed as transient in an early stage of epitaxial growth on rough surfaces. Initial-boundary-value problems for both equations are proven to be well-posed, and the solution regularity is also obtained. Galerkin spectral approximations are studied to provide both a priori bounds for proving the well-posedness and numerical schemes for simulation. Numerical results are presented to confirm part of the analysis and to explore the difference between the two models on coarsening dynamics.