The convective instability in a plane liquid layer with time-dependent temperature profile is investigated by means of a general method suitable for linear stability analysis of an unsteady basic flow. The method is based on a non-normal approach, and predicts the onset of instability, critical wavenumber and time. The method is applied to transient Rayleigh–Bénard–Marangoni convection due to cooling by evaporation. Numerical results as well as theoretical scalings for the critical parameters as function of the Biot number are presented for the limiting cases of purely buoyancy-driven and purely surface-tension-driven convection. Critical parameters from calculations are in good agreement with those from experiments on drying polymer solutions, where the surface cooling is induced by solvent evaporation.