Tempering cooling rate plays a significant role in the impact toughness of 2CrMoV weld metal. Three different tempering cooling rate experiments were carried out; it is found that the impact toughness of weld metal improved from 44.61 to 117.49 J as the cooling rate increased from 5 to 40 °C/h. Microstructure characterization revealed that the large blocky M–A constituents and cluster precipitation were considered to act as stress concentration sources and cleavage fracture initiators at a cooling rate of 5 °C/h. Under the cooling rate of 20 °C/h, the decrease of blocky M–A constituents as well as homogeneous distribution of precipitation induced the transition from cleavage to interfacial decohesion. The chance of crack propagation in intragranular ferrite matrix was increased, which needed to absorb more energy and improve impact toughness. When the tempering cooling rate reached at 40 °C/h, the cracks mainly propagated in the ferrite matrix; meanwhile, fine and homogeneous distribution of precipitation greatly inhibited crack propagation and led to higher impact toughness.