Zero velocity update (ZUPT) is widely discussed for error restriction in land vehicle Inertial Navigation Systems (INSs) and wearable pedestrian INSs to overcome the problems of Global Positioning System (GPS) unavailability in urban canyons or indoor scenarios. In this paper, an online smoothing method for ZUPT-aided INSs is presented. By introducing the Rauch–Tung–Striebel (RTS) smoothing method into the ZUPT-aided INS, position errors can be effectively restrained not only at stop points but during the whole trajectory. By integrating reverse navigation with a ZUPT smoother, the method realises forward and real-time processing. Compared with existing approaches, it can improve the position accuracy in real time without any other sensors, which is well suited for applications on high-accuracy navigation in GPS-challenging environments. Accuracy test results with different Inertial Measurement Units (IMUs) show that the developed method can significantly decrease position errors from hundreds or thousands of metres to below ten metres. During the whole trajectory, the online smoothing method ensures the maximum position errors at non-stop points can reach the same level of accuracy at stop points. A delay test result proves that the delay of the reverse online smoothing method proposed in this paper is much shorter than existing online smoothing methods.