To set the sleeping mode for the Yutu-2 rover, a visual pose prediction algorithm including terrain reconstruction and pose estimation was first studied. The terrain reconstruction precision is affected by using only the stereo navigation camera (Navcam) images and the rotation angles of the mast. However, the hazard camera (Hazcam) pose is fixed, and an image network was constructed by linking all of the Navcam and Hazcam stereoimages. Then, the Navcam pose was refined based on a multiview block bundle adjustment. The experimental results show that the mean absolute errors of the check points in the proposed algorithm were 10.4 mm over the range of $\boldsymbol{L}$ from 2.0 to 6.1 m, and the proposed algorithm achieved good prediction results for the rover pose (the average differences of the values of the pitch angle and the roll angle were −0.19 degrees and 0.29 degrees, respectively). Under the support of the proposed algorithm, engineers have completed the remote setting of the sleeping mode for Yutu-2 successfully in the Chang’e-4 mission operations.