It is very challenging for robots to perform grinding and polishing tasks on surfaces with unknown geometry. Most existing methods solve this problem by modeling the relationship between the force sensing information and surface normal vectors by analyzing the forces on special end tools such as spherical tools and cylindrical tools and simplified friction model. In this paper, we propose a normal vectors learning method to simultaneously control end-effector force and direction on unknown surfaces. First, the relation that mapping the force sensing information to the surface normal vectors is learned from the demonstrated data on the known plane using locally weighted regression. Next, the learned relation is used to estimate surface normal vectors on the unknown surface. To improve the force control precision on the unknown geometry surface, the adaptive force control is developed. To improve the direction control precision due to friction, the iterative learning control is developed. The proposed method is verified by comparative simulations and experiments using the Franka robot. Results show that the end-effector can be controlled perpendicular to the surface with a certain force.