At present, the study on autonomous unmanned ground vehicle navigation in an unstructured environment is still facing great challenges and is of great significance in scenarios where search and rescue robots, planetary exploration robots, and agricultural robots are needed. In this paper, we proposed an autonomous navigation method for unstructured environments based on terrain constraints. Efficient path search and trajectory optimization on octree map are proposed to generate trajectories, which can effectively avoid various obstacles in off-road environments, such as dynamic obstacles and negative obstacles, to reach the specified destination. We have conducted empirical experiments in both simulated and real environments, and the results show that our approach achieved superior performance in dynamic obstacle avoidance tasks and mapless navigation tasks compared to the traditional 2-dimensional or 2.5-dimensional navigation methods.