This paper considers the lateral undulation motion of snake robots. The first contribution of the paper is a model of lateral undulation dynamics developed for control design and stability analysis purposes. The second contribution is an analysis of the simplified model that shows that any asymptotically stabilizing control law for the snake robot to an equilibrium point must be time varying. Furthermore, the analysis shows that a snake robot (with four links) is strongly accessible from almost any equilibrium point, except for certain singular configurations, and that the robot does not satisfy sufficient conditions for small-time local controllability. The third contribution is based on using averaging theory to prove that the average velocity of the robot during lateral undulation will converge exponentially fast to a steady-state velocity which is given analytically as a function of the gait pattern parameters. From the averaging analysis, we also derive a set of fundamental relationships between the gait parameters of lateral undulation and the resulting forward velocity of the snake robot. The paper presents simulation results and results from experiments with a physical snake robot that support the theoretical findings.