Nonlinear dynamic stability of rotary drilling system is studied. Dynamic behavior of axial rigid-body motion is included. The effects of nominal rotational speed of the drill string and nominal weight on bit are investigated. It is observed that drilling operation is unstable for speeds lower than a threshold value. The results show that both velocity-weakening friction and regenerative cutting have great effect on stability of the system.