In the development of a series hybrid electric vehicle, the compact design of traction motor is required for the cost and efficiency of the vehicle. To solve the problem of the low driving torque by the motor downsizing, a reduction gear is installed between the motor and the wheel to increase the driving torque. The gear ratio of the reduction gear has an influence on the driving performances, and thus the selection of the gear ratio is an important factor for the compact design of the traction motor. In this paper, a dynamic simulation model of the vehicle has developed using a model based design to analyze the driving performances. Using the optimization algorithm to the simulation of the vehicle dynamic model, the optimal gear ratio which minimizes the motor maximum torque and power for the motor downsizing has obtained.