This work proposes a design optimization approach for a high-speed rotor system, commonly used in a wide range of industrial applications, in order to address the issue with the flexural flexibility of a rotor system which can cause significant vibration and noise problems due to shaft whirl. For this purpose, the numerical modeling of a flexible rotor system was used to carry out the optimization process considering the effects of the mass and transverse natural frequency to the optimized design. Torque and geometrical sizing were also incorporated as the constraints in the optimization. The rotor system was modelled as a multi-disc Jeffcott rotor system that allows a rapid design optimization process. A case study on a high-speed rotor was carried out to demonstrate the effectiveness of the developed optimization approach, yielding the reduction of the rotor mass by at least 45.6%, while the fundamental transverse natural frequency has been increased by 2.91% as the result..