The rotor coupling ability of brushless doubly fed machine (BDFM) is the most important factor that affects the performance of the machine. In order to improve the rotor coupling capability, the optimization design of rotor structure is significant extremely. In this paper, the mutual inductance between different winding is seen as the optimal target, and the improved particle swarm optimization (IPSO), which applies chaos theory, uses a random method to constrain boundary conditions and various inertia weight so as to improve the local convergence of the traditional particle swarm optimization (PSO), is proposed to optimize the rotor of BDFM with magnetic barrier rotor. Since there are both discrete variables and continuous variables in the optimization problem, binary processing is carried out on the discrete variables to realize the optimization of mixed variables. The simulation results verify the effectiveness and feasibility of IPSO on rotor optimization of BDFM.