This brief describes an active control method to prevent unwanted nonlinear vibration response modes of a rotor-dynamic system. Nonlinear stiffness of components that support or surround a machine rotor can cause a response branch that extends critical vibration (resonance) over a wide interval of rotational speeds. Within this interval, jump transitions between alternative low amplitude and high amplitude response modes become possible. This paper explains how such behavior can be eliminated by applying control forces to the rotor based on dynamic feedback of strains measured in the stator structure. An optimal model-based controller synthesis is considered that combines a Lur'e-type Lyapunov function with a quadratic cost measure to penalize controller gain and bandwidth. Results are presented for an experimental flexible rotor system where nonlinear rotor-stator interaction occurs through a bearing with radial clearance. An active magnetic bearing applies control forces to the rotor in a separate plane. The results show the control technique can eliminate jump response modes and can significantly reduce mechanical stress associated with rub interaction of the rotor and stator. The influence of key parameters in the model and controller formulation are shown.