Typical rotor/active magnetic bearing (AMB) system layouts involving large, external stator AMBs may be difficult or inconvenient to apply to some rotor systems. Where space in the machine working envelope is at a premium, the space required by traditional AMBs may preclude them from inclusion in the design. To open up the possibility of using AMBs in next generation compact, high speed machines, a system topology whereby the magnetic bearing stators are positioned inside of hollow-shaft rotors is suggested. This leaves the entire rotor surface available for other machine elements. In such designs, it is probably that both the rotor and the secondary shaft may exhibit flexible behaviour, which adds complexities to the design of the AMB controller compared to the requirements in typical AMB systems. Satisfactory performance can only be achieved if the dynamic characteristics of both rotor and AMB support structure are considered. This paper investigates solutions to this control issue, particularly through the use of model based techniques. A unique experimental facility based on this system topology is presented. The rotor is sufficiently unbalanced so as to be unable to pass its first critical speed without experiencing excessive vibration. It is demonstrated how an appropriately designed AMB controller can reduce the vibration to a level which allows the rotor to reach up to three times its first critical speed. This also includes the rotor speed (i.e. excitation frequency) exceeding the natural frequency of the AMB support structure.