It is well known that when the geometry and/or coefficients allow stable trapped rays, the solution operator of the Helmholtz equation (a.k.a. the resolvent of the Laplacian) grows exponentially through a sequence of real frequencies tending to infinity. In this paper we show that, even in the presence of the strongest-possible trapping, if a set of frequencies of arbitrarily small measure is excluded, the Helmholtz solution operator grows at most polynomially as the frequency tends to infinity. One significant application of this result is in the convergence analysis of several numerical methods for solving the Helmholtz equation at high frequency that are based on a polynomial-growth assumption on the solution operator (e.g. $hp$-finite elements, $hp$-boundary elements, certain multiscale methods). The result of this paper shows that this assumption holds, even in the presence of the strongest-possible trapping, for most frequencies.