We propose a modified voter model with locally conserved magnetization and investigate its phase ordering dynamics in two dimensions in numerical simulations. Imposing a local constraint on the dynamics has the surprising effect of speeding up the phase ordering process. The system is shown to exhibit a scaling regime characterized by algebraic domain growth, at odds with the logarithmic coarsening of the standard voter model. A phenomenological approach based on cluster diffusion and similar to Smoluchowski ripening correctly predicts the observed scaling regime. Our analysis exposes unexpected complexity in the phase ordering dynamics without thermodynamic potential.