We describe experiments and computer simulations of molecular deposition on a substrate in which the molecules (substituted adenine derivatives) self-assemble into ordered structures. The resulting structures depend strongly on the deposition rate (flux). In particular, there are two competing surface morphologies (α and β), which differ by their topology (interdigitated vs lamellar structure). Experimentally, the α phase dominates at both low and high flux, with the β phase being most important in the intermediate regime. A similar nonmonotonic behavior is observed on varying the substrate temperature. To understand these effects from a theoretical perspective, a lattice model is devised which reproduces qualitatively the topological features of both phases. Via extensive Monte Carlo studies we can, on the one hand, reproduce the experimental results and, on the other hand, obtain a microscopic understanding of the mechanisms behind this anomalous behavior. The results are discussed in terms of an interplay between kinetic trapping and temporal exploration of configuration space.