Offspring number and size are key traits determining an individual’s fitness and a crop’s yield. Yet, extensive natural variation within species is observed for these traits. Such variation is typically explained by trade-offs between fecundity and quality, for which an optimal solution is environmentally dependent. Understanding the genetic basis of seed size and number, as well as any possible genetic constraints preventing the maximization of both, is crucial from both an evolutionary and applied perspective. We investigated the genetic basis of natural variation in seed size and number using a set of Arabidopsis thaliana multiparent advanced generation intercross (MAGIC) lines. We also tested whether life history affects seed size, number, and their trade-off. We found that both seed size and seed number are affected by a large number of mostly nonoverlapping QTL, suggesting that seed size and seed number can evolve independently. The allele that increases seed size at most identified QTL is from the same natural accession, indicating past occurrence of directional selection for seed size. Although a significant trade-off between seed size and number is observed, its expression depends on life-history characteristics, and generally explains little variance. We conclude that the trade-off between seed size and number might have a minor role in explaining the maintenance of variation in seed size and number, and that seed size could be a valid target for selection.