Research on phenotypic plasticity has often focused on how a given genotype responds to the changing physical environments such as temperature or diet. However, for many species the social environment has an equally important role because of competition for resources. During early development, the level of competition for limited (maternally provided) resources will often depend critically on the number of siblings. Therefore, competition among siblings should drive the evolution of genes that allow flexible responses to realized levels of competition and maternal resource availability. However, it is unknown whether genetically based differences between individuals exist in their response to the social environment that affect their future development. Using a quantitative trait locus approach in an experimental population of mice we demonstrate that effects of sibling number on body weight depend on individual genotype at seven loci, over and above the general negative litter size effect. Overall, these litter size-by-genotype interactions considerably modified the degree to which increasing litter size caused reduced weight. For example at one locus this effect leads to a 7% difference in body weight at week 7 between individuals experiencing the extremes of the normal range of litter sizes in our population (five to nine litter mates). The observed interaction between genotype and the competitive environment can produce differences in body weight that are similar in magnitude to the main effect of litter size on weight. Our results show that different genotypes respond to the social environment differentially and that interaction effects of genotype with litter size can be as important as genotype-independent effects of litter size.