This paper investigates the productivity and efficiency of large bank holding companies (BHCs) in the United States over the period 2004–2013, by estimating a translog stochastic distance frontier (SDF) model with time-varying heterogeneity. The main feature of this model is that a multi-factor structure is used to disentangle time-varying unobserved heterogeneity from inefficiency. Our empirical results strongly suggest that unobserved heterogeneity is not only present in the U.S. banking industry, but also varies over time. Our results from the translog SDF model with time- varying heterogeneity show that the majority of large BHCs in the U.S. exhibit increasing returns to scale, a small percentage exhibit constant returns to scale, and an even smaller percentage exhibit decreasing returns to scale. Our results also show that on average the BHCs have experienced small positive or even negative technical change and productivity growth.