In this paper, we propose a semiparametric varying-coefficient categorical panel data model in which covariates (variables affecting the coefficients) are purely categorical. This model has two features: first, fixed effects are included to allow for correlation between individual unobserved heterogeneity and the regressors; second, it allows for cross-sectional dependence through a general spatial error dependence structure. We derive a semiparametric estimator for our model by using a modified within transformation, and then show the asymptotic and finite properties for this estimator under large N and T . The Monte Carlo study shows that our methodology works well for both large N and T , and large N and small T cases. Finally, we illustrate our model by analyzing the effects of state-level banking regulations on the returns to scale of commercial banks in the US. Our empirical results suggest that returns to scale is higher in more regulated states than in less regulated states.