Reproducability in forecasting research

Reproducibility is a necessary condition for replicability, because an inability to reproduceresults implies that the methods have not been specified sufficiently, thus precluding replication.This paper describes how two independent teams of researchers attempted to reproducethe empirical findings of an important paper, ‘‘Shrinkage estimators of time seriesseasonal factors and their effect on forecasting accuracy’’ (Miller & Williams, 2003). The twoteams proceeded systematically, reporting results both before and after receiving clarificationsfrom the authors of the original study. The teams were able to approximately reproduceeach other’s results, but not those of Miller and Williams. These discrepancies led todifferences in the conclusions as to the conditions under which seasonal damping outperformsclassical decomposition. The paper specifies the forecasting methods employed usinga flowchart. It is argued that this approach to method documentation is complementaryto the provision of computer code, as it is accessible to a broader audience of forecastingpractitioners and researchers. The significance of this research lies not only in its lessonsfor seasonal forecasting but also, more generally, in its approach to the reproduction offorecasting research.