Replications and reproducible research


2 December 2009

reproducible research

Reproducible research

One of the best ways to get started with research in a new area is to try to replicate some existing research. In doing so, you will usually gain a much better understanding of the topic, and you will often discover some problems with the research, or develop ideas that will lead to a new research paper.

Unfortunately, a lot of papers are not reproducible because the data are not made available, or the description of the methods are not detailed enough. The good news is that there is a growing move amongst funding agencies and journals to make more research reproducible.  Peng, Dominici and Zeger (2006) and Koenker and Zeileis (2009) provide helpful discussions of new tools (especially Sweave) for making research easier to reproduce.

The International Journal of Forecasting is also encouraging researchers to make their data and computer code available in order to allow others to replicate the research. I have just written an editorial on this topic which will appear in the first issue of 2010. Here is an excerpt from the article:

As the leading journal in forecasting, the IJF has a responsibility to set research standards.

So, a couple of years ago, we started asking authors to make their data and code available on our website. Then last year we changed our guidelines for authors to say

Authors will normally be expected to submit a complete set of any data used in electronic form, or provide instructions for how to obtain them. Exceptions to this requirement may be made at the discretion of the handling editor. The author must describe methods and data sufficiently so the research can be replicated. The provision of code as well as data is encouraged, but not required.

This is consistent with the moves of many granting agencies which are now starting to require publicly funded research to make the data publicly available. Once the data are public, other researchers can verify (or otherwise) the conclusions drawn.

Six months ago, the International Journal of Forecasting website ( was redesigned to allow supplements and comments on each published paper. Supplementary information about a paper can be provided by authors and is freely available online. This can include data, computer code, large tables, extra figures, extended footnotes, extra relevant material, etc. Authors are required to provide whatever material is needed allow their results to be replicated without excessive difficulty.

Replication articles

It has become standard in most sciences for results to be replicated before being widely accepted. Remember cold fusion? Research findings that cannot be independently verified under the same or very similar conditions are little more than published opinions. In fact, the painstaking step-by-step duplication of published research is often the only way to properly assess the work done by others (Laine et al, 2007). While replicating research is accepted practice in medicine, chemistry, physics, and many other areas of science, it has not been part of the research culture in statistics, econometrics and other fields associated with forecasting.

The International Journal of Forecasting is trying to change this culture, and is willing to publish replication articles, especially if they provide new insights into published results. For example, we published Gardner & Diaz-Saiz (2008) which attempted to replicate Fildes et al (1998) and provided some useful new insight into the original results. In the next issue of the journal, there will also be an invited paper by Heiner Evanschitzky and Scott Armstrong on replications in forecasting research.  I hope everyone working in forecasting, statistics, econometrics and related fields will soon come to see replication studies as an important part of the research process.


  1. Evanschitzky, H. & Armstrong, J. S. (2010). Replications of forecasting research. International Journal of Forecasting, 26, to appear.

  2. Fildes, R., Hibon, M., Makridakis, S., & Meade, N. (1998). Generalising about univariate forecasting methods: Further empirical evidence. International Journal of Forecasting, 14, 339–358.

  3. Gardner, Jr., E. S. & Diaz-Saiz, J. (2008). Exponential smoothing in the telecommunications data. International Journal of Forecasting, 24, 170–174.

  4. Hyndman, R.J. (2010) Encouraging replications and reproducible research. International Journal of Forecasting, 26, to appear.

  5. Koenker, R. & Zeilis, A. (2009). On reproducible econometric research. Journal of Applied Econometrics, 24(5), 833–847.

  6. Laine, C., Goodman, S. N., Griswold, M. E., & Sox, H. C. (2007). Reproducible research: moving toward research the public can really trust. Annals of Internal Medicine, 146 (6), 450–453.

  7. Peng, R. D., Dominici, F., & Zeger, S. L. (2006). Reproducible epidemiologic research. American Journal of Epidemiology, 163 (9), 783–789.

Some useful websites