Repro­ducible research

One of the best ways to get started with research in a new area is to try to repli­cate some exist­ing research. In doing so, you will usu­ally gain a much bet­ter under­stand­ing of the topic, and you will often dis­cover some prob­lems with the research, or develop ideas that will lead to a new research paper.

Unfor­tu­nately, a lot of papers are not repro­ducible because the data are not made avail­able, or the descrip­tion of the meth­ods are not detailed enough. The good news is that there is a grow­ing move amongst fund­ing agen­cies and jour­nals to make more research repro­ducible.  Peng, Dominici and Zeger (2006) and Koenker and Zeileis (2009) pro­vide help­ful dis­cus­sions of new tools (espe­cially Sweave) for mak­ing research eas­ier to reproduce.

The Inter­na­tional Jour­nal of Fore­cast­ing is also encour­ag­ing researchers to make their data and com­puter code avail­able in order to allow oth­ers to repli­cate the research. I have just writ­ten an edi­to­r­ial on this topic which will appear in the first issue of 2010. Here is an excerpt from the article:

As the lead­ing jour­nal in fore­cast­ing, the IJF has a respon­si­bil­ity to set research standards.

So, a cou­ple of years ago, we started ask­ing authors to make their data and code avail­able on our web­site. Then last year we changed our guide­lines for authors to say

Authors will nor­mally be expected to sub­mit a com­plete set of any data used in elec­tronic form, or pro­vide instruc­tions for how to obtain them. Excep­tions to this require­ment may be made at the dis­cre­tion of the han­dling edi­tor. The author must describe meth­ods and data sufficiently so the research can be repli­cated. The
pro­vi­sion of code as well as data is encour­aged, but not required.

This is con­sis­tent with the moves of many grant­ing agen­cies which are now start­ing to require pub­licly funded research to make the data pub­licly avail­able. Once the data are pub­lic, other researchers can ver­ify (or oth­er­wise) the con­clu­sions drawn.

Six months ago, the Inter­na­tional Jour­nal of Fore­cast­ing web­site (www.forecasters.org/ijf) was redesigned to allow sup­ple­ments and com­ments on each pub­lished paper. Sup­ple­men­tary infor­ma­tion about a paper can be pro­vided by authors and is freely avail­able online. This can include data, com­puter code, large tables, extra figures, extended foot­notes, extra rel­e­vant mate­r­ial, etc. Authors are required to pro­vide what­ever mate­r­ial is needed allow their results to be repli­cated with­out exces­sive difficulty.

Repli­ca­tion articles

It has become stan­dard in most sci­ences for results to be repli­cated before being widely accepted. Remem­ber cold fusion? Research find­ings that can­not be inde­pen­dently ver­i­fied under the same or very sim­i­lar con­di­tions are lit­tle more than pub­lished opin­ions. In fact, the painstak­ing step-by-step dupli­ca­tion of pub­lished research is often the only way to prop­erly assess the work done by oth­ers (Laine et al, 2007). While repli­cat­ing research is accepted prac­tice in med­i­cine, chem­istry, physics, and many other areas of sci­ence, it has not been part of the research cul­ture in sta­tis­tics, econo­met­rics and other fields asso­ci­ated with forecasting.

The Inter­na­tional Jour­nal of Fore­cast­ing is try­ing to change this cul­ture, and is will­ing to pub­lish repli­ca­tion arti­cles, espe­cially if they pro­vide new insights into pub­lished results. For exam­ple, we pub­lished Gard­ner & Diaz-Saiz (2008) which attempted to repli­cate Fildes et al (1998) and pro­vided some use­ful new insight into the orig­i­nal results. In the next issue of the jour­nal, there will also be an invited paper by Heiner Evan­schitzky and Scott Arm­strong on repli­ca­tions in fore­cast­ing research.  I hope every­one work­ing in fore­cast­ing, sta­tis­tics, econo­met­rics and related fields will soon come to see repli­ca­tion stud­ies as an impor­tant part of the research process.

Ref­er­ences

  1. Evan­schitzky, H. & Arm­strong, J. S. (2010). Repli­ca­tions of fore­cast­ing research. Inter­na­tional Jour­nal of Fore­cast­ing, 26, to appear.
  2. Fildes, R., Hibon, M., Makri­dakis, S., & Meade, N. (1998). Gen­er­al­is­ing about uni­vari­ate fore­cast­ing meth­ods: Fur­ther empir­i­cal evi­dence. Inter­na­tional Jour­nal of Fore­cast­ing, 14, 339–358.
  3. Gard­ner, Jr., E. S. & Diaz-Saiz, J. (2008). Expo­nen­tial smooth­ing in the telecom­mu­ni­ca­tions data. Inter­na­tional Jour­nal of Fore­cast­ing, 24, 170–174.
  4. Hyn­d­man, R.J. (2010) Encour­ag­ing repli­ca­tions and repro­ducible research. Inter­na­tional Jour­nal of Fore­cast­ing, 26, to appear.
  5. Koenker, R. & Zeilis, A. (2009). On repro­ducible econo­met­ric research. Jour­nal of Applied Econo­met­rics, 24(5), 833–847.
  6. Laine, C., Good­man, S. N., Gris­wold, M. E., & Sox, H. C. (2007). Repro­ducible research: mov­ing toward research the pub­lic can really trust. Annals of Inter­nal Med­i­cine, 146 (6), 450–453.
  7. Peng, R. D., Dominici, F., & Zeger, S. L. (2006). Repro­ducible epi­demi­o­logic research. Amer­i­can Jour­nal of Epi­demi­ol­ogy, 163 (9), 783–789.

Some use­ful websites

  • Share/Bookmark