How to get your paper rejected quickly

I sent this rejec­tion let­ter this morn­ing about a paper sub­mit­ted to the Inter­na­tional Jour­nal of Forecasting.

Dear XXXXX.

I am writ­ing to you regard­ing man­u­script ????? enti­tled “xxxxxxxxxxxx” which you sub­mit­ted to the Inter­na­tional Jour­nal of Fore­cast­ing.

It so hap­pens that I am aware that this paper was pre­vi­ously reviewed for the YYYYYYY jour­nal. It seems that you have not both­ered to make any of the changes rec­om­mended by the review­ers of your sub­mis­sion to YYYYYYY. Just sub­mit­ting the same paper to another jour­nal is extremely poor prac­tice, and I am dis­ap­pointed that you have taken this path. Review­ers spend a great deal of time pro­vid­ing com­ments, and it is dis­re­spect­ful to ignore them. I don’t expect you to do every­thing they say, but I would expect some of their com­ments to be helpful.

I am unwill­ing to con­sider the paper fur­ther for the Inter­na­tional Jour­nal of Fore­cast­ing. Read the pre­vi­ous reviews to know why. And before you sub­mit the paper to a new jour­nal, take the time to con­sider the reviews you have already been given.

Sin­cerely,

Rob Hyn­d­man
(Editor-​​in-​​Chief, Inter­na­tional Jour­nal of Fore­cast­ing)

I have writ­ten on this issue before. The peer-​​review sys­tem requires peo­ple to donate con­sid­er­able amounts of time to writ­ing reviews. In gen­eral, they do a great job and pro­vide help­ful com­ments. So it really annoys me when authors treat the sys­tem as a game with the aim to get a paper accepted with min­i­mal work, and with no inter­est in learn­ing from feedback.

Read the literature

I’ve just fin­ished another reviewer report for a jour­nal, and yet again I’ve had to make com­ments about read­ing the lit­er­a­ture. It’s not dif­fi­cult. Before you write a paper, read what other peo­ple have done. A sim­ple search on Google scholar will usu­ally do the trick. And before you sub­mit a paper, check again that you haven’t missed any­thing important.

The paper I reviewed today did not cite a sin­gle ref­er­ence from either of the two most active research groups in the area in the last ten years. Any search on the topic would have turned up about a dozen papers from these two groups alone.

I don’t mind if papers miss a ref­er­ence or two, espe­cially if they have been pub­lished in an obscure out­let. But I will rec­om­mend a straight reject if a paper hasn’t cited any of the most impor­tant papers from the last five years. Part of a researcher’s task is to engage with what has already been done, and show how any new ideas dif­fer from or extend on pre­vi­ous work.

Refereeing a journal article

I’ve writ­ten briefly on this before. For an excel­lent and more detailed dis­cus­sion of what is involved, there is a series of excel­lent posts on Pat Thomson’s blog:

  1. Ref­er­ee­ing a jour­nal arti­cle part 1: reading
  2. Ref­er­ee­ing a jour­nal arti­cle part 2: mak­ing a recommendation
  3. Ref­er­ee­ing a jour­nal arti­cle part 3: writ­ing the feedback

If every reviewer fol­lowed her advice, my life as an edi­tor would be much eas­ier, and the qual­ity of research would be improved.

Becoming a referee

I reg­u­larly get emails from peo­ple want­ing to be ref­er­ees for the Inter­na­tional Jour­nal of Fore­cast­ing, usu­ally with an accom­pa­ny­ing CV. This is not how the process works.

Ref­er­ees are almost always selected because they have pre­vi­ously writ­ten papers on a sim­i­lar topic to the man­u­script under review. If you want to be a ref­eree, then write good papers and get them pub­lished in schol­arly jour­nals. Very quickly you will be invited to ref­eree papers in the same jour­nals. But until you have demon­strated your own research skills, no edi­tor is going to trust you to assess the research of some­one else.

How to avoid annoying a referee

It’s not a good idea to annoy the ref­er­ees of your paper. They make rec­om­men­da­tions to the edi­tor about your work and it is best to keep them happy. There is an inter­est­ing dis­cus­sion on stats​.stack​ex​change​.com on this sub­ject. This inspired my own list below.

  • Explain what you’ve done clearly, avoid­ing unnec­es­sary jargon.
  • Don’t claim your paper con­tributes more than it actu­ally does. (I ref­er­eed a paper this week where the author claimed to have invented prin­ci­pal com­po­nent analysis!)
  • Ensure all fig­ures have clear cap­tions and labels.
  • Include cita­tions to the referee’s own work. Obvi­ously you don’t know who is going to ref­eree your paper, but you should aim to cite the main work in the area. It places your work in con­text, and keeps the ref­er­ees happy if they are the authors.
  • Make sure the cited papers say what you think they say. Sight what you cite!
  • Include proper cita­tions for all soft­ware pack­ages. If you are unsure how to cite an R pack­age, try the com­mand citation("packagename").
  • Never pla­gia­rise from other papers — not even sen­tence frag­ments. Use your own words. I’ve ref­er­eed a the­sis which had slabs taken from my own lec­ture notes includ­ing the typos.
  • Don’t pla­gia­rise from your own papers. Either ref­er­ence your ear­lier work, or pro­vide a sum­mary in new words.
  • Pro­vide enough detail so your work can be repli­cated. Where pos­si­ble, pro­vide the data and code. Make sure the code works.
  • When respond­ing to ref­eree reports, make sure you answer every­thing asked of you. (See my ear­lier post “Always lis­ten to review­ers”)
  • If you’ve revised the paper based on ref­er­ees’ com­ments, then thank them in the acknowl­edge­ments section.

For some applied papers, there are spe­cific sta­tis­ti­cal issues that need attention:

  • Give effect sizes with con­fi­dence inter­vals, not just p-​​values.
  • Don’t describe data using the mean and stan­dard devi­a­tion with­out indi­cat­ing whether the data were more-​​or-​​less sym­met­ric and unimodal.
  • Don’t split con­tin­u­ous data into groups.
  • Make sure your data sat­isfy the assump­tions of the sta­tis­ti­cal meth­ods used.

More tongue-​​in-​​cheek advice is pro­vided by Strat­ton and Neil (2005), “How to ensure your paper is rejected by the sta­tis­ti­cal reviewer”. Dia­betic Med­i­cine, 22(4), 371–373.

Feel free to add your own sug­ges­tions over at stats​.stack​ex​change​.com.

Always listen to reviewers

This week I was asked to review a paper that I had seen before. It had been sub­mit­ted to a jour­nal a few months ago and I had writ­ten a detailed report describ­ing some prob­lems with the paper, and not­ing a large num­ber of typos that needed fix­ing. That jour­nal had rejected the paper, the authors had sub­mit­ted it to a sec­ond jour­nal, and the paper ended up on my desk again for review. I was inter­ested to see what the authors had done about the prob­lems I had described. Alas, noth­ing had changed. Not even the typos. It was iden­ti­cal to the pre­vi­ous ver­sion with every error still there. So I sent the same report off to the sec­ond jour­nal advis­ing the edi­tor of the situation.

I’m not sure what the authors imag­ined would hap­pen. It is not uncom­mon for a paper to be sent to the same reviewer after it has been rejected by one jour­nal, espe­cially when the field of poten­tial review­ers for some top­ics is quite small. This paper was on an exten­sion to the auto­matic time series mod­el­ling pro­ce­dures pro­vided in the fore­cast pack­age for R. Since I am the author of the fore­cast pack­age for R, the prob­a­bil­ity of me being asked to review the paper is approx­i­mately one.

In gen­eral, always lis­ten to review­ers, even if the paper has been rejected by the jour­nal that sent you the reviews. While you are not obliged to respond to every­thing in a review when you are not send­ing a revi­sion back to the same jour­nal, you should think about what has been said and revise the paper accord­ingly. At least fix the typos! If the same review­ers see it again, they won’t be happy if you’ve ignored them. Also, they may just have some use­ful com­ments that will lead to improve­ments in your paper.

Joining an editorial board

Being on the edi­to­r­ial board of a jour­nal is a lot of work. I’m cur­rently Editor-​​in-​​Chief of the Inter­na­tional Jour­nal of Fore­cast­ing and pre­vi­ously I’ve been The­ory & Meth­ods Edi­tor of the Aus­tralian & New Zealand Jour­nal of Sta­tis­tics. Although it is time-​​consuming and often goes un-​​noticed, there are some impor­tant rewards that make it worth­while in my opinion.

  1. You are forced to read care­fully a lot of papers in your area of inter­est. Every­one intends read­ing the papers pub­lished in their area, but this activ­ity often gets neglected for more urgent activ­i­ties. When you are an edi­tor or asso­ciate edi­tor, you have to read the papers, and you have to read them thor­oughly. That way you are forced to keep up-​​to-​​date with new ideas.
  2. You become bet­ter known in your field. This tends to lead to invi­ta­tions to speak at con­fer­ences, write invited papers, etc.
  3. You get to shape the dis­ci­pline to some extent, at least once you are a man­ag­ing edi­tor. For exam­ple, you can arrange a spe­cial issue or a review paper on a topic that you think needs addressing.
  4. It brings with it some prestige.
  5. You get to see a lot of the lat­est research before every­one else, although these days, that is becom­ing less of an issue as most peo­ple put their papers online in repos­i­to­ries such as RePEc or arXiv.

I some­times get emails from peo­ple ask­ing to join the edi­to­r­ial board of the IJF (usu­ally accom­pa­nied by a resumé which shows the per­son has lit­tle expe­ri­ence in fore­cast­ing research). One such email prompted this post today. I will never invite some­one to join the IJF edi­to­r­ial board in this way. Please don’t ask. Instead, if you want to join an edi­to­r­ial board, there is no sub­sti­tute for the fol­low­ing approach:

  • write good papers that are pub­lished in the rel­e­vant jour­nal (if you’ve never pub­lished in the IJF you are unlikely to be con­sid­ered for the edi­to­r­ial board);
  • write good papers that are pub­lished in other jour­nals on the rel­e­vant topic (e.g., pub­lish good fore­cast­ing papers in JASA, JRSSBJF, etc.);
  • reg­u­larly attend any con­fer­ences asso­ci­ated with the jour­nal (that’s the ISF);
  • pro­vide high qual­ity ref­eree reports for papers sub­mit­ted to the jour­nal (almost all our asso­ciate edi­tors were peo­ple who did a ster­ling job as ref­er­ees first);
  • work in an area where we need addi­tional asso­ciate edi­tors (every year we con­sider what areas we need new asso­ciate edi­tors, and then we find the appro­pri­ate people).

Being on an edi­to­r­ial board gives you a posi­tion of lead­er­ship in your research com­mu­nity. But you don’t get to be a leader until you’ve proven your­self by writ­ing papers, attend­ing con­fer­ences, writ­ing ref­eree reports, etc.