How to get your paper rejected quickly

I sent this rejec­tion let­ter this morn­ing about a paper sub­mit­ted to the Inter­na­tional Jour­nal of Forecasting.

Dear XXXXX.

I am writ­ing to you regard­ing man­u­script ????? enti­tled “xxxxxxxxxxxx” which you sub­mit­ted to the Inter­na­tional Jour­nal of Fore­cast­ing.

It so hap­pens that I am aware that this paper was pre­vi­ously reviewed for the YYYYYYY jour­nal. It seems that you have not both­ered to make any of the changes rec­om­mended by the review­ers of your sub­mis­sion to YYYYYYY. Just sub­mit­ting the same paper to another jour­nal is extremely poor prac­tice, and I am dis­ap­pointed that you have taken this path. Review­ers spend a great deal of time pro­vid­ing com­ments, and it is dis­re­spect­ful to ignore them. I don’t expect you to do every­thing they say, but I would expect some of their com­ments to be helpful.

I am unwill­ing to con­sider the paper fur­ther for the Inter­na­tional Jour­nal of Fore­cast­ing. Read the pre­vi­ous reviews to know why. And before you sub­mit the paper to a new jour­nal, take the time to con­sider the reviews you have already been given.

Sin­cerely,

Rob Hyn­d­man
(Editor-​​in-​​Chief, Inter­na­tional Jour­nal of Fore­cast­ing)

I have writ­ten on this issue before. The peer-​​review sys­tem requires peo­ple to donate con­sid­er­able amounts of time to writ­ing reviews. In gen­eral, they do a great job and pro­vide help­ful com­ments. So it really annoys me when authors treat the sys­tem as a game with the aim to get a paper accepted with min­i­mal work, and with no inter­est in learn­ing from feedback.

Read the literature

I’ve just fin­ished another reviewer report for a jour­nal, and yet again I’ve had to make com­ments about read­ing the lit­er­a­ture. It’s not dif­fi­cult. Before you write a paper, read what other peo­ple have done. A sim­ple search on Google scholar will usu­ally do the trick. And before you sub­mit a paper, check again that you haven’t missed any­thing important.

The paper I reviewed today did not cite a sin­gle ref­er­ence from either of the two most active research groups in the area in the last ten years. Any search on the topic would have turned up about a dozen papers from these two groups alone.

I don’t mind if papers miss a ref­er­ence or two, espe­cially if they have been pub­lished in an obscure out­let. But I will rec­om­mend a straight reject if a paper hasn’t cited any of the most impor­tant papers from the last five years. Part of a researcher’s task is to engage with what has already been done, and show how any new ideas dif­fer from or extend on pre­vi­ous work.

Refereeing a journal article

I’ve writ­ten briefly on this before. For an excel­lent and more detailed dis­cus­sion of what is involved, there is a series of excel­lent posts on Pat Thomson’s blog:

  1. Ref­er­ee­ing a jour­nal arti­cle part 1: reading
  2. Ref­er­ee­ing a jour­nal arti­cle part 2: mak­ing a recommendation
  3. Ref­er­ee­ing a jour­nal arti­cle part 3: writ­ing the feedback

If every reviewer fol­lowed her advice, my life as an edi­tor would be much eas­ier, and the qual­ity of research would be improved.

Becoming a referee

I reg­u­larly get emails from peo­ple want­ing to be ref­er­ees for the Inter­na­tional Jour­nal of Fore­cast­ing, usu­ally with an accom­pa­ny­ing CV. This is not how the process works.

Ref­er­ees are almost always selected because they have pre­vi­ously writ­ten papers on a sim­i­lar topic to the man­u­script under review. If you want to be a ref­eree, then write good papers and get them pub­lished in schol­arly jour­nals. Very quickly you will be invited to ref­eree papers in the same jour­nals. But until you have demon­strated your own research skills, no edi­tor is going to trust you to assess the research of some­one else.

How to avoid annoying a referee

It’s not a good idea to annoy the ref­er­ees of your paper. They make rec­om­men­da­tions to the edi­tor about your work and it is best to keep them happy. There is an inter­est­ing dis­cus­sion on stats​.stack​ex​change​.com on this sub­ject. This inspired my own list below.

  • Explain what you’ve done clearly, avoid­ing unnec­es­sary jargon.
  • Don’t claim your paper con­tributes more than it actu­ally does. (I ref­er­eed a paper this week where the author claimed to have invented prin­ci­pal com­po­nent analysis!)
  • Ensure all fig­ures have clear cap­tions and labels.
  • Include cita­tions to the referee’s own work. Obvi­ously you don’t know who is going to ref­eree your paper, but you should aim to cite the main work in the area. It places your work in con­text, and keeps the ref­er­ees happy if they are the authors.
  • Make sure the cited papers say what you think they say. Sight what you cite!
  • Include proper cita­tions for all soft­ware pack­ages. If you are unsure how to cite an R pack­age, try the com­mand citation("packagename").
  • Never pla­gia­rise from other papers — not even sen­tence frag­ments. Use your own words. I’ve ref­er­eed a the­sis which had slabs taken from my own lec­ture notes includ­ing the typos.
  • Don’t pla­gia­rise from your own papers. Either ref­er­ence your ear­lier work, or pro­vide a sum­mary in new words.
  • Pro­vide enough detail so your work can be repli­cated. Where pos­si­ble, pro­vide the data and code. Make sure the code works.
  • When respond­ing to ref­eree reports, make sure you answer every­thing asked of you. (See my ear­lier post “Always lis­ten to review­ers”)
  • If you’ve revised the paper based on ref­er­ees’ com­ments, then thank them in the acknowl­edge­ments section.

For some applied papers, there are spe­cific sta­tis­ti­cal issues that need attention:

  • Give effect sizes with con­fi­dence inter­vals, not just p-​​values.
  • Don’t describe data using the mean and stan­dard devi­a­tion with­out indi­cat­ing whether the data were more-​​or-​​less sym­met­ric and unimodal.
  • Don’t split con­tin­u­ous data into groups.
  • Make sure your data sat­isfy the assump­tions of the sta­tis­ti­cal meth­ods used.

More tongue-​​in-​​cheek advice is pro­vided by Strat­ton and Neil (2005), “How to ensure your paper is rejected by the sta­tis­ti­cal reviewer”. Dia­betic Med­i­cine, 22(4), 371–373.

Feel free to add your own sug­ges­tions over at stats​.stack​ex​change​.com.

Always listen to reviewers

This week I was asked to review a paper that I had seen before. It had been sub­mit­ted to a jour­nal a few months ago and I had writ­ten a detailed report describ­ing some prob­lems with the paper, and not­ing a large num­ber of typos that needed fix­ing. That jour­nal had rejected the paper, the authors had sub­mit­ted it to a sec­ond jour­nal, and the paper ended up on my desk again for review. I was inter­ested to see what the authors had done about the prob­lems I had described. Alas, noth­ing had changed. Not even the typos. It was iden­ti­cal to the pre­vi­ous ver­sion with every error still there. So I sent the same report off to the sec­ond jour­nal advis­ing the edi­tor of the situation.

I’m not sure what the authors imag­ined would hap­pen. It is not uncom­mon for a paper to be sent to the same reviewer after it has been rejected by one jour­nal, espe­cially when the field of poten­tial review­ers for some top­ics is quite small. This paper was on an exten­sion to the auto­matic time series mod­el­ling pro­ce­dures pro­vided in the fore­cast pack­age for R. Since I am the author of the fore­cast pack­age for R, the prob­a­bil­ity of me being asked to review the paper is approx­i­mately one.

In gen­eral, always lis­ten to review­ers, even if the paper has been rejected by the jour­nal that sent you the reviews. While you are not obliged to respond to every­thing in a review when you are not send­ing a revi­sion back to the same jour­nal, you should think about what has been said and revise the paper accord­ingly. At least fix the typos! If the same review­ers see it again, they won’t be happy if you’ve ignored them. Also, they may just have some use­ful com­ments that will lead to improve­ments in your paper.

Joining an editorial board

Being on the edi­to­r­ial board of a jour­nal is a lot of work. I’m cur­rently Editor-​​in-​​Chief of the Inter­na­tional Jour­nal of Fore­cast­ing and pre­vi­ously I’ve been The­ory & Meth­ods Edi­tor of the Aus­tralian & New Zealand Jour­nal of Sta­tis­tics. Although it is time-​​consuming and often goes un-​​noticed, there are some impor­tant rewards that make it worth­while in my opinion.

  1. You are forced to read care­fully a lot of papers in your area of inter­est. Every­one intends read­ing the papers pub­lished in their area, but this activ­ity often gets neglected for more urgent activ­i­ties. When you are an edi­tor or asso­ciate edi­tor, you have to read the papers, and you have to read them thor­oughly. That way you are forced to keep up-​​to-​​date with new ideas.
  2. You become bet­ter known in your field. This tends to lead to invi­ta­tions to speak at con­fer­ences, write invited papers, etc.
  3. You get to shape the dis­ci­pline to some extent, at least once you are a man­ag­ing edi­tor. For exam­ple, you can arrange a spe­cial issue or a review paper on a topic that you think needs addressing.
  4. It brings with it some prestige.
  5. You get to see a lot of the lat­est research before every­one else, although these days, that is becom­ing less of an issue as most peo­ple put their papers online in repos­i­to­ries such as RePEc or arXiv.

I some­times get emails from peo­ple ask­ing to join the edi­to­r­ial board of the IJF (usu­ally accom­pa­nied by a resumé which shows the per­son has lit­tle expe­ri­ence in fore­cast­ing research). One such email prompted this post today. I will never invite some­one to join the IJF edi­to­r­ial board in this way. Please don’t ask. Instead, if you want to join an edi­to­r­ial board, there is no sub­sti­tute for the fol­low­ing approach:

  • write good papers that are pub­lished in the rel­e­vant jour­nal (if you’ve never pub­lished in the IJF you are unlikely to be con­sid­ered for the edi­to­r­ial board);
  • write good papers that are pub­lished in other jour­nals on the rel­e­vant topic (e.g., pub­lish good fore­cast­ing papers in JASA, JRSSBJF, etc.);
  • reg­u­larly attend any con­fer­ences asso­ci­ated with the jour­nal (that’s the ISF);
  • pro­vide high qual­ity ref­eree reports for papers sub­mit­ted to the jour­nal (almost all our asso­ciate edi­tors were peo­ple who did a ster­ling job as ref­er­ees first);
  • work in an area where we need addi­tional asso­ciate edi­tors (every year we con­sider what areas we need new asso­ciate edi­tors, and then we find the appro­pri­ate people).

Being on an edi­to­r­ial board gives you a posi­tion of lead­er­ship in your research com­mu­nity. But you don’t get to be a leader until you’ve proven your­self by writ­ing papers, attend­ing con­fer­ences, writ­ing ref­eree reports, etc.

Should you make your working papers public?

There seems to be two points of view on this with dif­fer­ent prac­tices in dif­fer­ent disciplines.

  1. Some researchers do not make their work pub­lic until after it has been accepted for pub­li­ca­tion in a jour­nal. Until that time, drafts of papers are only cir­cu­lated to close con­fi­dants and usu­ally marked “Do not distribute”.
  2. Work­ing papers are pub­lished on web sites and in web repos­i­to­ries (such as arXiv or RePEc) as soon as they are fin­ished, at about the same time they are sub­mit­ted to a journal.

Because I work with peo­ple in lots of dif­fer­ent fields, I come across both of these prac­tices. In the first sit­u­a­tion, I don’t post the work­ing paper on my web­site until all coau­thors agree, which is not until the paper is accepted at a jour­nal. In the sec­ond sit­u­a­tion, I post the work­ing paper on my web­site (and usu­ally also on RePEc) as soon as possible.

I don’t like the secrecy model at all, but it is hard to con­vince coau­thors who have been trained under that process to change. Dif­fer­ent jus­ti­fi­ca­tions are given for keep­ing things secret, depend­ing on who I ask. Here are some of them (in bold) with my thoughts on why the stated rea­sons make lit­tle sense.

  1. It pre­vents rival research groups know­ing what you are up to, and so allows you to stay one step ahead of every­one else. Of course, if every­one does this, then it is just as likely that your rival researchers are ahead already in ways you don’t know about. The result is that there is slower progress because there is not a free flow of infor­ma­tion between research groups. Also, since you don’t know what every­one else is doing, you are more likely to miss some­thing impor­tant that some­one else is work­ing on and waste a lot of time in the process. The most effi­cient pro­ce­dure is for infor­ma­tion to be shared as quickly and com­pletely as pos­si­ble. Yes, that helps your rivals, but it also helps you, and it helps progress in research.
  2. It pre­vents other researchers steal­ing your ideas before they are pub­lished. Pre­sum­ably the fear is that the work­ing paper will be leaked and some­one will copy the ideas and pub­lish it under their own name. There is a sim­ple solu­tion to this: pub­lish the work­ing paper under your own name with a date on it, prefer­ably in a pub­lic repos­i­tory. Then there is no motive for steal­ing the idea because it will eas­ily be shown that you did it first. Keep­ing work­ing papers secret makes it more likely that some­one will steal your ideas, not less likely.
  3. The work­ing paper may change sub­stan­tially before pub­li­ca­tion. That is true, but so what? Every­one knows that a work­ing paper is sub­ject to revi­sion before pub­li­ca­tion. It should be seen as an advance draft to sig­nal to every­one what you have done, and to enable them to start cit­ing it. There is the prob­lem of embar­rass­ing mis­takes being made pub­lic. Wait­ing until a jour­nal accepts the paper reduces the like­li­hood of embar­rass­ing mis­takes, but it doesn’t remove it entirely. Every­one who has pub­lished more than a hand­ful of papers will have writ­ten papers that con­tain errors, even with the ref­er­ee­ing process. If you are wor­ried about never mak­ing a pub­lic mis­take, you prob­a­bly shouldn’t be involved in research.
  4. Hav­ing a pub­lished work­ing paper may be against the jour­nal rules. I don’t know of any jour­nal that won’t pub­lish a paper if it has appeared in work­ing paper form. Most jour­nals not only explic­itly allow it, but also allow the work­ing paper to con­tinue to appear online even after the paper has appeared in a journal.
  5. The ref­er­ees will know who wrote it. This is true. A ref­eree can use Google to dis­cover the authors of a pub­lished work­ing paper. But does that really mat­ter? The blind ref­er­ee­ing model is based on the assump­tion that ref­er­ees will give bet­ter assess­ments if they don’t know who the authors are. I’m not sure that is true, and I haven’t seen any empir­i­cal evi­dence to sup­port it. Any­way, I don’t care if the ref­er­ees know that I am the author of the papers they are reviewing.

On the other hand, there are good rea­sons to have your work­ing papers dis­trib­uted widely and early.

  1. It increases your cita­tions. The more widely the paper is dis­trib­uted the more likely peo­ple are to cite it. Fur­ther, pub­lic repos­i­to­ries such as arXiv and RePEc are free, so a lot more peo­ple have access to the papers stored there then the papers pub­lished in the jour­nals which require expen­sive sub­scrip­tions. If the paper is only being pub­lished (and made pub­lic) a cou­ple of years after the ideas have been devel­oped, it is likely that research has moved on and your paper is not so rel­e­vant and there­fore not so citable.
  2. It pre­vents other researchers steal­ing your ideas because the ideas are dated and doc­u­mented ear­lier, as explained above.
  3. It allows feed­back from a wider range of peo­ple. I get email from a lot of peo­ple who read my work­ing papers, and some of them have some use­ful com­ments that can lead to improve­ments in the paper. It would be too late if these com­ments were received after it was published.

Part of the rea­son for this post is to con­vince my coau­thors that the secrecy prac­tice is a bad idea, even if every­one does it in your field. The only way to change the sit­u­a­tion is to start pub­lish­ing work­ing papers, and try­ing to con­vince every­one else to do the same. I hope this post will help that happen.

Feel free to com­ment if you agree or dis­agree. I’m espe­cially inter­ested in any other rea­sons peo­ple have for and against pub­lish­ing work­ing papers.