I’ve been an editor of JSS for the last few years, and as a result I tend to get email from people asking me about publishing papers describing R packages in JSS. So for all those wondering, here are some general comments. Continue reading →
I sent this rejection letter this morning about a paper submitted to the International Journal of Forecasting.
I am writing to you regarding manuscript ????? entitled “xxxxxxxxxxxx” which you submitted to the International Journal of Forecasting.
It so happens that I am aware that this paper was previously reviewed for the YYYYYYY journal. It seems that you have not bothered to make any of the changes recommended by the reviewers of your submission to YYYYYYY. Just submitting the same paper to another journal is extremely poor practice, and I am disappointed that you have taken this path. Reviewers spend a great deal of time providing comments, and it is disrespectful to ignore them. I don’t expect you to do everything they say, but I would expect some of their comments to be helpful.
I am unwilling to consider the paper further for the International Journal of Forecasting. Read the previous reviews to know why. And before you submit the paper to a new journal, take the time to consider the reviews you have already been given.
(Editor-in-Chief, International Journal of Forecasting)
I have written on this issue before. The peer-review system requires people to donate considerable amounts of time to writing reviews. In general, they do a great job and provide helpful comments. So it really annoys me when authors treat the system as a game with the aim to get a paper accepted with minimal work, and with no interest in learning from feedback.
The nature of research is that other people are probably working on similar ideas to you, and it is possible that someone will beat you to publishing them. Continue reading →
I’ve just finished another reviewer report for a journal, and yet again I’ve had to make comments about reading the literature. It’s not difficult. Before you write a paper, read what other people have done. A simple search on Google scholar will usually do the trick. And before you submit a paper, check again that you haven’t missed anything important.
The paper I reviewed today did not cite a single reference from either of the two most active research groups in the area in the last ten years. Any search on the topic would have turned up about a dozen papers from these two groups alone.
I don’t mind if papers miss a reference or two, especially if they have been published in an obscure outlet. But I will recommend a straight reject if a paper hasn’t cited any of the most important papers from the last five years. Part of a researcher’s task is to engage with what has already been done, and show how any new ideas differ from or extend on previous work.
- Refereeing a journal article part 1: reading
- Refereeing a journal article part 2: making a recommendation
- Refereeing a journal article part 3: writing the feedback
If every reviewer followed her advice, my life as an editor would be much easier, and the quality of research would be improved.
I regularly get emails from people wanting to be referees for the International Journal of Forecasting, usually with an accompanying CV. This is not how the process works.
Referees are almost always selected because they have previously written papers on a similar topic to the manuscript under review. If you want to be a referee, then write good papers and get them published in scholarly journals. Very quickly you will be invited to referee papers in the same journals. But until you have demonstrated your own research skills, no editor is going to trust you to assess the research of someone else.
It’s not a good idea to annoy the referees of your paper. They make recommendations to the editor about your work and it is best to keep them happy. There is an interesting discussion on stats.stackexchange.com on this subject. This inspired my own list below.
- Explain what you’ve done clearly, avoiding unnecessary jargon.
- Don’t claim your paper contributes more than it actually does. (I refereed a paper this week where the author claimed to have invented principal component analysis!)
- Ensure all figures have clear captions and labels.
- Include citations to the referee’s own work. Obviously you don’t know who is going to referee your paper, but you should aim to cite the main work in the area. It places your work in context, and keeps the referees happy if they are the authors.
- Make sure the cited papers say what you think they say. Sight what you cite!
- Include proper citations for all software packages. If you are unsure how to cite an R package, try the command
- Never plagiarise from other papers — not even sentence fragments. Use your own words. I’ve refereed a thesis which had slabs taken from my own lecture notes including the typos.
- Don’t plagiarise from your own papers. Either reference your earlier work, or provide a summary in new words.
- Provide enough detail so your work can be replicated. Where possible, provide the data and code. Make sure the code works.
- When responding to referee reports, make sure you answer everything asked of you. (See my earlier post “Always listen to reviewers”)
- If you’ve revised the paper based on referees’ comments, then thank them in the acknowledgements section.
For some applied papers, there are specific statistical issues that need attention:
- Give effect sizes with confidence intervals, not just p-values.
- Don’t describe data using the mean and standard deviation without indicating whether the data were more-or-less symmetric and unimodal.
- Don’t split continuous data into groups.
- Make sure your data satisfy the assumptions of the statistical methods used.
More tongue-in-cheek advice is provided by Stratton and Neil (2005), “How to ensure your paper is rejected by the statistical reviewer”. Diabetic Medicine, 22(4), 371–373.
Feel free to add your own suggestions over at stats.stackexchange.com.
This week I was asked to review a paper that I had seen before. It had been submitted to a journal a few months ago and I had written a detailed report describing some problems with the paper, and noting a large number of typos that needed fixing. That journal had rejected the paper, the authors had submitted it to a second journal, and the paper ended up on my desk again for review. I was interested to see what the authors had done about the problems I had described. Alas, nothing had changed. Not even the typos. It was identical to the previous version with every error still there. So I sent the same report off to the second journal advising the editor of the situation.
I’m not sure what the authors imagined would happen. It is not uncommon for a paper to be sent to the same reviewer after it has been rejected by one journal, especially when the field of potential reviewers for some topics is quite small. This paper was on an extension to the automatic time series modelling procedures provided in the forecast package for R. Since I am the author of the forecast package for R, the probability of me being asked to review the paper is approximately one.
In general, always listen to reviewers, even if the paper has been rejected by the journal that sent you the reviews. While you are not obliged to respond to everything in a review when you are not sending a revision back to the same journal, you should think about what has been said and revise the paper accordingly. At least fix the typos! If the same reviewers see it again, they won’t be happy if you’ve ignored them. Also, they may just have some useful comments that will lead to improvements in your paper.
Being on the editorial board of a journal is a lot of work. I’m currently Editor-in-Chief of the International Journal of Forecasting and previously I’ve been Theory & Methods Editor of the Australian & New Zealand Journal of Statistics. Although it is time-consuming and often goes un-noticed, there are some important rewards that make it worthwhile in my opinion.
- You are forced to read carefully a lot of papers in your area of interest. Everyone intends reading the papers published in their area, but this activity often gets neglected for more urgent activities. When you are an editor or associate editor, you have to read the papers, and you have to read them thoroughly. That way you are forced to keep up-to-date with new ideas.
- You become better known in your field. This tends to lead to invitations to speak at conferences, write invited papers, etc.
- You get to shape the discipline to some extent, at least once you are a managing editor. For example, you can arrange a special issue or a review paper on a topic that you think needs addressing.
- It brings with it some prestige.
- You get to see a lot of the latest research before everyone else, although these days, that is becoming less of an issue as most people put their papers online in repositories such as RePEc or arXiv.
I sometimes get emails from people asking to join the editorial board of the IJF (usually accompanied by a resumé which shows the person has little experience in forecasting research). One such email prompted this post today. I will never invite someone to join the IJF editorial board in this way. Please don’t ask. Instead, if you want to join an editorial board, there is no substitute for the following approach:
- write good papers that are published in the relevant journal (if you’ve never published in the IJF you are unlikely to be considered for the editorial board);
- write good papers that are published in other journals on the relevant topic (e.g., publish good forecasting papers in JASA, JRSSB, JF, etc.);
- regularly attend any conferences associated with the journal (that’s the ISF);
- provide high quality referee reports for papers submitted to the journal (almost all our associate editors were people who did a sterling job as referees first);
- work in an area where we need additional associate editors (every year we consider what areas we need new associate editors, and then we find the appropriate people).
Being on an editorial board gives you a position of leadership in your research community. But you don’t get to be a leader until you’ve proven yourself by writing papers, attending conferences, writing referee reports, etc.
There seems to be two points of view on this with different practices in different disciplines.
- Some researchers do not make their work public until after it has been accepted for publication in a journal. Until that time, drafts of papers are only circulated to close confidants and usually marked “Do not distribute”.
- Working papers are published on web sites and in web repositories (such as arXiv or RePEc) as soon as they are finished, at about the same time they are submitted to a journal.
Because I work with people in lots of different fields, I come across both of these practices. In the first situation, I don’t post the working paper on my website until all coauthors agree, which is not until the paper is accepted at a journal. In the second situation, I post the working paper on my website (and usually also on RePEc) as soon as possible.
I don’t like the secrecy model at all, but it is hard to convince coauthors who have been trained under that process to change. Different justifications are given for keeping things secret, depending on who I ask. Here are some of them (in bold) with my thoughts on why the stated reasons make little sense.
- It prevents rival research groups knowing what you are up to, and so allows you to stay one step ahead of everyone else. Of course, if everyone does this, then it is just as likely that your rival researchers are ahead already in ways you don’t know about. The result is that there is slower progress because there is not a free flow of information between research groups. Also, since you don’t know what everyone else is doing, you are more likely to miss something important that someone else is working on and waste a lot of time in the process. The most efficient procedure is for information to be shared as quickly and completely as possible. Yes, that helps your rivals, but it also helps you, and it helps progress in research.
- It prevents other researchers stealing your ideas before they are published. Presumably the fear is that the working paper will be leaked and someone will copy the ideas and publish it under their own name. There is a simple solution to this: publish the working paper under your own name with a date on it, preferably in a public repository. Then there is no motive for stealing the idea because it will easily be shown that you did it first. Keeping working papers secret makes it more likely that someone will steal your ideas, not less likely.
- The working paper may change substantially before publication. That is true, but so what? Everyone knows that a working paper is subject to revision before publication. It should be seen as an advance draft to signal to everyone what you have done, and to enable them to start citing it. There is the problem of embarrassing mistakes being made public. Waiting until a journal accepts the paper reduces the likelihood of embarrassing mistakes, but it doesn’t remove it entirely. Everyone who has published more than a handful of papers will have written papers that contain errors, even with the refereeing process. If you are worried about never making a public mistake, you probably shouldn’t be involved in research.
- Having a published working paper may be against the journal rules. I don’t know of any journal that won’t publish a paper if it has appeared in working paper form. Most journals not only explicitly allow it, but also allow the working paper to continue to appear online even after the paper has appeared in a journal.
- The referees will know who wrote it. This is true. A referee can use Google to discover the authors of a published working paper. But does that really matter? The blind refereeing model is based on the assumption that referees will give better assessments if they don’t know who the authors are. I’m not sure that is true, and I haven’t seen any empirical evidence to support it. Anyway, I don’t care if the referees know that I am the author of the papers they are reviewing.
On the other hand, there are good reasons to have your working papers distributed widely and early.
- It increases your citations. The more widely the paper is distributed the more likely people are to cite it. Further, public repositories such as arXiv and RePEc are free, so a lot more people have access to the papers stored there then the papers published in the journals which require expensive subscriptions. If the paper is only being published (and made public) a couple of years after the ideas have been developed, it is likely that research has moved on and your paper is not so relevant and therefore not so citable.
- It prevents other researchers stealing your ideas because the ideas are dated and documented earlier, as explained above.
- It allows feedback from a wider range of people. I get email from a lot of people who read my working papers, and some of them have some useful comments that can lead to improvements in the paper. It would be too late if these comments were received after it was published.
Part of the reason for this post is to convince my coauthors that the secrecy practice is a bad idea, even if everyone does it in your field. The only way to change the situation is to start publishing working papers, and trying to convince everyone else to do the same. I hope this post will help that happen.
Feel free to comment if you agree or disagree. I’m especially interested in any other reasons people have for and against publishing working papers.