Research tips

Why God never received tenure

2Sep 2010

He had only one major publication.
It was in Hebrew.
It had no references.
It wasn’t published in a refereed journal.
Some even doubt he wrote it by himself.
It may be true that he created the world, but what has he done since then?
The scientific community has had a hard time replicating his results.
He never applied to the ethics board for permission to use human subjects.
When one experiment went awry he tried to cover it by drowning his subjects.
When subjects didn’t behave as predicted, he deleted them from the sample.
He rarely came to class, just told students to read the book.
Some say he had his son teach the class.
He expelled his first two students for learning.
Although there were only 10 requirements, most of his students failed his tests.
His office hours were infrequent and often held on limited access mountain tops.
No record of working well with colleagues.

This list must have appeared on thousands of sites and I’ve not been able to track down the source. In fact, a search on the phrase yields over 43,000 results on Google. There are another 3,700 where it is titled “Why God never received a PhD”. If anyone knows the original source, please post a comment. (After all, this is a research site and we have to credit sources appropriately.)

Tags: humour, phd

How to fail a PhD

2Sep 2010

No Comments

I read an interesting post today by Matt Might on “10 reasons PhD students fail”, and I thought it might be helpful to reflect on some of the barriers to PhD completion that I’ve seen. Matt’s ideas are not all relevant to Australian PhDs, so I have come up with my own list below. Here are the seven steps to failure.

1. Wait for your supervisor to tell you what to do

A good supervisor will not tell you what to do. PhD students are not meant to be research assistants, and a PhD is not an extended undergraduate assignment. So waiting to be told what to do next will usually get you nowhere.

By the time you graduate with a PhD, you are supposed to be an independent researcher. That means having your own ideas, setting your own research directions, and choosing what to do yourself. In practice, your supervisor will usually need to tell you what to do for the first year, but eventually you need to set the research agenda yourself. By the third year you should certainly know more about your topic than your supervisor, and so are in a better position to know what to do next.

2. Wait for inspiration

Sitting around waiting for great ideas to pop into your ahead is unlikely to work. Most of my best ideas come after a lot of work trying different things and becoming totally immersed in the problem.

A good way to start is often to try to replicate someone else’s research, or apply someone’s method on a different data set. In the process you might notice something that doesn’t quite work, or you might think of a better way to do it. At the very least you will have a deeper understanding of what they have done than you will get by simply reading their paper.

Research often involves dead-ends, wrong turns, and failures. It’s a little like exploring a previously unmapped part of the world. You have no idea what you’ll find there, but unless you start wandering around you’ll never discover anything.

3. Aim for perfection

Perfection takes forever, and so students who are aiming for perfection never finish. Instead they spend years trying to make the thesis that little bit better, polishing every sentence until it gleams. Every researcher needs to accept that research involves making mistakes, often publicly. That’s the nature of the activity.

Don’t wait until your paper or thesis is perfect. Work through a few drafts, and then stop, recognizing that there are probably still some errors remaining.

4. Aim too high

Many students imagine they will write a thesis that will revolutionise the field and lead to wide acclaim and a brilliant academic career. Occasionally that does happen, but extremely rarely. A PhD is an apprenticeship in research, and like all apprenticeships, you are learning the craft, making mistakes, and you are unlikely to produce your best work at such an early stage in your research career.

It really doesn’t matter what your topic is provided you find it interesting and that you find something to say about it. Your PhD is a demonstration that you know how to do research, but your most important and high impact research will probably come later.

My own PhD research was on stochastic nonlinear differential equations and I haven’t touched them since. It showed I could do high level research, but I’d lost interest by the time I finished and I’ve moved onto other things. Few people ever cite the research that came out of my PhD, but it served its purpose.

5. Aim too low

My rule-of-thumb for an Australian PhD is about three to four pieces of publishable work. They don’t have to actually be published, but the examiners like to see enough material to make up three papers that would be acceptable in a reputable scholarly journal. Just writing 200 pages is not enough if the material is not sufficiently original or innovative to be publishable in a journal. Pointing out errors in everyone else’s work is usually not enough either, as most journals will expect you to have something to say yourself in addition to whatever critiques you make of previous work.

6. Follow every side issue

Just because you use a maximum likelihood method, doesn’t mean you have to read the entire likelihood literature. Of course you will learn something if you do, but that isn’t the point. The purpose of a PhD is not so that you can learn as much as you can about everything. A PhD is training in research, and researchers need to be able to publish their findings without having to be expert in every area that is somehow related to their chosen topic.

Of course, you do need to read as much of the relevant literature as possible. A key skill in research is learning what is relevant and what is not. Ask your supervisor if you are not sure.

7. Leave all the writing to the end

In some fields it seems to be standard practice to have a “writing up” phase after doing the research. Perhaps that works in experimental sciences, but it doesn’t work in the mathematical sciences. You haven’t a hope of remembering all the good ideas you had in first and second year if you don’t attempt to write them down until near the end of your third year.

I encourage all my students to start writing from the first week. In the first year, write a series of notes summarizing what you’ve learned and what research ideas you’ve had. It can be helpful to use these notes to show your supervisor what you’ve been up to each time you meet. In the second year, you should have figured out your specific topic and have a rough idea of the table of contents. So start writing the parts you can. You should be able to turn some of your first-year notes into sections of the relevant chapters. By the third year you are filling in the gaps, adding simulation results, tidying up proofs, etc.

Tags: phd, productivity, progress, supervision

Econometrics and R

31Aug 2010

2 Comments

Econometricians seem to be rather slow to adopt new methods and new technology (compared to other areas of statistics), but slowly the use of R is spreading. I’m now receiving requests for references showing how to use R in econometrics, and so I thought it might be helpful to post a few suggestions here.

	A useful on-line and free resource is “Econometrics in R” by Grant Farnsworth. It covers some common econometric methods including heteroskedasticity in regression, probit and logit models, tobit regression, and quantile regression. In the time series area, it covers ARIMA, ARFIMA, ARCH and GARCH models, as well as a few of the standard tests for unit roots and autocorrelation. It’s brief but it does provide code that will help people familiar with econometrics to get started using R.
	If you are prepared to pay, an excellent book is Kleiber and Zeilis’s Applied Econometrics with R. It covers similar ground to Farnsworth but in more detail. This is the book I usually recommend to anyone with an econometrics background who is wanting to get started with R. It would also be very suitable for someone studying econometrics at about upper undergraduate level. Achim Zeileis is a well-known expert in R programming, so you can be sure the code in this book is efficient and well-written.
	Another useful book is Pfaff’s Analysis of Integrated and Cointegrated Time Series with R which covers unit root tests, cointegration, VECM models, etc.
	Vinod’s Hands-On Intermediate Econometrics Using R contains a lot of examples and code-snippets which can be very helpful. Unfortunately, the examples do not always show the best practice in R coding.
	More detailed case studies using R are provided in Advances in Social Science Research Using R, edited by H.D. Vinod. Many of the case studies are from econometrics including an excellent chapter by Bruce McCullough on econometric computing.

There are of course dozens of books on R with a more statistical perspective, including several on time series. But I will leave them for another post.

Tags: R, references

Job advertisements

26Aug 2010

1 Comment

Employers often contact me asking how to find a good statistician, econometrician or forecaster for their organization. Students also ask me how to go about finding a job when they finish their degree. This post is for both groups, hopefully making it easier for them to pair up appropriately.

First, the mainstream media outlets are not usually good places to advertise. It seems that few people read printed newspapers anymore.

The general online job sites such as seek.com.au are ok, but job-seekers can find it hard to find the relevant openings because job titles are so varied. In the general area of statistics, a job can appear under the titles “statistician”, “analyst”, “data miner”, “data manager”, “financial engineer” and a few dozen other labels. Many employers don’t place the job in the best category, often because they don’t understand what skills are required to do the job. Nevertheless, if I was looking for a job, I would certainly set up some automated searches on these sites.

In statistics, there are well-established job websites that are the best places for both employers and potential employees to meet up.

Australia & New Zealand: www.statsci.org/jobs. This is a fantastic service from the Statistical Society of Australia and includes a lot of jobs, particularly those requiring higher degrees.
United States: amstat.org/jobweb. This is a similar service from the American Statistical Association for jobs in the USA.

Unfortunately, there is no similar service in the UK, and I do not know what is provided in other countries.

There is a list of econometric jobs sites at econometriclinks.

There are e-mail lists that are widely subscribed and often contain job postings.

Australia and New Zealand statistics: ANZstat (anything advertised here will automatically also appear on www.statsci.org/jobs.)
Australia and New Zealand econometrics: ANZEcmet
United Kingdom statistics: Allstat
United Kingdom econometrics: Econometric-research (for research jobs)

A lot more email lists are mentioned on econometriclinks, some of which may be appropriate for job advertisements.

If I’ve missed any good places to advertise jobs, please add them in the comments.

Tags: jobs

Benchmarks for forecasting

25Aug 2010

2 Comments

Every week I reject papers submitted to the International Journal of Forecasting because they present new methods without ever attempting to demonstrate that the new methods are better than existing methods. It is a policy of the journal that every new method must be compared to standard benchmarks and existing methods before the paper will even be considered for publication.

For univariate time series methods, it is not difficult. As a minimum, comparisons should be made against a naive method and a standard method such as an ARIMA model.

The naive method for non-seasonal data is based on a random walk — all forecasts are equal to the last observation. For seasonal data, the best naive method is to use the last observation from the same season. That is, for monthly data, forecasts for February are all equal to the last February observation.
Comparisons with ARIMA models used to be problematic because some authors did not have sufficient expertise to fit a good ARIMA model, and so comparisons were sometimes made, for example, against a non-seasonal AR model when the data were obviously seasonal. This should no longer be a problem as there are now good automatic ARIMA algorithms such as auto.arima() in the forecast package for R.

For multivariate time series, the same univariate benchmarks can be used.

For methods involving covariates, a standard linear regression can often provide a basic benchmark. Authors sometimes argue that linear regression is not appropriate for their data (e.g., because of non-linear relationships or correlations), but that is not the point. I don’t care if the linear regression is appropriate — I just want them to be able to show that their method provides better predictions than a standard and simple benchmark. If it can’t beat a simple standard regression, especially if it is inappropriate, there is not much point proceeding.

The best benchmarks are those that are already published. For example, new univariate time series methods can be compared with the M-competition or M3 competition data where there are already published evaluations on large numbers of observations. In this case, authors do not even have to implement the benchmarks themselves. All they have to do is use the same test sets and compare their MAPE or sMAPE values with those published for other methods.

Just beating the benchmarks is not, of itself, justification for publication, but it helps. It is also necessary to be able to describe your new method in enough detail and clarity that others could implement it. It is usually also necessary to show that the method works on more than one data set. It is relatively easy to find a method that outperforms the benchmarks on a single data set; but that is no reason to think it will be useful on other data sets. The M-competitions are useful as they provide a large set of data for comparisons. If a method does well on 1001 or 3003 time series, then I know it is not a fluke.

Similarly, not being able to beat the benchmarks does not, of itself, mean the paper is dead. It may be that the new method is not far behind the benchmarks but has other advantages. Or the new method may be particularly good in some circumstances or for a small subset of problems.

The job of the author is to carefully and persuasively present the case for their proposed method. As an editor, I am looking for authors to convince me of the value of their ideas. Papers proposing new forecasting methods must include comparisons with standard benchmarks, and should involve large scale empirical evaluations.

Tags: forecasting

Transforming data with zeros

13Aug 2010

1 Comment

I’m currently working with a hydrologist and he raised a question that occurs quite frequently with real data — what do you do when the data look like they need a log transformation, but there are zero values?

I asked the question on stats.stackexchange.com and received some useful suggestions. What follows is a summary based on these answers, my own experience, plus a few papers I discovered that deal with the topic. In general, the most appropriate course of action depends on the model and the context. Zeros can arise for several different reasons each of which may have to be treated differently.

Box-Cox (BC) transformations

There is a two-parameter version of the Box-Cox transformation that allows a shift before transformation:

$g(y;\lambda_{1}, \lambda_{2}) =<br /> \begin{cases}<br /> \frac {(y+\lambda_{2})^{\lambda_1} - 1} {\lambda_{1}} & \mbox{when } \lambda_{1} \neq 0 \\\ \log (y + \lambda_{2}) & \mbox{when } \lambda_{1} = 0<br /> \end{cases}.$

The usual Box-Cox transformation sets $\lambda_2=0$ . One common choice with the two-parameter version is $\lambda_1=0$ and $\lambda_2=1$ which has the neat property of mapping zero to zero. There is even an R function for this: log1p(). More generally, both parameters can be estimated. In R, the boxcox.fit() function in package geoR will fit the parameters.

Alternatively, when $\lambda_1=0$ , it has been suggested that $\lambda_2$ should be approximately one half of the smallest, non-zero value. Another suggestion is that $\lambda_2$ should be the square of the first quartile divided by the third quartile (Stahel, 2002).

I’ve used functions like this several times including in Hyndman & Grunwald (2000) where we used $\log(y+\lambda_2)$ applied to daily rainfall data.

One simple special case is the square root where $\lambda_2=0$ and $\lambda_1=0.5$ . This works fine with zeros (although not with negative values). However, often the square root is not a strong enough transformation to deal with the high levels of skewness seen in real data.

Inverse hyperbolic sine (IHS) transformation

An alternative transformation family was proposed by Johnson (1949) and is defined by

$f(y,\theta) = \text{sinh}^{-1}(\theta y)/\theta = \log(\theta y + (\theta^2y^2+1)^{1/2})/\theta,$

where $\theta>0$ . For any value of $\theta$ , zero maps to zero. There is also a two parameter version allowing a shift, just as with the two-parameter BC transformation. Burbidge, Magee and Robb (1988) also discuss the IHS transformation including estimation of $\theta$ .

The IHS transformation works with data defined on the whole real line including negative values and zeros. For large values of it behaves like a log transformation, regardless of the value of $\theta$ (except 0). As $\theta\rightarrow0$ , $f(y,\theta)\rightarrow y$ .

Mixed models

For continuous data, there can be a discrete spike at zero which can be associated with the sensitivity of the measurements. For example in wind energy, wind below 2m/s is often recorded as zero and the distribution of wind energy produced is continuous with a spike at zero.

With rainfall data, there is a spike at zero for a different reason — it didn’t rain. These are genuine zeros (rather than indetectably small values).

With insurance data, a similar phenomenon occurs — the distribution of claims is continuous with a large spike at zero.

A fourth example might be income data — zero if someone is not in paid work, but a continuous positive value otherwise.

In each of these cases, a mixture model is probably the most appropriate where part of the model determines the probability of a zero, and the other part of the model determines the distribution of the data when it is positive. We also used something like this in Hyndman and Grunwald (2000).

Tags: statistics

The tourism forecasting competition

9Aug 2010

2 Comments

Recently I wrote a paper entitled “The tourism forecasting competition” in which we (i.e., George Athanasopoulos, Haiyan Song, Doris Wu and I) compared various forecasting methods on a relatively large set of tourism-related time series. The paper has been accepted for publication in the International Journal of Forecasting. (When I submit a paper to the IJF it is always handled by another editor. In this case, Mike Clements handled the paper and it went through several revisions before it was finally accepted. Just to show the process is unbiased, I have had a paper rejected by the journal during the period I have been Editor-in-Chief.)

We are now opening up the competition to anyone who thinks they can do better than the best methods we implemented in the paper. Methods will be evaluated based on the smallest MASE (Mean Absolute Scaled Error) — see Hyndman & Koehler (2006) for details of this statistic.

To make it interesting, there is a prize. The overall winner will collect $AUD500 and will be invited to contribute a discussion paper to the International Journal of Forecasting describing their methodology and giving their results, provided either the monthly MASE results are better than 1.38, the quarterly results are better than 1.43 or the yearly results are better than 2.28. These thresholds are the best performing methods in the analysis of these data described in Athanasopoulos et al (2010). In other words, the winner has to beat the best results in this paper for at least one of the three sets of series. It will also be necessary that the winner be able to describe their method clearly, in sufficient detail to enable replication and in a form suitable for the International Journal of Forecasting. The paper would appear in the April 2011 issue of the IJF.

The competition is being hosted by the innovative folks at kaggle.com. Head over to kaggle.com/tourism1 to get the data and enter the competition.

The competition will be in two stages. Stage 1 involves only the annual data — 518 time series. You need to submit forecasts of the next four observations for each series before 20 September 2010. Stage 2 will involve the monthly and quarterly data and will begin after Stage 1 closes.

Good luck!

Tags: forecasting

Twenty rules for good graphics

6Aug 2010

8 Comments

One of the things I repeatedly include in referee reports, and in my responses to authors who have submitted papers to the International Journal of Forecasting, are comments designed to include the quality of the graphics. Recently someone asked on stats.stackexchange.com about best practices for producing plots. So I thought it might be helpful to collate some of the answers given there and add a few comments of my own taken from things I’ve written for authors.

The following “rules” are in no particular order.

Use vector graphics such as eps or pdf. These scale properly and do not look fuzzy when enlarged. Do not use jpeg, bmp or png files as these will look fuzzy when enlarged, or if saved at very high resolutions will be enormous files. Jpegs in particular are designed for photographs not statistical graphics.
Use readable fonts. For graphics I prefer sans-serif fonts such as Helvetica or Arial. Make sure the font size is readable after the figure is scaled to whatever size it will be printed.
Avoid cluttered legends. Where possible, add labels directly to the elements of the plot rather than use a legend at all. If this won’t work, then keep the legend from obscuring the plotted data, and make it small and neat.
If you must use a legend, move it inside the plot, in a blank area.
No dark shaded backgrounds. Excel always adds a nasty dark gray background by default, and I’m always asking authors to remove it. Graphics print much better with a white background. The ggplot for R also uses a gray background (although it is lighter than the Excel default). I don’t mind the ggplot version so much as it is used effectively with white grid lines. Nevertheless, even the light gray background doesn’t lend itself to printing/photocopying. White is better.
Avoid dark, dominating grid lines (such as those produced in Excel by default). Grid lines can be useful, but they should be in the background (light gray on white or white on light gray).
Keep the axis limits sensible. You don’t have to include a zero (even if Excel wants you to). The defaults in R work well. The basic idea is to avoid lots of white space around the plotted data.
Make sure the axes are scaled properly. Another Excel problem is that the horizontal axis is sometimes treated categorically instead of numerically. If you are plotting a continuous numerical variable, then the horizontal axis should be properly scaled for the numerical variable.
Do not forget to specify units.
Tick intervals should be at nice round numbers.
Axes should be properly labelled.
Use linewidths big enough to read. 1pt lines tend to disappear if plots are shrunk.
Avoid overlapping text on plotting characters or lines.
Follow Tufte’s principles by removing chart junk and keeping a high data-ink ratio.
Plots should be self-explanatory, so included detailed captions.
Use a sensible aspect ratio. I think width:height of about 1.6 works well for most plots.
Prepare graphics in the final aspect ratio to be used in the publication. Distorted fonts look awful.
Use points not lines if element order is not relevant.
When preparing plots that are meant to be compared, use the same scale for all of them. Even better, combine plots into a single graph if they are related.
Avoid pie-charts. Especially 3d pie-charts. Especially 3d pie-charts with exploding wedges. I promise all my students an instant fail if I ever see anything so appalling.

The classic books on graphics are:

These are both highly recommended. (If you can’t see the books above, turn off your ad-blocker.)

Tags: graphics, journals, R

Statistical Analysis StackExchange site now available

27Jul 2010

4 Comments

The Q&A site for statistical analysis, data mining, data visualization, and everything else to do with data analysis has finally been launched. Please head over to

stats.StackExchange.com

and start asking and answering questions.

Also, spread the word to everyone else who may be interested — work colleagues, students, etc. The more people who use the site, the better it will be. There are already 170 questions, 513 answers and 387 users.

Eventually the site will move to a different domain name and have its own logo, etc. For now it is in “public beta” which means that it is fully functional, but we are still working out some of the details (such as what it will be called, who will be the moderators, etc.).

R questions are allowed on this new site as well as on the original StackOverflow.com. We are still figuring out how to avoid the problem of having answers on two sites. For now, more statistical questions should be directed to stats.StackExchange.com and more programming-oriented questions should go to StackOverflow.com.

Tags: computing, R, StackExchange, statistics

More StackExchange sites

17Jul 2010

No Comments

The StackExchange site on Statistical Analysis is about to go into private beta testing. This is your last chance to commit if you want to be part of the private beta testing. Don’t worry if you miss out — it will only be a week before it is then open to the public.

There is also a StackExchange site proposal for TeX, LaTeX and friends. Presumably that means that most of the LaTeX questions on StackOverflow will then move to this new site. It still needs a couple of hundred more people to commit before it can be launched, so if you are interested in LaTeX, please commit to being part of it.

Another site proposal that may be of interest to readers of this blog is the one on English language usage.

A few proposals are already open to the public for beta testing. One that I’ve been using a little is Web Apps which is useful for questions on Gmail, Google reader, WordPress, etc.

Tags: computing, LaTeX, R, StackExchange, writing

Rob J Hyndman: Research tips

Why God never received tenure