Prediction intervals too narrow

Almost all prediction intervals from time series models are too narrow. This is a well-known phenomenon and arises because they do not account for all sources of uncertainty. In my 2002 IJF paper, we measured the size of the problem by computing the actual coverage percentage of the prediction intervals on hold-out samples. We found that for ETS models, nominal 95% intervals may only provide coverage between 71% and 87%. The difference is due to missing sources of uncertainty.

There are at least four sources of uncertainty in forecasting using time series models:

  1. The random error term;
  2. The parameter estimates;
  3. The choice of model for the historical data;
  4. The continuation of the historical data generating process into the future.

When we produce prediction intervals for time series models, we generally only take into account the first of these sources of uncertainty. It would be possible to account for 2 and 3 using simulations, but that is almost never done because it would take too much time to compute. As computing speeds increase, it might become a viable approach in the future.

Even if we ignore the model uncertainty and the DGP uncertainty (sources 3 and 4), and just try to allow for parameter uncertainty as well as the random error term (sources 1 and 2), there are no closed form solutions apart from some simple special cases.

One such special case is an ARIMA(0,1,0) model with drift, which can be written as
$$
y_t = y_{t-1} + c + e_t,
$$
where $e_t$ is a white noise process. In this case, it is easy to compute the uncertainty associated with the estimate of $c$, and then allow for it in the forecasts.

This model can be fitted using either the Arima function or the rwf function from the forecast package for R. If the Arima function is used, the uncertainty in $c$ is ignored, but if the rwf function is used, the uncertainty in $c$ is included in the prediction intervals. The difference can be seen in the following simulated example.

library(forecast)
 
set.seed(22)
x <-ts(cumsum(rnorm(50, -2.5, 4)))
 
RWD.x <- rwf(x,  h=40, drift=TRUE, level=95)
ARIMA.x <- Arima(x, c(0,1,0), include.drift=TRUE)
 
plot(forecast(ARIMA.x, h=40, level=95))
lines(RWD.x$lower, lty=2)
lines(RWD.x$upper, lty=2)

Rplot


Related Posts:


  • Morgan

    Thanks for the interesting article. Just one question though, wouldn’t yt = yt-1 + c + et be an ARIMA(1,0,0) with drift?

  • kadee

    Thanks for the many extremely useful online information with regard to forecasting, most of it originating from your work!

    I’m using Python statsmodels, but I assume the same question holds for the R forecast package:

    Looking at the forecast interval of ARIMA(0,0,0)+c (white noise with offset), I see that the prediction interval corresponds with the prediction interval of a normal distribution when assuming the mean and variance are known ànd equal to the sample mean and mle variance.

    I am a bit puzzled by this result, as with forecasting, mean and variance are generally unknown. Is my conclusion correct that one gets more accurate results (ànd wider prediction intervals) by calculating the prediction interval by multiplying the (mle) variance by (n+1)/(n-1) (see e.g. https://en.wikipedia.org/wiki/Prediction_interval ) and then using the student’s T distribution percentiles? Can one apply the same correction to other, more complex forecast models?

    • Adjusting the variance by multiplying by (n+1)/(n-1) is only valid in the very simple example discussed in the wiki article. Besides, it makes almost no difference.

      Using a t-distribution would be better, but again it doesn’t make much difference in practice.

      The prediction intervals are too narrow for several reasons, as explained in this article. The t-distribution fix only corrects for the unknown variance, not for the parameter uncertainty, model uncertainty, etc.