WAPE: Weighted Absolute Percentage Error
The WAPE was introduced by Kolassa & Schütz (2007) who called it the MAD/Mean ratio. It is defined as \text{WAPE} = \frac{\sum_{t} |y_t - \hat{y}_t|}{\sum_{t} |y_t|} where
- y_t is the actual value at time t,
- \hat{y}_t is the forecast value at time t
You can think of it as a weighted percentage error by writing it as \text{WAPE} = \sum_{t} w_t \frac{|y_t - \hat{y}_t|}{|y_t|} where the weights are given by w_t = \frac{|y_t|}{\sum_{t} |y_t| }
It can also be considered a relative MAE where the comparison method has all forecasts equal to zero. In fact, 1-\text{WAPE} would give the skill score relative to the forecast of zero for all t.
Or you could think of it as like a MASE (Hyndman & Koehler, 2006) but with scaling based on the sum of absolute values on the test set rather than the sum of absolute differences on the training set.
This has some obvious advantages over the MAPE:
- The MAPE is undefined when any actual value in the test set is zero. The WAPE is defined even when some actuals are zero. It is only undefined when all the actuals used in the denominator are zero.
- Optimising the MAPE does not lead to a sensible point forecasts (Gneiting, 2011), but optimising the WAPE will lead to the median forecast.
However, I think there are a couple of problems that do not seem to have been widely recognized.
- The resulting estimate is only consistent when the time series is stationary. So it should not be used with data that has trends, or seasonality, or heteroscedasticity.
- It is quite possible to have all actuals in the test set equal to zero, especially with intermittent demand time series and small test sets. Then the denominator is zero, and the WAPE is undefined.
For these reasons, I think the Mean Absolute Scaled Error (MASE) (Hyndman & Koehler, 2006) is a better choice than the WAPE. The MASE is defined as \text{MASE} = \frac{\frac{1}{n}\sum_{t} |y_t - \hat{y}_t|}{\frac{1}{T-m}\sum_{t=m+1}^T |y_t - y_{t-m}|} where m=1 for non-seasonal data, and m is the seasonal period for seasonal data, and the sum in the numerator is over the test set (of size n), and the sum in the denominator is over the training set (of size T). Because the denominator is defined on the training data, not the test data, it avoids the above problems with the WAPE:
- It is a consistent estimator provided the series is difference stationary, which is a much weaker condition than stationarity.
- The training data is also usually much longer than the test data, so it is much less likely to contain only zeros. In fact, if the training data did contain only zeros, then the obvious forecasts would all be zeros too.
I don’t want to suggest there are no problems with the MASE. Kolassa & Schütz (2007) point out one potential drawback of MASE — when there are structural breaks or outliers in the training data. Also, like the WAPE, optimising the MASE will lead to the median forecast, which is probably not what you want, especially when you have intermittent demand time series.
These days, if I want a scale-free accuracy measure, I prefer the Root Mean Squared Scaled Error (RMSSE): \text{RMSSE} = \sqrt{\frac{\frac{1}{n}\sum_{t} (y_t - \hat{y}_t)^2}{\frac{1}{T-m}\sum_{t=m+1}^T (y_t - y_{t-m})^2}}, where again the sum in the numerator is over the test set, and the sum in the denominator is over the training set. This has all the advantages (and most disadvantages) of the MASE, but optimising it leads to the mean forecast rather than the median. It also aligns better with how models are estimated. Almost all models are estimated by minimising the sum of squared errors, so it makes sense to evaluate them using squared errors as well.
Point forecast reconciliation also works on means rather than medians, and is optimised using least squares, so it is more natural to evaluate using a squared error measure.
Update on consistency and stationarity
The comments below suggest that I need to explain the consistency issue in more detail. Imagine you are forecasting a time series that is a simple linear trend with some noise. So y_t = a + bt + \varepsilon_t, where \varepsilon_t is Gaussian white noise with mean 0 and variance \sigma^2. As a good forecaster, you would fit a linear trend to this series, so your forecasts would be \hat{y}_t = \hat{a} + \hat{b}t. To keep it simple, let’s ignore the estimation error in this model and set \hat{a} = a and \hat{b} = b.
Now let’s generate forecasts for this series over a test set of n observations (either the next n time periods, or over a cross-validated set). If you use the RMSE to evaluate these forecasts, you obtain \text{RMSE} = \sqrt{\frac{1}{n}\sum_{t} (y_{t} - \hat{y}_{t})^2} = \sqrt{\frac{1}{n} \sum_{t} \varepsilon_{t}^2}, which is an estimate of the variance of the noise. As more observations are included in the calculation, the RMSE will converge to \sigma. That makes it “consistent” in a statistical sense. Practically, it also means that the expected RMSE does not depend on which observations you use to evaluate the forecasts — the RMSE values will be similar regardless of the test set used in the calculation. That is, if you evaluate the forecasts over the next week, the RMSE will be similar to the RMSE calculated on a cross-validated set.
Suppose, instead, we use the RMSSE to evaluate the forecasts. The RMSSE is equal to the RMSE divided by \sqrt{\frac{1}{T-1}\sum_{t=2}^T (y_{t} - y_{t-1})^2} = \sqrt{\frac{1}{T-1}\sum_{t=2}^T (b + \varepsilon_t - \varepsilon_{t-1})^2} So the RMSSE is still consistent (it converges to \sigma/\sqrt{b^2 + 2\sigma^2}), and the expected RMSSE does not depend on which observations you use to evaluate the forecasts.
Now, suppose we use the WAPE to evaluate these forecasts. Then we obtain \text{WAPE} = \frac{\sum_{t} |y_t - \hat{y}_t|}{\sum_{t} |y_t|} = \frac{\frac1n \sum_{t} |\varepsilon_t|}{\frac1n \sum_{t} |a + bt + \varepsilon_t|}, where both sums are over the test set. As n increases, the numerator will converge to the mean absolute value of the noise, approximately 0.798 \sigma. But the denominator won’t converge at all as it depends on the trend in the series. So the WAPE is not a consistent estimator of anything.
To see the effect of this, imagine the time series is daily sales, and you produce forecasts each week for the next 7 observations, and evaluate them using the WAPE. Over time, your WAPE values get smaller and smaller, because the denominator is growing larger and larger as the trend accumulates. So you have the illusion of improving accuracy, but your forecasts are really no more accurate than they were at the start.
While I have used a linear trend as the example here, similar problems can occur whenever the underlying time series is not stationary.