Fore­cast­ing com­pe­ti­tions are a great way to test new meth­ods and obtain a real­is­tic eval­u­a­tion of how good they are. So I’m delighted that the IEEE is orga­niz­ing an energy fore­cast­ing com­pe­ti­tion as out­lined by Tao Hong below.

Fore­cast­ing Ace is look­ing for par­tic­i­pants to develop improved meth­ods for pre­dict­ing future events and out­comes. Their goal is to develop meth­ods for aggre­gat­ing many indi­vid­ual judg­ments in a man­ner that yields more accu­rate pre­dic­tions than any one per­son or small group alone could pro­vide. Poten­tial appli­ca­tions of the sys­tem include fore­cast­ing eco­nomic con­di­tions, polit­i­cal changes, tech­no­log­i­cal devel­op­ment and med­ical breakthroughs.

And the win­ners are … Jeremy Howard and Lee C Baker. (See my ear­lier post for infor­ma­tion about the com­pe­ti­tion.) Jeremy describes his approach to sea­sonal time series in a blog post on Kag​gle​.com. Lee described his approach to annual time series in an ear­lier post. A few lessons that come out of this: For data from a sin­gle indus­try, using a global trend (i.e., esti­mated across all series) can be use­ful. Com­bin­ing fore­casts is a good idea. (This les­son seems to be re-​​​​learned in every fore­cast­ing com­pe­ti­tion!) The MASE can be very sen­si­tive to a few series, and to opti­mize MASE it is worth con­cen­trat­ing on these. (This is actu­ally not a good mes­sage for fore­cast­ing over­all, as we want good fore­casts for all series. Maybe we need to find a met­ric with sim­i­lar prop­er­ties to MASE but with a less skewed dis­tri­b­u­tion.) Out­lier removal before fore­cast­ing can be effec­tive. (This is an inter­est­ing result as out­lier removal algo­rithms used in the M3 com­pe­ti­tion did not help fore­cast accu­racy.) Jeremy and Lee receive 500 will restore vision to 20 peo­ple. They will also write up their meth­ods in more detail for the Inter­na­tional Jour­nal

(More)…

The first stage of the tourism fore­cast­ing com­pe­ti­tion on kag­gle has fin­ished. This stage involved fore­cast­ing 518 annual time series. Twenty one teams beat our Theta method bench­mark which is a great result, and well beyond our expec­ta­tions. Con­grat­u­la­tions to Lee Baker for win­ning stage one. I am yet to learn what meth­ods the top teams were using, but we hope to write up a paper for the IJF describ­ing the results. Of course, the win­ning team (over­all) gets to write their own dis­cus­sion paper for the IJF. Stage 2 of the com­pe­ti­tion is now open and involves fore­cast­ing 366 monthly time series and 427 quar­terly time series. In this case, the best result in our paper for the monthly data was the auto­matic ARIMA algo­rithm (Hyn­d­man & Khan­dakar, 2008) with a MASE of 1.38. For quar­terly data, the ETS(A,Ad,A) model per­formed slightly bet­ter than our ARIMA algo­rithm with a MASE of 1.43. Let’s see how much bet­ter every­one else can do! Head over to kag­gle and get the data. Entries close on 31 October 2010.

Recently I wrote a paper enti­tled “The tourism fore­cast­ing com­pe­ti­tion” in which we (i.e., George Athana­sopou­los, Haiyan Song, Doris Wu and I) com­pared var­i­ous fore­cast­ing meth­ods on a rel­a­tively large set of tourism-​​​​related time series. The paper has been accepted for pub­li­ca­tion in the Inter­na­tional Jour­nal of Fore­cast­ing. (When I sub­mit a paper to the IJF it is always han­dled by another edi­tor. In this case, Mike Clements han­dled the paper and it went through sev­eral revi­sions before it was finally accepted. Just to show the process is unbi­ased, I have had a paper rejected by the jour­nal dur­ing the period I have been Editor-​​​​in-​​​​Chief.) We are now open­ing up the com­pe­ti­tion to any­one who thinks they can do bet­ter than the best meth­ods we imple­mented in the paper. Meth­ods will be eval­u­ated based on the small­est MASE (Mean Absolute Scaled Error) — see Hyn­d­man & Koehler (2006) for details of this sta­tis­tic. To make it inter­est­ing, there is a prize. The over­all win­ner will col­lect $AUD500 and will be invited to con­tribute a dis­cus­sion paper to the Inter­na­tional Jour­nal of Fore­cast­ing describ­ing their method­ol­ogy and giv­ing their results, pro­vided either the monthly MASE results are bet­ter than 1.38, the quar­terly results are bet­ter than 1.43 or the yearly results are

(More)…