All participants have walked a long way to the final stage, and experienced the dramatic codalab crash. Taking a more in-depth review of the facts, we feel, still, insecure about the fairness of the Rule.
1. Experienced participants will take advantage. The organizer does not state clearly or at least imply the potential risks (discrepancy of data distribution, data size, time and memory requirements) of Check Phase in the rule description. The experienced, or to put another way, teams more familiar with the organizer are more probable to pass the Check Phase. Here are some facts related:
1.1) After the website crash We run through our code in a 8G docker environment without problem, making sure Memory Consumption consistently behind baseline, leaving 50% time buffer, but still failed on-line.
1.2) For Logic error, it is easy to construct a data and let baseline solution fail (For example, let t_01 be the linking key of some relation table, which represents many cases in the real-world).
1.3) We rank 2nd in feedback phase, ranking score distributions between feedback phase and check phase are similar.
2. The crash of codalab and rerun could bring unfairness. Consider the following two cases:
2.1) Teams that resubmitted and got feedback (Pass, Fail) before the crash actually has more info than the one who failed to resubmitted or resubmitted but did not get feedback.
2.2) Among Teams that resubmitted and got clear feedback Fail, the one previously notified Memory/Time error know for sure that it must be the same error type as long as they did not modify the logic; however, the one previously notified Logic Error has no idea about the error type, they has less information.
In the end, we have two suggestions:
1. Disclose the 10 datasets' AUC of baseline and all teams .
2. The rule had better take fairness into consideration more thoroughly.
Dear Deep_wisdom team,
Many thanks for your participantion at KDD Cup 2019 AutoML competition! Conguratulations on your amazing position at Feedback phase!
AutoML competition is different from regular competition. As we stated in the competition overview "- How to make the solution more generic, i.e., how to make it applicable for unseen tasks?", this is one of the key challenges of AutoML competition. We are also trying to remove the bug fixing chance in the coming AutoML competitions as we did in NeurIPS 2018. As you can see from our competition history, we removed one more chance for bug fixing at this competition. The rationale is that: generalization ability is the most important thing for machine learning solutions. When the AutoML solutions are deployed in real-world applications, you can not go to your users to fix it every time (it is impossible for Google Cloud AutoML to fix its code for every dataset).
There are differences between participants, this is the meaning of competitions. We are trying to improve the state of the art by gathering the most clever brains together. We are honored to see the amazing improvment over the baseline method you've made. We are willing to deliver awards to every participant, because you all have really done very excellent jobs. But there's must be a champion because this is a competition.
Your team got a really amazing position at Feedback phase! That's really impressive! We were looking forward to seeing you at the top position in AutoML phase. But unfortunately your submission was failed caused by logical error. We really thank your team very much for your great contribution on excellent AutoML solutions. But there are rules for competitions.
Thank your very much for your meaningful suggestions to us!
For your first suggestion, instead of releasing all the scores of each dataset, we will make this competition endless for at least two years, meaning that you could resubmit after the competition to test your improved AutoML solutions. We will start this after KDD Cup Day. Welcome to the post-competition challenge.
For your second suggestion, we are always trying to make the competition fair for everyone. At least, we should guarrantee that the rules are the same for all the participants. If you have any contructive suggestion, please feel free to contact us. Indeed, we need your suggestions, that's very important for us to make the competitions better.
Many thanks again for your participantion again. Really sorry for your failure at the final phase, it's really an unlucky thing. Wish you could keep your interest on AutoML research, because it's really important for applications. We really need great guys like you to improve state of the art. Hope to see you at the coming AutoX competitions.
Best,
Wei-Wei