Dear team tsinghua_wcy,
Thank you for sharing some ideas to enable more robust code validation for the data challenge. The data challenge committee had considered these type of tools before releasing the data challenge this year, however, there were a few situations we wanted to avoid. For example:
1. Terms and Conditions (T&Cs) for Open Source applications and services typically allow general use licenses (including forking) for code that is shared publicly (reference: GitHub Terms of Service D.4. – D.7. which apply to CodaLab). This is often acceptable in academia, but not for many industry participants due to intellectual property policies. Note: This year’s data challenge was split 60% academia and 40% industry so this was a significant consideration.
2. Additionally, many available tools of this type assume models will be written exclusively in Python. While Python is predominant, it is not an exclusive method we’d like to impose for the data challenge.