systemml-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Niketan Pansare" <npan...@us.ibm.com>
Subject Proposal to add 'accuracy test suite' before 1.0 release
Date Fri, 17 Feb 2017 18:48:29 GMT


Hi all,

We currently test the correctness of individual runtime operators using our
integration tests but not the "released" algorithms. To be fair, we do test
a subset of "simplified" algorithms on synthetic datasets and compare the
accuracy with R. Also, we are testing subset of released algorithms using
our Python tests, but it's intended purpose is to only test the integration
of the APIs:
Simplified algorithms:
https://github.com/apache/incubator-systemml/tree/master/src/test/scripts/applications
Released algorithms:
https://github.com/apache/incubator-systemml/tree/master/scripts/algorithms
Python tests:
https://github.com/apache/incubator-systemml/tree/master/src/main/python/tests

Though the released algorithm is tested when it is initially introduced,
other artifacts (spark versions, API changes, engine improvements, etc)
could cause them to return incorrect results over a period of time.
Therefore, similar to our performance test suite (
https://github.com/apache/incubator-systemml/tree/master/scripts/perftest),
I propose we create another test suite ("accuracy test suite" for lack of a
better term) that compares the accuracy (or some other metric) of our
released algorithms on standard datasets. Making it a requirement to add
tests to accuracy test suite when adding the new algorithm will greatly
improve the production-readiness of SystemML as well as serve as a usage
guide too. This implies we run both the performance as well as accuracy
test suite before our release. Alternative is to replace simplified
algorithms with our released algorithms.

Advantages of accuracy test suite approach:
1. No increase the running time of integration tests on Jenkins.
2. Accuracy test suite could use much larger datasets.
3. Accuracy test suite could include algorithms that take longer to
converge (for example: Deep Learning algorithms).

Advantage of replacing simplified algorithms:
1. No commit breaks any of the existing algorithms.

Thanks,

Niketan Pansare
IBM Almaden Research Center
E-mail: npansar At us.ibm.com
http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message