systemml-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthias Boehm <>
Subject Re: Proposal to add 'accuracy test suite' before 1.0 release
Date Fri, 17 Feb 2017 19:53:48 GMT
Yes, this has been discussed a couple of times now, most recently in 
SYSTEMML-546. It takes quite some effort though to create a 
sophisticated algorithm-level test suite as done for GLM. So by all 
means, please, go ahead and add these tests.

However, I would not impose any constraints on the contribution of new 
algorithms in that regard, or similarly on tests with simplified 
algorithms because it would raise the bar to high.


On 2/17/2017 10:48 AM, Niketan Pansare wrote:
> Hi all,
> We currently test the correctness of individual runtime operators using our
> integration tests but not the "released" algorithms. To be fair, we do test
> a subset of "simplified" algorithms on synthetic datasets and compare the
> accuracy with R. Also, we are testing subset of released algorithms using
> our Python tests, but it's intended purpose is to only test the integration
> of the APIs:
> Simplified algorithms:
> Released algorithms:
> Python tests:
> Though the released algorithm is tested when it is initially introduced,
> other artifacts (spark versions, API changes, engine improvements, etc)
> could cause them to return incorrect results over a period of time.
> Therefore, similar to our performance test suite (
> I propose we create another test suite ("accuracy test suite" for lack of a
> better term) that compares the accuracy (or some other metric) of our
> released algorithms on standard datasets. Making it a requirement to add
> tests to accuracy test suite when adding the new algorithm will greatly
> improve the production-readiness of SystemML as well as serve as a usage
> guide too. This implies we run both the performance as well as accuracy
> test suite before our release. Alternative is to replace simplified
> algorithms with our released algorithms.
> Advantages of accuracy test suite approach:
> 1. No increase the running time of integration tests on Jenkins.
> 2. Accuracy test suite could use much larger datasets.
> 3. Accuracy test suite could include algorithms that take longer to
> converge (for example: Deep Learning algorithms).
> Advantage of replacing simplified algorithms:
> 1. No commit breaks any of the existing algorithms.
> Thanks,
> Niketan Pansare
> IBM Almaden Research Center
> E-mail: npansar At

View raw message