systemml-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From dusenberr...@gmail.com
Subject Re: Proposal to add 'accuracy test suite' before 1.0 release
Date Sat, 18 Feb 2017 01:28:15 GMT
+1 for testing our actual (vs simplified test version) scripts against some metric of choice.
 This will allow us to (1) ensure that each script does not have a showstopper bug (engine
bug), and (2) that this script is still producing a reasonable mathematical result (math bug).

-Mike

--

Mike Dusenberry
GitHub: github.com/dusenberrymw
LinkedIn: linkedin.com/in/mikedusenberry

Sent from my iPhone.


> On Feb 17, 2017, at 4:17 PM, Niketan Pansare <npansar@us.ibm.com> wrote:
> 
> For now, I have updated our python mllearn tests to compare the prediction of our algorithm
to that of scikit-learn: https://github.com/apache/incubator-systemml/blob/master/src/main/python/tests/test_mllearn_numpy.py#L81
> 
> The test now uses scikit-learn predictions as the baseline and computes the scores (accuracy
score for classifiers and r2 score for regressors). If the score is greater than 95%, the
test pass. Though using this approach, we do not measure the generalization capability of
our algorithm, we at least ensure that our algorithm performs no worse than scikit-learn under
default setting. We can make the testing even more rigorous later. The next step would be
to enable these python tests through jenkins.
> 
> Thanks,
> 
> Niketan Pansare
> IBM Almaden Research Center
> E-mail: npansar At us.ibm.com
> http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar
> 
> Matthias Boehm ---02/17/2017 11:54:02 AM---Yes, this has been discussed a couple of times
now, most recently in SYSTEMML-546. It takes quite s
> 
> From: Matthias Boehm <mboehm7@googlemail.com>
> To: dev@systemml.incubator.apache.org
> Date: 02/17/2017 11:54 AM
> Subject: Re: Proposal to add 'accuracy test suite' before 1.0 release
> 
> 
> 
> 
> Yes, this has been discussed a couple of times now, most recently in 
> SYSTEMML-546. It takes quite some effort though to create a 
> sophisticated algorithm-level test suite as done for GLM. So by all 
> means, please, go ahead and add these tests.
> 
> However, I would not impose any constraints on the contribution of new 
> algorithms in that regard, or similarly on tests with simplified 
> algorithms because it would raise the bar to high.
> 
> Regards,
> Matthias
> 
> 
> On 2/17/2017 10:48 AM, Niketan Pansare wrote:
> >
> >
> > Hi all,
> >
> > We currently test the correctness of individual runtime operators using our
> > integration tests but not the "released" algorithms. To be fair, we do test
> > a subset of "simplified" algorithms on synthetic datasets and compare the
> > accuracy with R. Also, we are testing subset of released algorithms using
> > our Python tests, but it's intended purpose is to only test the integration
> > of the APIs:
> > Simplified algorithms:
> > https://github.com/apache/incubator-systemml/tree/master/src/test/scripts/applications
> > Released algorithms:
> > https://github.com/apache/incubator-systemml/tree/master/scripts/algorithms
> > Python tests:
> > https://github.com/apache/incubator-systemml/tree/master/src/main/python/tests
> >
> > Though the released algorithm is tested when it is initially introduced,
> > other artifacts (spark versions, API changes, engine improvements, etc)
> > could cause them to return incorrect results over a period of time.
> > Therefore, similar to our performance test suite (
> > https://github.com/apache/incubator-systemml/tree/master/scripts/perftest),
> > I propose we create another test suite ("accuracy test suite" for lack of a
> > better term) that compares the accuracy (or some other metric) of our
> > released algorithms on standard datasets. Making it a requirement to add
> > tests to accuracy test suite when adding the new algorithm will greatly
> > improve the production-readiness of SystemML as well as serve as a usage
> > guide too. This implies we run both the performance as well as accuracy
> > test suite before our release. Alternative is to replace simplified
> > algorithms with our released algorithms.
> >
> > Advantages of accuracy test suite approach:
> > 1. No increase the running time of integration tests on Jenkins.
> > 2. Accuracy test suite could use much larger datasets.
> > 3. Accuracy test suite could include algorithms that take longer to
> > converge (for example: Deep Learning algorithms).
> >
> > Advantage of replacing simplified algorithms:
> > 1. No commit breaks any of the existing algorithms.
> >
> > Thanks,
> >
> > Niketan Pansare
> > IBM Almaden Research Center
> > E-mail: npansar At us.ibm.com
> > http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar
> >
> 
> 
> 
> 

Mime
  • Unnamed multipart/alternative (inline, 7-Bit, 0 bytes)
View raw message