mxnet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anirudh Acharya <anirudhk...@gmail.com>
Subject Re: Nightly/Weekly tests for examples
Date Mon, 12 Nov 2018 18:38:20 GMT
Hi Ankit,

I have a few concerns about testing examples. Before writing tests for
examples,

   - you will need to first decide what constitutes a test for an example,
   because examples are not API calls, which will have return statements and
   the test can just call the API and assert for certain values. Just testing
   if an example is a compilable python script will not add much value in my
   opinion.
   - And testing for example output and results will require a re-write of
   many of the examples, because many of them currently just have print
   statements as outputs and does not return any value as such. I am not sure
   if it is worth the dev-effort.
   - the current set of examples in the mxnet repo are very diverse - some
   are written as python notebooks, some are just python scripts with paper
   implementations, and some are just illustrations of certain mxnet features.
   I am curious to know how you will write tests for these things.


Looking forward to seeing the design of this test bed/framework.


Thanks
Anirudh Acharya

On Fri, Nov 9, 2018 at 2:39 PM Marco de Abreu
<marco.g.abreu@googlemail.com.invalid> wrote:

> Hello Ankit,
>
> that's a great idea! Using the tutorial tests as reference is a great
> starting point. If you are interested, please don't hesitate to attend the
> Berlin user group in case you would like to discuss your first thoughts
> in-person before drafting a design.
>
> -Marco
>
>
> Am Fr., 9. Nov. 2018, 23:23 hat khedia.ankit@gmail.com <
> khedia.ankit@gmail.com> geschrieben:
>
> > Hi MXNet community,
> >
> > Recently, I and a few other contributors focussed on fixing examples in
> > our repository which were not working out of the box as expected.
> > https://github.com/apache/incubator-mxnet/issues/12800
> > https://github.com/apache/incubator-mxnet/issues/11895
> > https://github.com/apache/incubator-mxnet/pull/13196
> >
> > Some of the examples failed after API changes and remained uncaught until
> > a user reported the issue. While the community is actively working on
> > fixing it, it might re-occur after few days if we don’t have a proper
> > mechanism to catch regressions.
> >
> > So, I would like to propose to enable nightly/weekly tests for the
> > examples similar to what we have for tutorials to catch any such
> > regressions. The test could check only basic functionalities/working of
> the
> > examples. It can run small examples completely whereas it can run long
> > training examples for only few epochs.
> >
> > Any thoughts from the community? Any other suggestions for fixing the
> same?
> >
> > Regards,
> > Ankit Khedia
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message