mxnet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vandana Kannan <vandana...@gmail.com>
Subject Re: Testing examples in nightly build
Date Thu, 16 Aug 2018 16:26:59 GMT
@Marco: Most of the examples are standalone applications and not notebooks. We could start
off by enabling tests for examples that are either notebooks or those that can be executed
with command line options.

Some of the problems that we may come across are with examples that load large datasets (we
could probably use a smaller dataset, but it depends on the example), or examples that have
a long training time (we could probably execute them for 1 epoch). Straight Dope nightly tests
cap the execution time to 10 minutes per notebook. We could do the same here wherever possible.


@Kellen: Good idea to include flake8/pylint checks for examples in nightly builds - we would
need to fix the backlog (https://github.com/apache/incubator-mxnet/issues/12205) before enabling.
Yes, any contribution from the community on any of these TODO items, would be great.

- Vandana

On 2018/08/16 06:40:56, kellen sunderland <kellen.sunderland@gmail.com> wrote: 
> I think it would be very beneficial to start fleshing out the nighties.
> They provide a lot of value at a relatively small cost.  Any contributions
> from the community would be appreciated.
> 
> Things I could see being beneficial:
> * Long running tests
> * In depth flake8/pylint linters
> * cudamemcheck builds
> * Asan builds.
> 
> On Thu, Aug 16, 2018, 12:24 AM Marco de Abreu
> <marco.g.abreu@googlemail.com.invalid> wrote:
> 
> > Hello,
> >
> > I think this is a great idea! Thanks a lot for fixing all the problems in
> > our examples.
> >
> > Do you see any problems that could come up if we just run them one by one?
> > We already got a pipeline that allows us to verify jupyter notebooks, but
> > afaik the examples are standalone files with a main function, right?
> >
> > Best regards,
> > Marco
> >
> > Vandana Kannan <vandana268@gmail.com> schrieb am Mi., 15. Aug. 2018,
> > 23:44:
> >
> > > Hi All,
> > >
> > > Recently we saw that there were quite a few Pylint undefined-variable
> > > errors in MXNet code, a majority of them in the example folder. These
> > > errors highlight that there are paths in the examples that are broken and
> > > these have not been caught during testing.
> > >
> > > These errors are later reported as issues by users. For example,
> > > https://github.com/apache/incubator-mxnet/issues/11278.
> > >
> > > It would be a good idea to include tests for all examples in the nightly
> > > builds (Similar to
> > > incubator-mxnet/tests/nightly/test_image_classification.sh). Side note:
> > We
> > > might have to see what to do about examples that include special setup
> > > instructions or take too long to execute.
> > >
> > > Any thoughts or suggestions on writing these tests, feasibility or
> > > previous attempts at including these tests, would be helpful.
> > >
> > > Thanks,
> > > Vandana
> > >
> >
> 


Mime
View raw message