mxnet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Khedia, Ankit" <khe...@amazon.com.INVALID>
Subject Re: Nightly/Weekly tests for examples
Date Tue, 13 Nov 2018 19:25:58 GMT

We can start small consolidating few examples which are there in both tutorials/examples folder.
One of such example is GradCAM implementation:
In tutorials - https://github.com/apache/incubator-mxnet/blob/master/docs/tutorials/vision/cnn_visualization.md
In examples- https://github.com/apache/incubator-mxnet/blob/master/example/cnn_visualization
The only difference is we have a python script in place of notebook in examples folder for
this particular example.
Any thoughts/suggestions?

On the other hand, I agree with Sandeep that there should be some basic testing of examples
and we can start with a small set of python examples to begin with.

—Ankit


On Nov 13, 2018, at 10:31 AM, Naveen Swamy <mnnaveen@gmail.com<mailto:mnnaveen@gmail.com>>
wrote:

Aaron, IMO tutorials have a specific purpose, to introduce concepts and
APIs to the users and I think converting examples to tutorials would
overwhelm the users, we should carefully choose which examples we want to
turn into tutorials.

I agree that today examples are graveyard of untested code, my suggestion
is to add some testing to the example when you touch the example - at the
least to check the functionality. These can be run once a week.


On Tue, Nov 13, 2018 at 6:52 AM Aaron Markham <aaron.s.markham@gmail.com<mailto:aaron.s.markham@gmail.com>>
wrote:

I've been actively promoting moving examples to tutorials during reviews.
That way they fall under the testing umbrella and get added to the website.

Many times there's not really a great distinction as to why something is in
the examples folder, other than it's like a graveyard of untested sample
code.

I would suggest a starting strategy of when doing updates on examples, see
if with just a little more effort, ask yourself, can it be converted to a
tutorial?

The last thing CI needs is more flaky tutorial tests, so whatever is done
here should use the more robust approaches that are being discussed.

Cheers,
Aaron

On Mon, Nov 12, 2018, 16:24 sandeep krishnamurthy <
sandeep.krishna98@gmail.com<mailto:sandeep.krishna98@gmail.com> wrote:

Thanks, Ankit for bringing this up. @Anirudh - All the concerns you
raised
are very valid. Here are my thoughts:
1. There were several examples that were crashing or had compiler errors.
This is a very bad user experience. All example scripts should be at
least runnable!
2. While I agree examples are too diverse (python scripts, notebooks,
epochs, print statements etc..) We can always start small, we can start
with 5 examples. We can use this to streamline all examples to be python
scripts, print statements, with the main function invoker that can take
params like epoch, dataset etc.
3. We can start with running weekly tests to avoid too long nightly test
pipeline.
4. One possible issue can be on a few examples that depend on a large or
controlled dataset. I am not sure yet, how to solve this, but, we can
think.

Any suggestions?
Best,
Sandeep



On Mon, Nov 12, 2018 at 10:38 AM Anirudh Acharya <anirudhkrec@gmail.com<mailto:anirudhkrec@gmail.com>>
wrote:

Hi Ankit,

I have a few concerns about testing examples. Before writing tests for
examples,

  - you will need to first decide what constitutes a test for an
example,
  because examples are not API calls, which will have return
statements
and
  the test can just call the API and assert for certain values. Just
testing
  if an example is a compilable python script will not add much value
in
my
  opinion.
  - And testing for example output and results will require a re-write
of
  many of the examples, because many of them currently just have print
  statements as outputs and does not return any value as such. I am
not
sure
  if it is worth the dev-effort.
  - the current set of examples in the mxnet repo are very diverse -
some
  are written as python notebooks, some are just python scripts with
paper
  implementations, and some are just illustrations of certain mxnet
features.
  I am curious to know how you will write tests for these things.


Looking forward to seeing the design of this test bed/framework.


Thanks
Anirudh Acharya

On Fri, Nov 9, 2018 at 2:39 PM Marco de Abreu
<marco.g.abreu@googlemail.com.invalid<mailto:marco.g.abreu@googlemail.com.invalid>>
wrote:

Hello Ankit,

that's a great idea! Using the tutorial tests as reference is a great
starting point. If you are interested, please don't hesitate to
attend
the
Berlin user group in case you would like to discuss your first
thoughts
in-person before drafting a design.

-Marco


Am Fr., 9. Nov. 2018, 23:23 hat khedia.ankit@gmail.com<mailto:khedia.ankit@gmail.com>
<
khedia.ankit@gmail.com<mailto:khedia.ankit@gmail.com>> geschrieben:

Hi MXNet community,

Recently, I and a few other contributors focussed on fixing
examples
in
our repository which were not working out of the box as expected.
https://github.com/apache/incubator-mxnet/issues/12800
https://github.com/apache/incubator-mxnet/issues/11895
https://github.com/apache/incubator-mxnet/pull/13196

Some of the examples failed after API changes and remained uncaught
until
a user reported the issue. While the community is actively working
on
fixing it, it might re-occur after few days if we don’t have a
proper
mechanism to catch regressions.

So, I would like to propose to enable nightly/weekly tests for the
examples similar to what we have for tutorials to catch any such
regressions. The test could check only basic
functionalities/working
of
the
examples. It can run small examples completely whereas it can run
long
training examples for only few epochs.

Any thoughts from the community? Any other suggestions for fixing
the
same?

Regards,
Ankit Khedia





--
Sandeep Krishnamurthy



Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message