mxnet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Markham <>
Subject Re: Nightly/Weekly tests for examples
Date Tue, 13 Nov 2018 23:14:38 GMT
Naveen - I agree with you there. Not every example should be moved to
tutorials... just the ones that would be great to have on the site and
to be tested regularly.
I think that having examples of specific APIs will be useful on the website too.

Ankit, there's a cost/benefit analysis for keeping both a command line
example and a notebook. That has maintenance overhead.
A couple of options to debate and apply on a case-by-case basis:
1) Only maintain one or the other.
2) If it's really important, maintain both, but update both codebases
with notes and directions for maintenance, so that if one is found
broken and is getting updated, contributors know that they should
update the other.

Since the CNN Viz example/tutorial is for visualization, I'd lean
towards #1 and keep only the notebook. Then delete the example, but
since it requires a python file from that examples folder as a
utility, move this to a tutorials_utils folder in /docs. That folder
should be organized appropriately and excluded in the docs build
config in Look for "exclude_patterns".

My 3 cents... Cheers,

On Tue, Nov 13, 2018 at 11:26 AM Khedia, Ankit
<> wrote:
> We can start small consolidating few examples which are there in both tutorials/examples
> One of such example is GradCAM implementation:
> In tutorials -
> In examples-
> The only difference is we have a python script in place of notebook in examples folder
for this particular example.
> Any thoughts/suggestions?
> On the other hand, I agree with Sandeep that there should be some basic testing of examples
and we can start with a small set of python examples to begin with.
> —Ankit
> On Nov 13, 2018, at 10:31 AM, Naveen Swamy <<>>
> Aaron, IMO tutorials have a specific purpose, to introduce concepts and
> APIs to the users and I think converting examples to tutorials would
> overwhelm the users, we should carefully choose which examples we want to
> turn into tutorials.
> I agree that today examples are graveyard of untested code, my suggestion
> is to add some testing to the example when you touch the example - at the
> least to check the functionality. These can be run once a week.
> On Tue, Nov 13, 2018 at 6:52 AM Aaron Markham <<>>
> wrote:
> I've been actively promoting moving examples to tutorials during reviews.
> That way they fall under the testing umbrella and get added to the website.
> Many times there's not really a great distinction as to why something is in
> the examples folder, other than it's like a graveyard of untested sample
> code.
> I would suggest a starting strategy of when doing updates on examples, see
> if with just a little more effort, ask yourself, can it be converted to a
> tutorial?
> The last thing CI needs is more flaky tutorial tests, so whatever is done
> here should use the more robust approaches that are being discussed.
> Cheers,
> Aaron
> On Mon, Nov 12, 2018, 16:24 sandeep krishnamurthy <
><> wrote:
> Thanks, Ankit for bringing this up. @Anirudh - All the concerns you
> raised
> are very valid. Here are my thoughts:
> 1. There were several examples that were crashing or had compiler errors.
> This is a very bad user experience. All example scripts should be at
> least runnable!
> 2. While I agree examples are too diverse (python scripts, notebooks,
> epochs, print statements etc..) We can always start small, we can start
> with 5 examples. We can use this to streamline all examples to be python
> scripts, print statements, with the main function invoker that can take
> params like epoch, dataset etc.
> 3. We can start with running weekly tests to avoid too long nightly test
> pipeline.
> 4. One possible issue can be on a few examples that depend on a large or
> controlled dataset. I am not sure yet, how to solve this, but, we can
> think.
> Any suggestions?
> Best,
> Sandeep
> On Mon, Nov 12, 2018 at 10:38 AM Anirudh Acharya <<>>
> wrote:
> Hi Ankit,
> I have a few concerns about testing examples. Before writing tests for
> examples,
>   - you will need to first decide what constitutes a test for an
> example,
>   because examples are not API calls, which will have return
> statements
> and
>   the test can just call the API and assert for certain values. Just
> testing
>   if an example is a compilable python script will not add much value
> in
> my
>   opinion.
>   - And testing for example output and results will require a re-write
> of
>   many of the examples, because many of them currently just have print
>   statements as outputs and does not return any value as such. I am
> not
> sure
>   if it is worth the dev-effort.
>   - the current set of examples in the mxnet repo are very diverse -
> some
>   are written as python notebooks, some are just python scripts with
> paper
>   implementations, and some are just illustrations of certain mxnet
> features.
>   I am curious to know how you will write tests for these things.
> Looking forward to seeing the design of this test bed/framework.
> Thanks
> Anirudh Acharya
> On Fri, Nov 9, 2018 at 2:39 PM Marco de Abreu
> <<>>
> Hello Ankit,
> that's a great idea! Using the tutorial tests as reference is a great
> starting point. If you are interested, please don't hesitate to
> attend
> the
> Berlin user group in case you would like to discuss your first
> thoughts
> in-person before drafting a design.
> -Marco
> Am Fr., 9. Nov. 2018, 23:23 hat<>
><>> geschrieben:
> Hi MXNet community,
> Recently, I and a few other contributors focussed on fixing
> examples
> in
> our repository which were not working out of the box as expected.
> Some of the examples failed after API changes and remained uncaught
> until
> a user reported the issue. While the community is actively working
> on
> fixing it, it might re-occur after few days if we don’t have a
> proper
> mechanism to catch regressions.
> So, I would like to propose to enable nightly/weekly tests for the
> examples similar to what we have for tutorials to catch any such
> regressions. The test could check only basic
> functionalities/working
> of
> the
> examples. It can run small examples completely whereas it can run
> long
> training examples for only few epochs.
> Any thoughts from the community? Any other suggestions for fixing
> the
> same?
> Regards,
> Ankit Khedia
> --
> Sandeep Krishnamurthy

View raw message