mxnet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lausen, Leonard" <lau...@amazon.com.INVALID>
Subject Re: MXNet Bot Demo
Date Fri, 13 Mar 2020 19:27:40 GMT
On opt-out: People may be unaware of opt-out would not use it. There is no
incentive to use opt-out, as the PR author doesn't pay any money for CI run.

I agree with Marco though that opt-in alone may cause usability issues, as
contributors may not be aware of how to trigger the CI.
One solution is that the bot proactively comments on the PR and reminds the
author to trigger running CI once the author deems the PR ready.

But even if we choose opt-out, the bot will still add a lot of value, as PR
authors can retrigger single jobs that have failed due to flakiness.

> Is it possible for re-triggering a single job to be abused? For example,
> the author spends two days re-triggering a flaky job to make it pass.

Yes, this is possible. I suggest the committer who likes to merge a PR needs to
make a good judgement here if a PR is abusing the feature, and if so, retrigger
all CI runs.

Best regards
Leonard

On Fri, 2020-03-13 at 08:07 +0100, Marco de Abreu wrote:
> Thanks for the data Sandeep. In these cases it sounds like it would have
> rather been better when people explicitly turned off CI in that case.
> What's the argument against an opt-out instead of an opt-in?
> 
> My intention is that I consider it quite cumbersome to make it a *required*
> step to always trigger CI manually, even if just submitting a small PR. I'd
> rather see people explicitly turning off CI if they wouldn't like to use it
> - and there's also the "draft" stage for a PR which some contributors are
> using.
> 
> With regards to WIP and do not review: I think these are use cases where
> you want CI feedback, as otherwise you wouldn't have opened the PR. If you
> don't want human feedback and neither machine feedback, why open the PR at
> all?
> 
> -Marco
> 
> 
> sandeep krishnamurthy <sandeep.krishna98@gmail.com> schrieb am Fr., 13.
> März 2020, 05:24:
> 
> > I tried to gather some data for us to discuss this topic in this thread. I
> > tried to count number of un-necessary builds by looking at most recent (as
> > of 12, March 9 PM PST) 50 PRs merged to master and 50 PRs.
> > Identifying un-necessary builds is bit subjective. I tried to be more
> > conservative where I didn't count a build as un-necessary if I was in
> > doubt. Hence, I was not able to automate, but I made an effort to go
> > through PRs manually and use below criteria to identify un-necessary
> > commits triggering the builds.
> > 
> >    1. Explicitly marked as WIP / do not review  PR
> >    2. Incremental WIP commit and finally commenting a commit “trigger CI”
> >    3. Multiple commits to address all comments from single review. This is
> >    assuming we see a comment, address them, commit, next the following
> > comment
> >    4. Sequence of documentation only changes
> > 
> > I found there were around 42 avoidable builds from most recent 50 merged
> > PRs and around 86 builds from recent 50 open PRs.
> > 
> > I synced up with other contributors (Joe Evans, Chai) from Amazon who is
> > contributing to MXNet CI system. I was told that on an average it costs
> > around $84 per build and on an average 6 commits per merged PR (for year
> > 2019). Going by that, it is approximately 1/6 builds are avoidable. [100 /
> > 300 + 300 ]
> > 
> > 
> > Usability should be top most priority. But, since either a reviewer or pr
> > author can trigger the bot, is it really a hurdle for pr author or reviewer
> > to call a bot to trigger CI? Given that PR author and reviewer is already
> > actively commenting various details such as - PR description, review
> > comments and responses, adding labels etc.
> > 
> > 
> > Me too curious to know the behavior for Tao's above use case.
> > 
> > 
> > Best,
> > 
> > Sandeep
> > 
> > On Thu, Mar 12, 2020 at 7:18 PM Tao Lv <mutouorz@gmail.com> wrote:
> > 
> > > Is it possible for re-triggering a single job to be abused? For example,
> > > the author spends two days re-triggering a flaky job to make it pass. But
> > > other jobs which have passed the validation may be broken by other
> > commits
> > > during the two day without being noticed. And finally the PR is merged
> > with
> > > underlying problems.
> > > 
> > > On Fri, Mar 13, 2020 at 6:19 AM Marco de Abreu <marco.g.abreu@gmail.com>
> > > wrote:
> > > 
> > > > In the end it only comes down to money, considering that the system is
> > > auto
> > > > scaling, making the execution time constant.
> > > > 
> > > > If we're trading money for usability, I certainly would prefer
> > usability.
> > > > I'd rather recommend to spend time on parallelizing test execution or
> > > > getting rid of integration tests in the PR stage instead reducing the
> > > costs
> > > > by making people not use it. But taking a step back to requiring people
> > > to
> > > > manually trigger CI again doesn't feel right.
> > > > 
> > > > I'm happy to see that bot deployed, but I do not agree with removing
> > the
> > > > auto trigger functionality for new commits.
> > > > 
> > > > -Marco
> > > > 
> > > > Chaitanya Bapat <chai.bapat@gmail.com> schrieb am Do., 12. März
2020,
> > > > 22:47:
> > > > 
> > > > > @Marco Thanks for pointing that out.
> > > > > Tomorrow i.e. Friday, March 13, 2020 at 3:00 PM - 3:30 PM in
> > > (UTC-08:00)
> > > > > Pacific Time (US & Canada).
> > > > > 
> > > > > > When do we expect this bot to be deployed?
> > > > > @Lin If all goes well in the next week I can deploy it to public
> > Apache
> > > > > (provided I get permissions from Apache Infra)
> > > > > 
> > > > > @Marco Thanks for your feedback.
> > > > > > CI system has to support the community without requiring people
to
> > > > > constantly shepherd every single run
> > > > > We have data for the number of times CI was triggered unnecessarily
> > > which
> > > > > includes
> > > > > - Entire build triggered instead of specific build
> > > > > - CI triggered when PR is still work in progress or not yet ready
> > (say
> > > -
> > > > > intermediate commits)
> > > > > At the end its a trade-off
> > > > > Money, Resources, Time to build for each and every commit vs Pain
of
> > > > > triggering builds
> > > > > 
> > > > > 
> > > > > >  Scan trigger plugin would poll SCM. Can we use plugin at scale?
> > > > > 
> > > > > 1. I haven't tested it on scale. But I think with the current scale
> > of
> > > > > MXNet repo (191 open PRs i.e. checking for changes to 191 branches
-
> > It
> > > > > should be manageable)
> > > > > 2. What's the purpose of the plugin? tldr; Branch discovery or branch
> > > > > indexing.
> > > > > Scan trigger plugin comes into the picture only once per PR per job
> > > > (i.e. 8
> > > > > times per PR for 8 jobs). It is basically done when a new PR is made
> > > and
> > > > > the job (say unix-cpu hasn't discovered the new PR branch yet).
> > That's
> > > > it.
> > > > > So it shouldn't be a problem for public MXNet repo.
> > > > > 
> > > > > Thanks,
> > > > > Chai
> > > > > 
> > > > > 
> > > > > On Thu, 12 Mar 2020 at 14:22, Marco de Abreu <
> > marco.g.abreu@gmail.com>
> > > > > wrote:
> > > > > 
> > > > > > Btw you forgot to set a date and time for the metting
> > > > > > 
> > > > > > 
> > > > > > On Thu, Mar 12, 2020 at 10:18 PM Marco de Abreu <
> > > > marco.g.abreu@gmail.com
> > > > > > wrote:
> > > > > > 
> > > > > > > Thanks Chai, I generally like the idea of the bot. But
I'm not a
> > > > > > supporter
> > > > > > > of the idea to disable any automatic triggering (disabling
the
> > > > webhook
> > > > > is
> > > > > > > also not an option, considering that this will disable
master
> > > > > triggers).
> > > > > > > The CI system has to support the community without requiring
> > people
> > > > to
> > > > > > > constantly shepherd every single run. Disabling automatic
> > > triggering
> > > > > > seems
> > > > > > > like a step back to me.
> > > > > > > 
> > > > > > > Instead, I'd recommend that CI gets triggered upon every
commit
> > as
> > > > > usual,
> > > > > > > but people have the possibility to call a "command" (i.e.
make a
> > > > > message
> > > > > > > which results in the bot setting a label) to disable CI
until
> > they
> > > > > revoke
> > > > > > > it. But the standard should still be that a new commit
triggers a
> > > new
> > > > > CI
> > > > > > > run.
> > > > > > > 
> > > > > > > https://plugins.jenkins.io/multibranch-scan-webhook-trigger/
> > > seems
> > > > > like
> > > > > > > this would poll SCM. This will incur high quota restrictions.
Are
> > > you
> > > > > > sure
> > > > > > > that you can use that plugin at scale?
> > > > > > > 
> > > > > > > -Marco
> > > > > > > 
> > > > > > > 
> > > > > > > On Thu, Mar 12, 2020 at 10:04 PM Lin Yuan <apeforest@gmail.com>
> > > > wrote:
> > > > > > > > Chai,
> > > > > > > > 
> > > > > > > > Awesome work. When do we expect this bot to be deployed?
> > > > > > > > 
> > > > > > > > Best,
> > > > > > > > 
> > > > > > > > Lin
> > > > > > > > 
> > > > > > > > On Thu, Mar 12, 2020 at 2:00 PM Chaitanya Bapat <
> > > > chai.bapat@gmail.com
> > > > > > > > wrote:
> > > > > > > > 
> > > > > > > > > Hello MXNet community,
> > > > > > > > > 
> > > > > > > > > I have built an MXNet Bot <https://github.com/mxnet-bot>
that
> > > > > allows
> > > > > > PR
> > > > > > > > > Authors, Committers and Jenkins Admins to trigger
CI manually.
> > > > > > > > > It handles 2 problems
> > > > > > > > > 1. Manual CI trigger instead of existing automated
CI trigger
> > > > > > > > > 2. Gives permissions to PR Authors (in addition
to MXNet
> > > > Committers
> > > > > > and
> > > > > > > > > Jenkins Admins)
> > > > > > > > > 
> > > > > > > > > Design Doc :
> > > > > > > > > 
> > https://cwiki.apache.org/confluence/display/MXNET/MXNet+CI+Bot
> > > > > > > > > I urge you all to attend the demonstration meeting
and lend
> > your
> > > > > views
> > > > > > > > on
> > > > > > > > > the same.
> > > > > > > > > 
> > > > > > > > > Thank you,
> > > > > > > > > Chai
> > > > > > > > > 
> > > > > > > > > *Meeting Details*:
> > > > > > > > > ==============Conference Bridge Information==============
> > > > > > > > > You have been invited to an online meeting, powered
by Amazon
> > > > Chime.
> > > > > > > > > *Chime meeting ID*: *9272158344*
> > > > > > > > > Join via Chime clients (manually): Select 'Meetings
> Join a
> > > > > Meeting',
> > > > > > > > and
> > > > > > > > > enter 9272158344
> > > > > > > > > Join via Chime clients (auto-call): If you invite
auto-call as
> > > > > > attendee,
> > > > > > > > > Chime will call you when the meeting starts,
select 'Answer'
> > > > > > > > > *Join via browser screen share*: https://chime.aws/9272158344
> > > > > > > > > *Join via phone* (US): +1-929-432-4463,,,9272158344#
> > > > > > > > > *Join via phone (US toll-free)*: +1-855-552-4463,,,9272158344#
> > > > > > > > > International dial-in: https://chime.aws/dialinnumbers/
> > > > > > > > > In-room video system: Ext: 62000, Meeting PIN:
9272158344#
> > > > > > > > > 
> > > > > > > > > --
> > > > > > > > > *Chaitanya Prakash Bapat*
> > > > > > > > > *+1 (973) 953-6299*
> > > > > > > > > 
> > > > > > > > > [image: https://www.linkedin.com//in/chaibapat25]
> > > > > > > > > <https://github.com/ChaiBapchya>[image:
> > > > > > > > https://www.facebook.com/chaibapat
> > > > > > > > > ]
> > > > > > > > > <https://www.facebook.com/chaibapchya>[image:
> > > > > > > > > https://twitter.com/ChaiBapchya] <
> > > https://twitter.com/ChaiBapchya
> > > > > > > > > [image:
> > > > > > > > > https://www.linkedin.com//in/chaibapat25]
> > > > > > > > > <https://www.linkedin.com//in/chaibapchya/>
> > > > > > > > > 
> > > > > 
> > > > > --
> > > > > *Chaitanya Prakash Bapat*
> > > > > *+1 (973) 953-6299*
> > > > > 
> > > > > [image: https://www.linkedin.com//in/chaibapat25]
> > > > > <https://github.com/ChaiBapchya>[image:
> > > > https://www.facebook.com/chaibapat
> > > > > ]
> > > > > <https://www.facebook.com/chaibapchya>[image:
> > > > > https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya
> > > > > [image:
> > > > > https://www.linkedin.com//in/chaibapat25]
> > > > > <https://www.linkedin.com//in/chaibapchya/>
> > > > > 
> > 
> > --
> > Sandeep Krishnamurthy
> > 
Mime
View raw message