mxnet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chaitanya Bapat <chai.ba...@gmail.com>
Subject Re: MXNet Bot Demo
Date Mon, 23 Mar 2020 19:53:15 GMT
Hello,
Update: Apache Infra Ticket for MXNet Bot
Thanks once again, Marco for opening the ticket. But turns out, Apache
Infra folks closed it stating: "Security concerns around allowing unknown
person to submit PR and run our hardware". Furthermore, it goes onto state
that bot circumvents the dependence on Jenkins Admins which is like solving
a problem that doesn't exist.

I sense there is some confusion in the communication (maybe on my part). It
turns out the security concerns aren't actually correct.

1. Unknown person can submit a PR (before & after bot proposal), and run
our hardware (pre as well as post bot).
2. Code should be reviewed by somebody with an ICLA on file. This doesn't
change either. Prior to merging a PR, code has to be approved by a
committer just like before.
Overall it looks like the job of the bot isn't clear to folks in Apache
Infra. Bot simply is a means for triggering CI (which could be done
manually by Log In to Jenkins -> PR -> Job -> Build) and doesn't quite
tweak with merging procedure. Yes, only addition is now unknown person (PR
Author) can trigger CI with a message (but that was possible anyway by
pushing a commit. Bot just prevents users from pushing empty commits and
building entire suite).

As can be seen from last 10 open PRs as of Monday 23rd March, 12pm PT most
PRs fail on 1/2 jobs. In such a scenario, the proposed MXNet bot would come
in handy for just invoking CI on that specific build (instead of a
non-committer PR Author to push empty commit : hurting on the resource,
time & cost considerations apart from undesirable dev experience)

@Marco Since I am a non-committer, I guess these 2 clarifications need to
be conveyed to the Apache Infra by someone with Committer access.

What do you think?

Thanks,
Chai

On Sat, 21 Mar 2020 at 16:08, Marco de Abreu <marco.g.abreu@gmail.com>
wrote:

> Hello,
>
> the ticket has been created:
> https://issues.apache.org/jira/browse/INFRA-20005
>
> Best regards,
> Marco
>
> On Thu, Mar 19, 2020 at 11:49 PM Marco de Abreu <marco.g.abreu@gmail.com>
> wrote:
>
> > Sounds like a good plan!
> >
> > Please send me the URL (please make sure it's backed by DNS and not just
> > the gateway URL) of the webhook handler, GitHub events you're interested
> in
> > and the shared secret in a private email to my personal email address. I
> > will then create the ticket with Apache infra.
> >
> > -Marco
> >
> > Chaitanya Bapat <chai.bapat@gmail.com> schrieb am Do., 19. März 2020,
> > 23:07:
> >
> >> @Marco Alright, it makes total sense to test out the Bot feature
> alongside
> >> auto-trigger as a transition.
> >>
> >> Path Forward:
> >> 1. Setup MXNet Bot on apache/incubator-mxnet repo (GitHub WebHook and
> >> Infra)
> >> 2. We don't turn off automatic trigger of PR builds for now.
> >> 3. Hopefully, bot is used by developers to trigger specific jobs
> >> 4. Later on (say around April 20), let's discuss the possibility of
> >> switching off auto-trigger (with appropriate data) if it makes sense.
> >> Thanks Marco for volunteering to help enable the web hook on
> >> apache/incubator-mxnet. Let me know if we can sync up on Slack channel
> to
> >> get the ball rolling.
> >>
> >> Thanks once again for the entire community to step in and help try out
> >> this
> >> Bot.
> >> Chai
> >>
> >> On Wed, 18 Mar 2020 at 17:07, Marco de Abreu <marco.g.abreu@gmail.com>
> >> wrote:
> >>
> >> > Hi, that's correct. But as stated previously, it's not an option to
> >> remove
> >> > the hook. For now, I'd like to see how the system behaves while it's
> >> > optional. Later on, we can talk about revisiting this decision. But to
> >> me
> >> > it's not an option to deploy an entirely new system and approach
> without
> >> > having a transition or even a timeframe in which we are able to fall
> >> back.
> >> >
> >> > I'm happy to support the deployment of the bot and add an additional
> >> > webhook to enable it's functionality to support selective triggering
> by
> >> PR
> >> > authors and committers, but I will not support the disabling of
> >> automatic
> >> > triggering of branches or PRs.
> >> >
> >> > -Marco
> >> >
> >> > Chaitanya Bapat <chai.bapat@gmail.com> schrieb am Mi., 18. März 2020,
> >> > 21:00:
> >> >
> >> > > Hey Marco,
> >> > >
> >> > > I thought currently every commit on PR and master triggers CI
> >> > > because
> >> > > a. github webhook points to Jenkins Server
> >> > > b. GH Webhook events trigger builds on Jenkins for all commits to
> any
> >> > > branch in apache/incubator-mxnet
> >> > > may it be master/PR/non-PR
> >> > > Reason:
> >> > > Because all the 3 types of branches are discovered by Jenkins
> (non-PR
> >> > > (including master) and PR)
> >> > >
> >> > > Proposal: Remove GitHub WebHook to Jenkins and replace with GH
> >> Webhook to
> >> > > Lambda
> >> > > But after I remove the github webhook that points to Jenkins : *N**o
> >> > commit
> >> > > will trigger Jenkins build by default* (as Jenkins wont receive GH
> >> > events)
> >> > > Only those that Bot deems fit will be triggered (using Jenkins API
> >> > invoked
> >> > > by Lambda).
> >> > > Hence its needed to handle that case of master merge.
> >> > > Am I understanding this correctly?
> >> > >
> >> > > On Wed, 18 Mar 2020 at 04:23, Marco de Abreu <
> marco.g.abreu@gmail.com
> >> >
> >> > > wrote:
> >> > >
> >> > > > Thanks Chai, sounds good to me. Could you elaborate a bit on the
> >> point
> >> > > > about triggering a CI run after the PR has been merged? We already
> >> to
> >> > > that
> >> > > > automatically for the master, so what's the benefit to do it
> twice?
> >> > > >
> >> > > > -Marco
> >> > > >
> >> > > > Chaitanya Bapat <chai.bapat@gmail.com> schrieb am Mi., 18. März
> >> 2020,
> >> > > > 09:30:
> >> > > >
> >> > > > > Update:
> >> > > > >
> >> > > > > >  we can ensure that all CI runs ran on the commit that will be
> >> > merged
> >> > > > > @Sam Skalicky <samskalicky@gmail.com> Branch Protection is
> added
> >> to
> >> > > > public
> >> > > > > MXNet repo. It ensures that for every PR to be merged, the CI
> >> passes.
> >> > > All
> >> > > > > the jobs selected "required" jobs will have to be green for the
> >> PR to
> >> > > be
> >> > > > > merged. Ofcourse, users with "Adminstrator" access can merge
> >> without
> >> > it
> >> > > > but
> >> > > > > that's just a backdoor. It is the case now and will continue to
> be
> >> > the
> >> > > > case
> >> > > > > with the inclusion of Bot.
> >> > > > >
> >> > > > > > easily verify that the CI has executed all runs on the commit
> >> that
> >> > > will
> >> > > > > be merged
> >> > > > > GitHub UI shows all the jobs and the status corresponding to it
> on
> >> > > every
> >> > > > > commit. That should suffice. For the merged commits, Repo ->
> >> Commits
> >> > ->
> >> > > > > Commit ID (Status) can be tracked currently (only way that I
> know
> >> > of).
> >> > > > > Moreover, it is beyond the scope of this project (and possibly
> >> out of
> >> > > our
> >> > > > > control since this is purely GitHub UI specific use-case).
> >> > > > >
> >> > > > > Thanks @przemyslaw for supporting the opt-in.
> >> > > > >
> >> > > > > Thanks everyone in the community for sharing concerns, voicing
> >> your
> >> > > > opinion
> >> > > > > and participating in the discussion.
> >> > > > > Thanks to those who attended the demo last Friday.
> >> > > > >
> >> > > > > Action items from that discussion
> >> > > > > 1. Handle master merge builds [Done]
> >> > > > > Bot runs entire CI suite after the PR is merged and comments on
> >> the
> >> > PR
> >> > > > > about the same.
> >> > > > > Design decision :
> >> > > > > MXNet Bot comment about master merge build on the *merge commit
> vs
> >> > PR*.
> >> > > > > After the PR is merged, Bot runs entire CI and comments the
> >> result of
> >> > > CI
> >> > > > > trigger on the PR (because it is easy to track on a PR rather
> than
> >> > > > > commenting inside the merge commit)
> >> > > > >
> >> > > > > 2. Idempotent condition
> >> > > > > In case of already running build, if an attempt is made to
> >> retrigger
> >> > > the
> >> > > > > job then what should be the response
> >> > > > > a. Not to re-trigger, let the ongoing build continue till
> >> completion
> >> > > > > b. End the ongoing build and re-trigger
> >> > > > > c. Let the ongoing build continue, re-trigger new build
> >> > > > >
> >> > > > > From resource saving point of view, *c* looks costly and a can
> be
> >> > > > > avoided/optimized by B.
> >> > > > > In case when a re-trigger was started "erroneously" then killing
> >> > > ongoing
> >> > > > > build and re-trigger is a waste.
> >> > > > > In case when ongoing build failed in one sub-part, then
> >> re-triggering
> >> > > is
> >> > > > > justified.
> >> > > > > Erroneous re-triggers would be less often while conscious
> >> re-triggers
> >> > > to
> >> > > > > suppress failure is more common use-case. It looks like a safe
> >> > > assumption
> >> > > > > to make given the trade-off.
> >> > > > > [Open to debate]
> >> > > > >
> >> > > > > 3. Add security consideration [Use of secret manager, but
> without
> >> > > > > auto-rotation due to Jenkins manual config requirement] [Done]
> >> > > > > 4. New PR Instruction message by the Bot [Done]
> >> > > > > Thanks to the suggestion of Leonard, supported by others. I've
> now
> >> > > added
> >> > > > > the feature where the Bot comments a help message. [For
> reference
> >> -
> >> > > > > https://github.com/ChaiBapchya/incubator-mxnet/pull/52]
> >> > > > >
> >> > > > > Barring the opt-in vs opt-out debate & idempotency, consensus
> was
> >> > > quickly
> >> > > > > reached for the rest.
> >> > > > >
> >> > > > > In the coming days, I hope to roll-out this feature into Prod
> >> (public
> >> > > > > MXNet) for all devs to use.
> >> > > > >
> >> > > > > Thanks,
> >> > > > > Chai
> >> > > > >
> >> > > > > On Mon, 16 Mar 2020 at 11:57, Marco de Abreu <
> >> > marco.g.abreu@gmail.com>
> >> > > > > wrote:
> >> > > > >
> >> > > > > > Well that's generally a problem with a deferred CI approach
> (CI
> >> is
> >> > > run
> >> > > > at
> >> > > > > > commit and not at merge time). This can either be solved
> through
> >> > the
> >> > > > > other
> >> > > > > > proposal that's currently on dev@, by having a bot which does
> >> > merges
> >> > > > by
> >> > > > > > having a global lock and a merge queue or by accepting the
> >> issue.
> >> > > > Reality
> >> > > > > > right now is that we're running that model where two PRs which
> >> are
> >> > > > merged
> >> > > > > > in parallel might break one another. One thing to consider
> >> though
> >> > is
> >> > > > that
> >> > > > > > this breakage would have to be introduced in two separate
> parts
> >> > since
> >> > > > > > otherwise there'd be merge conflicts. I think we had that
> >> situation
> >> > > > twice
> >> > > > > > so far and the result was a quick revert, so I'd say that
> it's a
> >> > > > problem
> >> > > > > > that can happily be accepted. All other solutions basically
> >> require
> >> > > > some
> >> > > > > > form of single-threaded and globally locked solution which
> >> limits
> >> > us
> >> > > in
> >> > > > > > scalability. I'd recommend to just accept that risk and revert
> >> a PR
> >> > > in
> >> > > > > case
> >> > > > > > it actually had a conflict.
> >> > > > > >
> >> > > > > > -Marco
> >> > > > > >
> >> > > > > > On Mon, Mar 16, 2020 at 6:29 PM Skalicky, Sam
> >> > > > <sskalic@amazon.com.invalid
> >> > > > > >
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > We probably need some way to track which CI runs ran for
> which
> >> > > commit
> >> > > > > > too,
> >> > > > > > > that way we can ensure that all CI runs ran on the commit
> that
> >> > will
> >> > > > be
> >> > > > > > > merged.  Maybe the bot can comment with the commit hash when
> >> > users
> >> > > > > > command
> >> > > > > > > it to do something. Although since users can trigger
> >> individual
> >> > CI
> >> > > > runs
> >> > > > > > its
> >> > > > > > > possible to have some commits run some CI runs but not
> >> others. We
> >> > > > need
> >> > > > > > some
> >> > > > > > > way to easily verify that the CI has executed all runs on
> the
> >> > > commit
> >> > > > > that
> >> > > > > > > will be merged.
> >> > > > > > >
> >> > > > > > > Sam
> >> > > > > > >
> >> > > > > > > > On Mar 13, 2020, at 8:28 PM, Przemysław Trędak <
> >> > > ptrendx@apache.org
> >> > > > >
> >> > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > CAUTION: This email originated from outside of the
> >> > organization.
> >> > > Do
> >> > > > > not
> >> > > > > > > click links or open attachments unless you can confirm the
> >> sender
> >> > > and
> >> > > > > > know
> >> > > > > > > the content is safe.
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > I personally like the idea of opt-in more than opt-out:
> >> > > > > > > > - ultimately PR author wants the PR to be merged so they
> (or
> >> > > > > committer
> >> > > > > > > reviewing the PR) will trigger the CI
> >> > > > > > > > - if it is easy to trigger the PR via the bot command then
> >> the
> >> > > > amount
> >> > > > > > of
> >> > > > > > > work per PR should be less than with opt-out (since most of
> >> the
> >> > > > commits
> >> > > > > > > should then be marked as [skip ci] or something similar
> >> > > > > > > >
> >> > > > > > > > +1 to the bot making a comment on each new PR with its
> >> commands
> >> > > > (and
> >> > > > > > > also explaining, or at least giving links to the general PR
> >> > process
> >> > > > so
> >> > > > > > new
> >> > > > > > > PR authors are not lost). Maybe we could make the bot
> >> recognize
> >> > if
> >> > > > the
> >> > > > > PR
> >> > > > > > > author is new or existing contributor and offer advice based
> >> on
> >> > > that?
> >> > > > > > > >
> >> > > > > > > > Thanks
> >> > > > > > > > Przemek
> >> > > > > > > >
> >> > > > > > > > On 2020/03/13 22:06:58, Marco de Abreu <
> >> > marco.g.abreu@gmail.com>
> >> > > > > > wrote:
> >> > > > > > > >> Hi,
> >> > > > > > > >>
> >> > > > > > > >> since it's no longer necessary to push a new commit to
> >> trigger
> >> > > CI,
> >> > > > > it
> >> > > > > > > will
> >> > > > > > > >> already reduce the costs. But to me, requiring an action
> to
> >> > > enable
> >> > > > > CI
> >> > > > > > > after
> >> > > > > > > >> a PR has been created initially, is a no go. User can opt
> >> out
> >> > of
> >> > > > CI,
> >> > > > > > but
> >> > > > > > > >> the default has to be CI being triggered automatically
> for
> >> > every
> >> > > > > > commit
> >> > > > > > > >> unless specifically disabled by a participant. I'm also
> >> fine
> >> > > with
> >> > > > > > > >> triggering certain additional jobs (think about running a
> >> > > nightly
> >> > > > > job
> >> > > > > > > upon
> >> > > > > > > >> request for a PR) to require a manual step, but the PR
> >> > > validation
> >> > > > > > > pipelines
> >> > > > > > > >> have to run automatically. Every check that is marked as
> >> > > > "Required"
> >> > > > > in
> >> > > > > > > >> GitHub has to be automatically kicked off.
> >> > > > > > > >>
> >> > > > > > > >> -Marco
> >> > > > > > > >>
> >> > > > > > > >> On Fri, Mar 13, 2020 at 9:50 PM Chaitanya Bapat <
> >> > > > > chai.bapat@gmail.com
> >> > > > > > >
> >> > > > > > > >> wrote:
> >> > > > > > > >>
> >> > > > > > > >>> Firstly,
> >> > > > > > > >>> Sorry I missed out on attaching the mail thread that was
> >> sent
> >> > > on
> >> > > > > 12th
> >> > > > > > > >>> February for notifying the community of the upcoming
> >> changes
> >> > to
> >> > > > the
> >> > > > > > > MXNet
> >> > > > > > > >>> CI
> >> > > > > > > >>> For reference :
> >> > > > > > > >>>
> >> > > > > > > >>>
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://lists.apache.org/thread.html/r09a6ab2803a996fc80e00fe39ed312fa4865e8805e08df847f1addad%40%3Cdev.mxnet.apache.org%3E
> >> > > > > > > >>>
> >> > > > > > > >>> Now to the questions,
> >> > > > > > > >>>> Is it possible for re-triggering a single job to be
> >> abused?
> >> > > > > > > >>> @Tao In the case when a user re-triggers a single job
> >> > multiple
> >> > > > > times,
> >> > > > > > > that
> >> > > > > > > >>> will be visible in the PR conversation thread. A
> >> committer,
> >> > > even
> >> > > > > > after
> >> > > > > > > he
> >> > > > > > > >>> has approved the PR before, generally takes a look at
> the
> >> > final
> >> > > > > state
> >> > > > > > > of
> >> > > > > > > >>> the PR before merging. Would it be fair to assume the
> >> > committer
> >> > > > > could
> >> > > > > > > take
> >> > > > > > > >>> the multiple re-trigger of a single job into account
> >> before
> >> > > > > merging?
> >> > > > > > > The
> >> > > > > > > >>> committer then has the option to invoke `@mxnet-bot run
> ci
> >> > > [all]
> >> > > > `
> >> > > > > to
> >> > > > > > > >>> trigger the entire build pipeline one last to counter
> the
> >> > > abuse.
> >> > > > > This
> >> > > > > > > is
> >> > > > > > > >>> aligned with what @Leonard said.
> >> > > > > > > >>>
> >> > > > > > > >>> @Sandeep Thanks a lot for collecting and sharing
> valuable
> >> > data.
> >> > > > I'd
> >> > > > > > > concur
> >> > > > > > > >>> with the opinion that given the existing things
> >> committers &
> >> > PR
> >> > > > > > Authors
> >> > > > > > > >>> take care of, invoking CI shouldn't be that big of an
> >> > > additional
> >> > > > > > > burden.
> >> > > > > > > >>>
> >> > > > > > > >>> @Marco With the opt-out, the onus remains on the PR
> >> Author.
> >> > It
> >> > > > > > doesn't
> >> > > > > > > help
> >> > > > > > > >>> reduce the resource usage. Hence, it was suggested to
> >> switch
> >> > to
> >> > > > > > > >>> opt-in. @Leo's suggestion for proactive commenting on
> the
> >> > part
> >> > > of
> >> > > > > bot
> >> > > > > > > makes
> >> > > > > > > >>> sense and is doable.
> >> > > > > > > >>>
> >> > > > > > > >>> Default : opt-out and User initiated opt-in (with
> >> addressing
> >> > > > Leo's
> >> > > > > > fix
> >> > > > > > > for
> >> > > > > > > >>> the usability issue you correctly pointed out )
> >> > > > > > > >>> @Marco How does this sound to you?
> >> > > > > > > >>>
> >> > > > > > > >>> Again, thank you all for chiming in and voicing your
> >> opinion.
> >> > > > > > > Appreciate
> >> > > > > > > >>> it.
> >> > > > > > > >>> We can take ahead these discussions in today's demo
> >> meeting.
> >> > > > > [Design
> >> > > > > > > Doc
> >> > > > > > > >>> <
> >> > > https://cwiki.apache.org/confluence/display/MXNET/MXNet+CI+Bot
> >> > > > >]
> >> > > > > > > [Demo
> >> > > > > > > >>> Video <https://www.youtube.com/watch?v=gfOGwZId8aU>]
> >> > > > > > > >>>
> >> > > > > > > >>> Thanks,
> >> > > > > > > >>> Chai
> >> > > > > > > >>>
> >> > > > > > > >>> On Fri, 13 Mar 2020 at 12:34, Marco de Abreu <
> >> > > > > > marco.g.abreu@gmail.com>
> >> > > > > > > >>> wrote:
> >> > > > > > > >>>
> >> > > > > > > >>>> I'd recommend that the bot makes an initial comment
> when
> >> a
> >> > PR
> >> > > > gets
> >> > > > > > > opened
> >> > > > > > > >>>> and informs the users of its commands. It then tells
> the
> >> > user
> >> > > > the
> >> > > > > > > commend
> >> > > > > > > >>>> to opt out of CI.
> >> > > > > > > >>>>
> >> > > > > > > >>>> -Marco
> >> > > > > > > >>>>
> >> > > > > > > >>>> Lausen, Leonard <lausen@amazon.com.invalid> schrieb am
> >> Fr.,
> >> > > 13.
> >> > > > > > März
> >> > > > > > > >>> 2020,
> >> > > > > > > >>>> 20:27:
> >> > > > > > > >>>>
> >> > > > > > > >>>>> On opt-out: People may be unaware of opt-out would not
> >> use
> >> > > it.
> >> > > > > > There
> >> > > > > > > is
> >> > > > > > > >>>> no
> >> > > > > > > >>>>> incentive to use opt-out, as the PR author doesn't pay
> >> any
> >> > > > money
> >> > > > > > for
> >> > > > > > > CI
> >> > > > > > > >>>>> run.
> >> > > > > > > >>>>>
> >> > > > > > > >>>>> I agree with Marco though that opt-in alone may cause
> >> > > usability
> >> > > > > > > issues,
> >> > > > > > > >>>> as
> >> > > > > > > >>>>> contributors may not be aware of how to trigger the
> CI.
> >> > > > > > > >>>>> One solution is that the bot proactively comments on
> >> the PR
> >> > > and
> >> > > > > > > reminds
> >> > > > > > > >>>> the
> >> > > > > > > >>>>> author to trigger running CI once the author deems the
> >> PR
> >> > > > ready.
> >> > > > > > > >>>>>
> >> > > > > > > >>>>> But even if we choose opt-out, the bot will still add
> a
> >> lot
> >> > > of
> >> > > > > > value,
> >> > > > > > > >>> as
> >> > > > > > > >>>> PR
> >> > > > > > > >>>>> authors can retrigger single jobs that have failed due
> >> to
> >> > > > > > flakiness.
> >> > > > > > > >>>>>
> >> > > > > > > >>>>>> Is it possible for re-triggering a single job to be
> >> > abused?
> >> > > > For
> >> > > > > > > >>>> example,
> >> > > > > > > >>>>>> the author spends two days re-triggering a flaky job
> to
> >> > make
> >> > > > it
> >> > > > > > > pass.
> >> > > > > > > >>>>>
> >> > > > > > > >>>>> Yes, this is possible. I suggest the committer who
> >> likes to
> >> > > > > merge a
> >> > > > > > > PR
> >> > > > > > > >>>>> needs to
> >> > > > > > > >>>>> make a good judgement here if a PR is abusing the
> >> feature,
> >> > > and
> >> > > > if
> >> > > > > > so,
> >> > > > > > > >>>>> retrigger
> >> > > > > > > >>>>> all CI runs.
> >> > > > > > > >>>>>
> >> > > > > > > >>>>> Best regards
> >> > > > > > > >>>>> Leonard
> >> > > > > > > >>>>>
> >> > > > > > > >>>>> On Fri, 2020-03-13 at 08:07 +0100, Marco de Abreu
> wrote:
> >> > > > > > > >>>>>> Thanks for the data Sandeep. In these cases it sounds
> >> like
> >> > > it
> >> > > > > > would
> >> > > > > > > >>>> have
> >> > > > > > > >>>>>> rather been better when people explicitly turned off
> >> CI in
> >> > > > that
> >> > > > > > > case.
> >> > > > > > > >>>>>> What's the argument against an opt-out instead of an
> >> > opt-in?
> >> > > > > > > >>>>>>
> >> > > > > > > >>>>>> My intention is that I consider it quite cumbersome
> to
> >> > make
> >> > > > it a
> >> > > > > > > >>>>> *required*
> >> > > > > > > >>>>>> step to always trigger CI manually, even if just
> >> > submitting
> >> > > a
> >> > > > > > small
> >> > > > > > > >>> PR.
> >> > > > > > > >>>>> I'd
> >> > > > > > > >>>>>> rather see people explicitly turning off CI if they
> >> > wouldn't
> >> > > > > like
> >> > > > > > to
> >> > > > > > > >>>> use
> >> > > > > > > >>>>> it
> >> > > > > > > >>>>>> - and there's also the "draft" stage for a PR which
> >> some
> >> > > > > > > contributors
> >> > > > > > > >>>> are
> >> > > > > > > >>>>>> using.
> >> > > > > > > >>>>>>
> >> > > > > > > >>>>>> With regards to WIP and do not review: I think these
> >> are
> >> > use
> >> > > > > cases
> >> > > > > > > >>>> where
> >> > > > > > > >>>>>> you want CI feedback, as otherwise you wouldn't have
> >> > opened
> >> > > > the
> >> > > > > > PR.
> >> > > > > > > >>> If
> >> > > > > > > >>>>> you
> >> > > > > > > >>>>>> don't want human feedback and neither machine
> feedback,
> >> > why
> >> > > > open
> >> > > > > > the
> >> > > > > > > >>> PR
> >> > > > > > > >>>>> at
> >> > > > > > > >>>>>> all?
> >> > > > > > > >>>>>>
> >> > > > > > > >>>>>> -Marco
> >> > > > > > > >>>>>>
> >> > > > > > > >>>>>>
> >> > > > > > > >>>>>> sandeep krishnamurthy <sandeep.krishna98@gmail.com>
> >> > schrieb
> >> > > > am
> >> > > > > > Fr.,
> >> > > > > > > >>>> 13.
> >> > > > > > > >>>>>> März 2020, 05:24:
> >> > > > > > > >>>>>>
> >> > > > > > > >>>>>>> I tried to gather some data for us to discuss this
> >> topic
> >> > in
> >> > > > > this
> >> > > > > > > >>>>> thread. I
> >> > > > > > > >>>>>>> tried to count number of un-necessary builds by
> >> looking
> >> > at
> >> > > > most
> >> > > > > > > >>>> recent
> >> > > > > > > >>>>> (as
> >> > > > > > > >>>>>>> of 12, March 9 PM PST) 50 PRs merged to master and
> 50
> >> > PRs.
> >> > > > > > > >>>>>>> Identifying un-necessary builds is bit subjective. I
> >> > tried
> >> > > to
> >> > > > > be
> >> > > > > > > >>> more
> >> > > > > > > >>>>>>> conservative where I didn't count a build as
> >> un-necessary
> >> > > if
> >> > > > I
> >> > > > > > was
> >> > > > > > > >>> in
> >> > > > > > > >>>>>>> doubt. Hence, I was not able to automate, but I made
> >> an
> >> > > > effort
> >> > > > > to
> >> > > > > > > >>> go
> >> > > > > > > >>>>>>> through PRs manually and use below criteria to
> >> identify
> >> > > > > > > >>> un-necessary
> >> > > > > > > >>>>>>> commits triggering the builds.
> >> > > > > > > >>>>>>>
> >> > > > > > > >>>>>>>   1. Explicitly marked as WIP / do not review  PR
> >> > > > > > > >>>>>>>   2. Incremental WIP commit and finally commenting a
> >> > commit
> >> > > > > > > >>> “trigger
> >> > > > > > > >>>>> CI”
> >> > > > > > > >>>>>>>   3. Multiple commits to address all comments from
> >> single
> >> > > > > review.
> >> > > > > > > >>>>> This is
> >> > > > > > > >>>>>>>   assuming we see a comment, address them, commit,
> >> next
> >> > the
> >> > > > > > > >>>> following
> >> > > > > > > >>>>>>> comment
> >> > > > > > > >>>>>>>   4. Sequence of documentation only changes
> >> > > > > > > >>>>>>>
> >> > > > > > > >>>>>>> I found there were around 42 avoidable builds from
> >> most
> >> > > > recent
> >> > > > > 50
> >> > > > > > > >>>>> merged
> >> > > > > > > >>>>>>> PRs and around 86 builds from recent 50 open PRs.
> >> > > > > > > >>>>>>>
> >> > > > > > > >>>>>>> I synced up with other contributors (Joe Evans,
> Chai)
> >> > from
> >> > > > > Amazon
> >> > > > > > > >>> who
> >> > > > > > > >>>>> is
> >> > > > > > > >>>>>>> contributing to MXNet CI system. I was told that on
> an
> >> > > > average
> >> > > > > it
> >> > > > > > > >>>> costs
> >> > > > > > > >>>>>>> around $84 per build and on an average 6 commits per
> >> > merged
> >> > > > PR
> >> > > > > > (for
> >> > > > > > > >>>>> year
> >> > > > > > > >>>>>>> 2019). Going by that, it is approximately 1/6 builds
> >> are
> >> > > > > > avoidable.
> >> > > > > > > >>>>> [100 /
> >> > > > > > > >>>>>>> 300 + 300 ]
> >> > > > > > > >>>>>>>
> >> > > > > > > >>>>>>>
> >> > > > > > > >>>>>>> Usability should be top most priority. But, since
> >> either
> >> > a
> >> > > > > > reviewer
> >> > > > > > > >>>> or
> >> > > > > > > >>>>> pr
> >> > > > > > > >>>>>>> author can trigger the bot, is it really a hurdle
> for
> >> pr
> >> > > > author
> >> > > > > > or
> >> > > > > > > >>>>> reviewer
> >> > > > > > > >>>>>>> to call a bot to trigger CI? Given that PR author
> and
> >> > > > reviewer
> >> > > > > is
> >> > > > > > > >>>>> already
> >> > > > > > > >>>>>>> actively commenting various details such as - PR
> >> > > description,
> >> > > > > > > >>> review
> >> > > > > > > >>>>>>> comments and responses, adding labels etc.
> >> > > > > > > >>>>>>>
> >> > > > > > > >>>>>>>
> >> > > > > > > >>>>>>> Me too curious to know the behavior for Tao's above
> >> use
> >> > > case.
> >> > > > > > > >>>>>>>
> >> > > > > > > >>>>>>>
> >> > > > > > > >>>>>>> Best,
> >> > > > > > > >>>>>>>
> >> > > > > > > >>>>>>> Sandeep
> >> > > > > > > >>>>>>>
> >> > > > > > > >>>>>>> On Thu, Mar 12, 2020 at 7:18 PM Tao Lv <
> >> > mutouorz@gmail.com
> >> > > >
> >> > > > > > wrote:
> >> > > > > > > >>>>>>>
> >> > > > > > > >>>>>>>> Is it possible for re-triggering a single job to be
> >> > > abused?
> >> > > > > For
> >> > > > > > > >>>>> example,
> >> > > > > > > >>>>>>>> the author spends two days re-triggering a flaky
> job
> >> to
> >> > > make
> >> > > > > it
> >> > > > > > > >>>>> pass. But
> >> > > > > > > >>>>>>>> other jobs which have passed the validation may be
> >> > broken
> >> > > by
> >> > > > > > > >>> other
> >> > > > > > > >>>>>>> commits
> >> > > > > > > >>>>>>>> during the two day without being noticed. And
> finally
> >> > the
> >> > > PR
> >> > > > > is
> >> > > > > > > >>>>> merged
> >> > > > > > > >>>>>>> with
> >> > > > > > > >>>>>>>> underlying problems.
> >> > > > > > > >>>>>>>>
> >> > > > > > > >>>>>>>> On Fri, Mar 13, 2020 at 6:19 AM Marco de Abreu <
> >> > > > > > > >>>>> marco.g.abreu@gmail.com>
> >> > > > > > > >>>>>>>> wrote:
> >> > > > > > > >>>>>>>>
> >> > > > > > > >>>>>>>>> In the end it only comes down to money,
> considering
> >> > that
> >> > > > the
> >> > > > > > > >>>>> system is
> >> > > > > > > >>>>>>>> auto
> >> > > > > > > >>>>>>>>> scaling, making the execution time constant.
> >> > > > > > > >>>>>>>>>
> >> > > > > > > >>>>>>>>> If we're trading money for usability, I certainly
> >> would
> >> > > > > prefer
> >> > > > > > > >>>>>>> usability.
> >> > > > > > > >>>>>>>>> I'd rather recommend to spend time on
> parallelizing
> >> > test
> >> > > > > > > >>>> execution
> >> > > > > > > >>>>> or
> >> > > > > > > >>>>>>>>> getting rid of integration tests in the PR stage
> >> > instead
> >> > > > > > > >>> reducing
> >> > > > > > > >>>>> the
> >> > > > > > > >>>>>>>> costs
> >> > > > > > > >>>>>>>>> by making people not use it. But taking a step
> back
> >> to
> >> > > > > > > >>> requiring
> >> > > > > > > >>>>> people
> >> > > > > > > >>>>>>>> to
> >> > > > > > > >>>>>>>>> manually trigger CI again doesn't feel right.
> >> > > > > > > >>>>>>>>>
> >> > > > > > > >>>>>>>>> I'm happy to see that bot deployed, but I do not
> >> agree
> >> > > with
> >> > > > > > > >>>>> removing
> >> > > > > > > >>>>>>> the
> >> > > > > > > >>>>>>>>> auto trigger functionality for new commits.
> >> > > > > > > >>>>>>>>>
> >> > > > > > > >>>>>>>>> -Marco
> >> > > > > > > >>>>>>>>>
> >> > > > > > > >>>>>>>>> Chaitanya Bapat <chai.bapat@gmail.com> schrieb am
> >> Do.,
> >> > > 12.
> >> > > > > > > >>> März
> >> > > > > > > >>>>> 2020,
> >> > > > > > > >>>>>>>>> 22:47:
> >> > > > > > > >>>>>>>>>
> >> > > > > > > >>>>>>>>>> @Marco Thanks for pointing that out.
> >> > > > > > > >>>>>>>>>> Tomorrow i.e. Friday, March 13, 2020 at 3:00 PM -
> >> 3:30
> >> > > PM
> >> > > > in
> >> > > > > > > >>>>>>>> (UTC-08:00)
> >> > > > > > > >>>>>>>>>> Pacific Time (US & Canada).
> >> > > > > > > >>>>>>>>>>
> >> > > > > > > >>>>>>>>>>> When do we expect this bot to be deployed?
> >> > > > > > > >>>>>>>>>> @Lin If all goes well in the next week I can
> >> deploy it
> >> > > to
> >> > > > > > > >>>> public
> >> > > > > > > >>>>>>> Apache
> >> > > > > > > >>>>>>>>>> (provided I get permissions from Apache Infra)
> >> > > > > > > >>>>>>>>>>
> >> > > > > > > >>>>>>>>>> @Marco Thanks for your feedback.
> >> > > > > > > >>>>>>>>>>> CI system has to support the community without
> >> > > requiring
> >> > > > > > > >>>>> people to
> >> > > > > > > >>>>>>>>>> constantly shepherd every single run
> >> > > > > > > >>>>>>>>>> We have data for the number of times CI was
> >> triggered
> >> > > > > > > >>>>> unnecessarily
> >> > > > > > > >>>>>>>> which
> >> > > > > > > >>>>>>>>>> includes
> >> > > > > > > >>>>>>>>>> - Entire build triggered instead of specific
> build
> >> > > > > > > >>>>>>>>>> - CI triggered when PR is still work in progress
> or
> >> > not
> >> > > > yet
> >> > > > > > > >>>> ready
> >> > > > > > > >>>>>>> (say
> >> > > > > > > >>>>>>>> -
> >> > > > > > > >>>>>>>>>> intermediate commits)
> >> > > > > > > >>>>>>>>>> At the end its a trade-off
> >> > > > > > > >>>>>>>>>> Money, Resources, Time to build for each and
> every
> >> > > commit
> >> > > > vs
> >> > > > > > > >>>>> Pain of
> >> > > > > > > >>>>>>>>>> triggering builds
> >> > > > > > > >>>>>>>>>>
> >> > > > > > > >>>>>>>>>>
> >> > > > > > > >>>>>>>>>>> Scan trigger plugin would poll SCM. Can we use
> >> plugin
> >> > > at
> >> > > > > > > >>>>> scale?
> >> > > > > > > >>>>>>>>>>
> >> > > > > > > >>>>>>>>>> 1. I haven't tested it on scale. But I think with
> >> the
> >> > > > > current
> >> > > > > > > >>>>> scale
> >> > > > > > > >>>>>>> of
> >> > > > > > > >>>>>>>>>> MXNet repo (191 open PRs i.e. checking for
> changes
> >> to
> >> > > 191
> >> > > > > > > >>>>> branches -
> >> > > > > > > >>>>>>> It
> >> > > > > > > >>>>>>>>>> should be manageable)
> >> > > > > > > >>>>>>>>>> 2. What's the purpose of the plugin? tldr; Branch
> >> > > > discovery
> >> > > > > > > >>> or
> >> > > > > > > >>>>> branch
> >> > > > > > > >>>>>>>>>> indexing.
> >> > > > > > > >>>>>>>>>> Scan trigger plugin comes into the picture only
> >> once
> >> > per
> >> > > > PR
> >> > > > > > > >>> per
> >> > > > > > > >>>>> job
> >> > > > > > > >>>>>>>>> (i.e. 8
> >> > > > > > > >>>>>>>>>> times per PR for 8 jobs). It is basically done
> >> when a
> >> > > new
> >> > > > PR
> >> > > > > > > >>> is
> >> > > > > > > >>>>> made
> >> > > > > > > >>>>>>>> and
> >> > > > > > > >>>>>>>>>> the job (say unix-cpu hasn't discovered the new
> PR
> >> > > branch
> >> > > > > > > >>> yet).
> >> > > > > > > >>>>>>> That's
> >> > > > > > > >>>>>>>>> it.
> >> > > > > > > >>>>>>>>>> So it shouldn't be a problem for public MXNet
> repo.
> >> > > > > > > >>>>>>>>>>
> >> > > > > > > >>>>>>>>>> Thanks,
> >> > > > > > > >>>>>>>>>> Chai
> >> > > > > > > >>>>>>>>>>
> >> > > > > > > >>>>>>>>>>
> >> > > > > > > >>>>>>>>>> On Thu, 12 Mar 2020 at 14:22, Marco de Abreu <
> >> > > > > > > >>>>>>> marco.g.abreu@gmail.com>
> >> > > > > > > >>>>>>>>>> wrote:
> >> > > > > > > >>>>>>>>>>
> >> > > > > > > >>>>>>>>>>> Btw you forgot to set a date and time for the
> >> metting
> >> > > > > > > >>>>>>>>>>>
> >> > > > > > > >>>>>>>>>>>
> >> > > > > > > >>>>>>>>>>> On Thu, Mar 12, 2020 at 10:18 PM Marco de Abreu
> <
> >> > > > > > > >>>>>>>>> marco.g.abreu@gmail.com
> >> > > > > > > >>>>>>>>>>> wrote:
> >> > > > > > > >>>>>>>>>>>
> >> > > > > > > >>>>>>>>>>>> Thanks Chai, I generally like the idea of the
> >> bot.
> >> > But
> >> > > > > > > >>> I'm
> >> > > > > > > >>>>> not a
> >> > > > > > > >>>>>>>>>>> supporter
> >> > > > > > > >>>>>>>>>>>> of the idea to disable any automatic triggering
> >> > > > > > > >>> (disabling
> >> > > > > > > >>>>> the
> >> > > > > > > >>>>>>>>> webhook
> >> > > > > > > >>>>>>>>>> is
> >> > > > > > > >>>>>>>>>>>> also not an option, considering that this will
> >> > disable
> >> > > > > > > >>>> master
> >> > > > > > > >>>>>>>>>> triggers).
> >> > > > > > > >>>>>>>>>>>> The CI system has to support the community
> >> without
> >> > > > > > > >>>> requiring
> >> > > > > > > >>>>>>> people
> >> > > > > > > >>>>>>>>> to
> >> > > > > > > >>>>>>>>>>>> constantly shepherd every single run. Disabling
> >> > > > automatic
> >> > > > > > > >>>>>>>> triggering
> >> > > > > > > >>>>>>>>>>> seems
> >> > > > > > > >>>>>>>>>>>> like a step back to me.
> >> > > > > > > >>>>>>>>>>>>
> >> > > > > > > >>>>>>>>>>>> Instead, I'd recommend that CI gets triggered
> >> upon
> >> > > every
> >> > > > > > > >>>>> commit
> >> > > > > > > >>>>>>> as
> >> > > > > > > >>>>>>>>>> usual,
> >> > > > > > > >>>>>>>>>>>> but people have the possibility to call a
> >> "command"
> >> > > > (i.e.
> >> > > > > > > >>>>> make a
> >> > > > > > > >>>>>>>>>> message
> >> > > > > > > >>>>>>>>>>>> which results in the bot setting a label) to
> >> disable
> >> > > CI
> >> > > > > > > >>>> until
> >> > > > > > > >>>>>>> they
> >> > > > > > > >>>>>>>>>> revoke
> >> > > > > > > >>>>>>>>>>>> it. But the standard should still be that a new
> >> > commit
> >> > > > > > > >>>>> triggers a
> >> > > > > > > >>>>>>>> new
> >> > > > > > > >>>>>>>>>> CI
> >> > > > > > > >>>>>>>>>>>> run.
> >> > > > > > > >>>>>>>>>>>>
> >> > > > > > > >>>>>>>>>>>>
> >> > > > > > > >>>>
> >> > https://plugins.jenkins.io/multibranch-scan-webhook-trigger/
> >> > > > > > > >>>>>>>> seems
> >> > > > > > > >>>>>>>>>> like
> >> > > > > > > >>>>>>>>>>>> this would poll SCM. This will incur high quota
> >> > > > > > > >>>>> restrictions. Are
> >> > > > > > > >>>>>>>> you
> >> > > > > > > >>>>>>>>>>> sure
> >> > > > > > > >>>>>>>>>>>> that you can use that plugin at scale?
> >> > > > > > > >>>>>>>>>>>>
> >> > > > > > > >>>>>>>>>>>> -Marco
> >> > > > > > > >>>>>>>>>>>>
> >> > > > > > > >>>>>>>>>>>>
> >> > > > > > > >>>>>>>>>>>> On Thu, Mar 12, 2020 at 10:04 PM Lin Yuan <
> >> > > > > > > >>>>> apeforest@gmail.com>
> >> > > > > > > >>>>>>>>> wrote:
> >> > > > > > > >>>>>>>>>>>>> Chai,
> >> > > > > > > >>>>>>>>>>>>>
> >> > > > > > > >>>>>>>>>>>>> Awesome work. When do we expect this bot to be
> >> > > > > > > >>> deployed?
> >> > > > > > > >>>>>>>>>>>>>
> >> > > > > > > >>>>>>>>>>>>> Best,
> >> > > > > > > >>>>>>>>>>>>>
> >> > > > > > > >>>>>>>>>>>>> Lin
> >> > > > > > > >>>>>>>>>>>>>
> >> > > > > > > >>>>>>>>>>>>> On Thu, Mar 12, 2020 at 2:00 PM Chaitanya
> Bapat
> >> <
> >> > > > > > > >>>>>>>>> chai.bapat@gmail.com
> >> > > > > > > >>>>>>>>>>>>> wrote:
> >> > > > > > > >>>>>>>>>>>>>
> >> > > > > > > >>>>>>>>>>>>>> Hello MXNet community,
> >> > > > > > > >>>>>>>>>>>>>>
> >> > > > > > > >>>>>>>>>>>>>> I have built an MXNet Bot <
> >> > > > > > > >>>> https://github.com/mxnet-bot>
> >> > > > > > > >>>>> that
> >> > > > > > > >>>>>>>>>> allows
> >> > > > > > > >>>>>>>>>>> PR
> >> > > > > > > >>>>>>>>>>>>>> Authors, Committers and Jenkins Admins to
> >> trigger
> >> > CI
> >> > > > > > > >>>>> manually.
> >> > > > > > > >>>>>>>>>>>>>> It handles 2 problems
> >> > > > > > > >>>>>>>>>>>>>> 1. Manual CI trigger instead of existing
> >> automated
> >> > > CI
> >> > > > > > > >>>>> trigger
> >> > > > > > > >>>>>>>>>>>>>> 2. Gives permissions to PR Authors (in
> >> addition to
> >> > > > > > > >>>> MXNet
> >> > > > > > > >>>>>>>>> Committers
> >> > > > > > > >>>>>>>>>>> and
> >> > > > > > > >>>>>>>>>>>>>> Jenkins Admins)
> >> > > > > > > >>>>>>>>>>>>>>
> >> > > > > > > >>>>>>>>>>>>>> Design Doc :
> >> > > > > > > >>>>>>>>>>>>>>
> >> > > > > > > >>>>>>>
> >> > > > https://cwiki.apache.org/confluence/display/MXNET/MXNet+CI+Bot
> >> > > > > > > >>>>>>>>>>>>>> I urge you all to attend the demonstration
> >> meeting
> >> > > > > > > >>> and
> >> > > > > > > >>>>> lend
> >> > > > > > > >>>>>>> your
> >> > > > > > > >>>>>>>>>> views
> >> > > > > > > >>>>>>>>>>>>> on
> >> > > > > > > >>>>>>>>>>>>>> the same.
> >> > > > > > > >>>>>>>>>>>>>>
> >> > > > > > > >>>>>>>>>>>>>> Thank you,
> >> > > > > > > >>>>>>>>>>>>>> Chai
> >> > > > > > > >>>>>>>>>>>>>>
> >> > > > > > > >>>>>>>>>>>>>> *Meeting Details*:
> >> > > > > > > >>>>>>>>>>>>>> ==============Conference Bridge
> >> > > > > > > >>>> Information==============
> >> > > > > > > >>>>>>>>>>>>>> You have been invited to an online meeting,
> >> > powered
> >> > > > > > > >>> by
> >> > > > > > > >>>>> Amazon
> >> > > > > > > >>>>>>>>> Chime.
> >> > > > > > > >>>>>>>>>>>>>> *Chime meeting ID*: *9272158344*
> >> > > > > > > >>>>>>>>>>>>>> Join via Chime clients (manually): Select
> >> > 'Meetings
> >> > > >
> >> > > > > > > >>>>> Join a
> >> > > > > > > >>>>>>>>>> Meeting',
> >> > > > > > > >>>>>>>>>>>>> and
> >> > > > > > > >>>>>>>>>>>>>> enter 9272158344
> >> > > > > > > >>>>>>>>>>>>>> Join via Chime clients (auto-call): If you
> >> invite
> >> > > > > > > >>>>> auto-call as
> >> > > > > > > >>>>>>>>>>> attendee,
> >> > > > > > > >>>>>>>>>>>>>> Chime will call you when the meeting starts,
> >> > select
> >> > > > > > > >>>>> 'Answer'
> >> > > > > > > >>>>>>>>>>>>>> *Join via browser screen share*:
> >> > > > > > > >>>>> https://chime.aws/9272158344
> >> > > > > > > >>>>>>>>>>>>>> *Join via phone* (US):
> >> > +1-929-432-4463,,,9272158344#
> >> > > > > > > >>>>>>>>>>>>>> *Join via phone (US toll-free)*:
> >> > > > > > > >>>>> +1-855-552-4463,,,9272158344#
> >> > > > > > > >>>>>>>>>>>>>> International dial-in:
> >> > > > > > > >>>> https://chime.aws/dialinnumbers/
> >> > > > > > > >>>>>>>>>>>>>> In-room video system: Ext: 62000, Meeting
> PIN:
> >> > > > > > > >>>>> 9272158344#
> >> > > > > > > >>>>>>>>>>>>>>
> >> > > > > > > >>>>>>>>>>>>>> --
> >> > > > > > > >>>>>>>>>>>>>> *Chaitanya Prakash Bapat*
> >> > > > > > > >>>>>>>>>>>>>> *+1 (973) 953-6299*
> >> > > > > > > >>>>>>>>>>>>>>
> >> > > > > > > >>>>>>>>>>>>>> [image:
> >> https://www.linkedin.com//in/chaibapat25]
> >> > > > > > > >>>>>>>>>>>>>> <https://github.com/ChaiBapchya>[image:
> >> > > > > > > >>>>>>>>>>>>> https://www.facebook.com/chaibapat
> >> > > > > > > >>>>>>>>>>>>>> ]
> >> > > > > > > >>>>>>>>>>>>>> <https://www.facebook.com/chaibapchya
> >[image:
> >> > > > > > > >>>>>>>>>>>>>> https://twitter.com/ChaiBapchya] <
> >> > > > > > > >>>>>>>> https://twitter.com/ChaiBapchya
> >> > > > > > > >>>>>>>>>>>>>> [image:
> >> > > > > > > >>>>>>>>>>>>>> https://www.linkedin.com//in/chaibapat25]
> >> > > > > > > >>>>>>>>>>>>>> <https://www.linkedin.com//in/chaibapchya/>
> >> > > > > > > >>>>>>>>>>>>>>
> >> > > > > > > >>>>>>>>>>
> >> > > > > > > >>>>>>>>>> --
> >> > > > > > > >>>>>>>>>> *Chaitanya Prakash Bapat*
> >> > > > > > > >>>>>>>>>> *+1 (973) 953-6299*
> >> > > > > > > >>>>>>>>>>
> >> > > > > > > >>>>>>>>>> [image: https://www.linkedin.com//in/chaibapat25
> ]
> >> > > > > > > >>>>>>>>>> <https://github.com/ChaiBapchya>[image:
> >> > > > > > > >>>>>>>>> https://www.facebook.com/chaibapat
> >> > > > > > > >>>>>>>>>> ]
> >> > > > > > > >>>>>>>>>> <https://www.facebook.com/chaibapchya>[image:
> >> > > > > > > >>>>>>>>>> https://twitter.com/ChaiBapchya] <
> >> > > > > > > >>>>> https://twitter.com/ChaiBapchya
> >> > > > > > > >>>>>>>>>> [image:
> >> > > > > > > >>>>>>>>>> https://www.linkedin.com//in/chaibapat25]
> >> > > > > > > >>>>>>>>>> <https://www.linkedin.com//in/chaibapchya/>
> >> > > > > > > >>>>>>>>>>
> >> > > > > > > >>>>>>>
> >> > > > > > > >>>>>>> --
> >> > > > > > > >>>>>>> Sandeep Krishnamurthy
> >> > > > > > > >>>>>>>
> >> > > > > > > >>>>>
> >> > > > > > > >>>>
> >> > > > > > > >>>
> >> > > > > > > >>>
> >> > > > > > > >>> --
> >> > > > > > > >>> *Chaitanya Prakash Bapat*
> >> > > > > > > >>> *+1 (973) 953-6299*
> >> > > > > > > >>>
> >> > > > > > > >>> [image: https://www.linkedin.com//in/chaibapat25]
> >> > > > > > > >>> <https://github.com/ChaiBapchya>[image:
> >> > > > > > > https://www.facebook.com/chaibapat
> >> > > > > > > >>> ]
> >> > > > > > > >>> <https://www.facebook.com/chaibapchya>[image:
> >> > > > > > > >>> https://twitter.com/ChaiBapchya] <
> >> > > > https://twitter.com/ChaiBapchya
> >> > > > > > > >[image:
> >> > > > > > > >>> https://www.linkedin.com//in/chaibapat25]
> >> > > > > > > >>> <https://www.linkedin.com//in/chaibapchya/>
> >> > > > > > > >>>
> >> > > > > > > >>
> >> > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > > >
> >> > > > > --
> >> > > > > *Chaitanya Prakash Bapat*
> >> > > > > *+1 (973) 953-6299*
> >> > > > >
> >> > > > > [image: https://www.linkedin.com//in/chaibapat25]
> >> > > > > <https://github.com/ChaiBapchya>[image:
> >> > > > https://www.facebook.com/chaibapat
> >> > > > > ]
> >> > > > > <https://www.facebook.com/chaibapchya>[image:
> >> > > > > https://twitter.com/ChaiBapchya] <
> https://twitter.com/ChaiBapchya
> >> > > > >[image:
> >> > > > > https://www.linkedin.com//in/chaibapat25]
> >> > > > > <https://www.linkedin.com//in/chaibapchya/>
> >> > > > >
> >> > > >
> >> > >
> >> > >
> >> > > --
> >> > > *Chaitanya Prakash Bapat*
> >> > > *+1 (973) 953-6299*
> >> > >
> >> > > [image: https://www.linkedin.com//in/chaibapat25]
> >> > > <https://github.com/ChaiBapchya>[image:
> >> > https://www.facebook.com/chaibapat
> >> > > ]
> >> > > <https://www.facebook.com/chaibapchya>[image:
> >> > > https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya
> >> > >[image:
> >> > > https://www.linkedin.com//in/chaibapat25]
> >> > > <https://www.linkedin.com//in/chaibapchya/>
> >> > >
> >> >
> >>
> >>
> >> --
> >> *Chaitanya Prakash Bapat*
> >> *+1 (973) 953-6299*
> >>
> >> [image: https://www.linkedin.com//in/chaibapat25]
> >> <https://github.com/ChaiBapchya>[image:
> >> https://www.facebook.com/chaibapat]
> >> <https://www.facebook.com/chaibapchya>[image:
> >> https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya
> >[image:
> >> https://www.linkedin.com//in/chaibapat25]
> >> <https://www.linkedin.com//in/chaibapchya/>
> >>
> >
>


-- 
*Chaitanya Prakash Bapat*
*+1 (973) 953-6299*

[image: https://www.linkedin.com//in/chaibapat25]
<https://github.com/ChaiBapchya>[image: https://www.facebook.com/chaibapat]
<https://www.facebook.com/chaibapchya>[image:
https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya>[image:
https://www.linkedin.com//in/chaibapat25]
<https://www.linkedin.com//in/chaibapchya/>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message