mxnet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marco de Abreu <marco.g.ab...@googlemail.com>
Subject Re: Commiter access to Jenkins Sevrer
Date Sat, 06 Jan 2018 12:15:34 GMT
While compile errors may be reproduced that way, it's hard to run any tests
due to the required softlink to the compiled binaries. It is possible, but
very inconvenient to use and thus it renders reproducing test results very
hard.

I have been thinking about giving the possibility to download the generated
artefacts during build stage - for debugging reasons only! This way, they
can be used in conjunction with unit tests to reproduce a test failure, but
this still needs some discussions and a security review.

-Marco

Am 06.01.2018 12:59 nachm. schrieb "kellen sunderland" <
kellen.sunderland@gmail.com>:

> Regarding the comments around reproducibility, what parts of the CI are
> people having trouble reproducing now?  I'm in favour of making our AMIs
> public for transparency reasons (and so that people can provide suggestions
> on how to improve them), but I'm not sure it would help in terms of
> reproducibility for any system other than Windows.  When I have an error in
> CI I generally just do a `make clean` in my mxnet root source dire, then
> copy the failing command from CI, i.e. `tests/ci_build/ci_build.sh cpu
> --dockerbinary docker make DEV=1 USE_PROFILER=1 USE_CPP_PACKAGE=1
> USE_BLAS=openblas -j$(nproc)`.  Are there CI tasks (other than Windows)
> that don't work for people?  If so maybe we can help fix those?
>
>
> On Sat, Jan 6, 2018 at 11:50 AM, kellen sunderland <
> kellen.sunderland@gmail.com> wrote:
>
> > +1, thanks for the work Marco.
> >
> > On Sat, Jan 6, 2018 at 12:24 AM, Naveen Swamy <mnnaveen@gmail.com>
> wrote:
> >
> >> this sounds fine to me as long as there is at least one MXNet committer
> >> who
> >> is also an admin.
> >>
> >> thanks Marco for making this happen :)
> >>
> >>  - Naveen
> >>
> >> On Fri, Jan 5, 2018 at 2:54 PM, Marco de Abreu <
> >> marco.g.abreu@googlemail.com
> >> > wrote:
> >>
> >> > I'm proposing following permissions: https://i.imgur.com/uiFBtuW.png.
> >> The
> >> > meaning of every permission is explained at https://wiki.jenkins.io/
> >> > display/JENKINS/Matrix-based+security.
> >> >
> >> > Any objections?
> >> >
> >> > On Fri, Jan 5, 2018 at 11:03 PM, Marco de Abreu <
> >> > marco.g.abreu@googlemail.com> wrote:
> >> >
> >> > > I'm currently working on a prototype of SSO based on GitHub and a
> few
> >> > > issues arose:
> >> > >
> >> > > We are not able to use the permission strategy which determines the
> >> > access
> >> > > rights based on the read/write permission to a project as the
> >> > > Jenkins-plugin is not able to resolve the link between Jenkins-jobs
> >> and
> >> > > GitHub-repositories. Instead I would propose to use a role-based
> >> approach
> >> > > using https://wiki.jenkins.io/display/JENKINS/Role+Strategy+Plugin.
> >> In
> >> > > this case we would have three roles: Anonymous, Administrator and
> >> > > Committer. While everybody would authenticate using their regular
> >> GitHub
> >> > > account, the role assignment would have to happen manually.
> >> Considering
> >> > > that the amount of administrators and committers doesn't change that
> >> > > frequently, this shouldn't be too much of an issue - auto populating
> >> the
> >> > > status is not possible unfortunately.
> >> > >
> >> > > Reason for splitting Administrators and Committers into two separate
> >> > roles
> >> > > has a security reason. At the moment, we're using Chris Oliviers
> >> GitHub
> >> > > credentials to populate the commit status. If all committers would
> >> gain
> >> > > full admin rights, they would have access to these credentials.
> Chris
> >> is
> >> > > not fine with this approach and would like to limit the amount of
> >> people
> >> > > with access to his credentials as much as possible.
> >> > >
> >> > > In order to address his concerns, I propose to add Chris to the
> >> committer
> >> > > as well as to the admin role, while all other committers will only
> >> > receive
> >> > > the committer role without read access to the credentials. In a
> later
> >> > > email, I will make a proposal for the detailed committer role
> rights.
> >> You
> >> > > can check all available options at https://wiki.jenkins.io/
> >> > > display/JENKINS/Matrix-based+security.
> >> > >
> >> > > All people who have access to the underlying AWS account would be
> >> granted
> >> > > the Administrator role with full access. At the moment, this would
> be
> >> > > Meghna Baijal, Gautam Kumar and myself.
> >> > >
> >> > > An alternative solution would be to create a bot account
> specifically
> >> for
> >> > > MXNet CI and use its credentials instead of Chris'. This account
> >> requires
> >> > > write permission to the repository, but would give us the advantage
> >> that
> >> > > these credentials would be shared within the committers and thus
> >> making
> >> > the
> >> > > restrictions regarding credentials obsolete (and Chris would be
> happy
> >> not
> >> > > the see his face within every single PR :P ). I've asked around and
> >> > > received the feedback from multiple people that Apache Infra does
> not
> >> > want
> >> > > to grant bot accounts write permission to a repository, but I would
> >> like
> >> > to
> >> > > confirm back considering that AppVeyor, for example, has a bot
> account
> >> > with
> >> > > write permission. I would like to check back with a mentor and
> create
> >> an
> >> > > Apache Infra ticket to request details and permission.
> >> > >
> >> > > I would propose to take both approaches at the same time, meaning
we
> >> can
> >> > > start with Chris in the committer AND admin role while trying to get
> >> > > permission for a bot account in the meantime.
> >> > >
> >> > > wdyt?
> >> > >
> >> > > On Fri, Jan 5, 2018 at 8:21 PM, Chris Olivier <
> cjolivier01@gmail.com>
> >> > > wrote:
> >> > >
> >> > >> I am fine without a vote unless a vote is required?  Any
> objections,
> >> > >> anyone?  You're sort of adding functionality here, not changing
or
> >> > >> restricting...  We can always change to Apache later.
> >> > >>
> >> > >> On Fri, Jan 5, 2018 at 11:18 AM, Marco de Abreu <
> >> > >> marco.g.abreu@googlemail.com> wrote:
> >> > >>
> >> > >> > I'd be in favour of GitHub. Shall we open a vote or would
you
> like
> >> me
> >> > to
> >> > >> > create a POC with GitHub first and afterwards we can check
if
> >> that's
> >> > >> > enough?
> >> > >> >
> >> > >> > -Marco
> >> > >> >
> >> > >> > On Fri, Jan 5, 2018 at 8:13 PM, Chris Olivier <
> >> cjolivier01@gmail.com>
> >> > >> > wrote:
> >> > >> >
> >> > >> > > Apparently Apache supports OATH, so I am open to either.
> >> > >> > > Good idea for the docker thing.
> >> > >> > >
> >> > >> > > On Fri, Jan 5, 2018 at 11:02 AM, Marco de Abreu <
> >> > >> > > marco.g.abreu@googlemail.com> wrote:
> >> > >> > >
> >> > >> > > > GitHub SSO allows the neat feature that login and
permission
> >> can
> >> > be
> >> > >> > > > selected depending on the access rights a user
has to a
> >> project.
> >> > >> > Somebody
> >> > >> > > > with write access (committers) would be get different
> >> permissions
> >> > >> than
> >> > >> > > > somebody with only read access.
> >> > >> > > >
> >> > >> > > > We could check back with Apache for SSO, but this
would
> involve
> >> > >> Apache
> >> > >> > > > infra. We could put it up to a vote whether to
use GitHub or
> >> > Apache
> >> > >> > SSO.
> >> > >> > > >
> >> > >> > > > In order to reproduce a build failure we have been
thinking
> >> about
> >> > >> > > changing
> >> > >> > > > the ci_build.sh in such a way that it can be run
manually
> >> without
> >> > >> > > Jenkins.
> >> > >> > > > The setup I took over binds the Jenkins work directory
into
> the
> >> > >> docker
> >> > >> > > > containers and uses a few hacks which are hard
to reproduce
> >> > >> locally. We
> >> > >> > > > plan to reengineer this script to make it easier
to run
> >> manually.
> >> > >> > > > But making the AMI public is a good idea! We plan
to make the
> >> > whole
> >> > >> > > > infrastructure code (based on Terraform) completely
public -
> at
> >> > the
> >> > >> > > moment
> >> > >> > > > it's in a private repository as it contains credentials,
but
> >> they
> >> > >> will
> >> > >> > be
> >> > >> > > > moved to KMS soon. It would definitely be a good
approach to
> >> just
> >> > >> > supply
> >> > >> > > > the AMI so everybody could recreate the environment
in their
> >> own
> >> > >> > account.
> >> > >> > > >
> >> > >> > > > -Marco
> >> > >> > > >
> >> > >> > > > Am 05.01.2018 7:51 nachm. schrieb "Chris Olivier"
<
> >> > >> > cjolivier01@gmail.com
> >> > >> > > >:
> >> > >> > > >
> >> > >> > > > Well, login to the Jenkins server, I would imagine.
> >> > >> > > >
> >> > >> > > > github or Apache SSO (does Apache support OAUTH?)
seems like
> a
> >> > good
> >> > >> > idea
> >> > >> > > as
> >> > >> > > > long as there's a way to not let everyone with
a github
> account
> >> > log
> >> > >> in.
> >> > >> > > >
> >> > >> > > > Access to actual slave machines could be more restricted,
I
> >> > imagine.
> >> > >> > > >
> >> > >> > > > Eventually, a public current AMI for a build slave
would be
> >> good
> >> > in
> >> > >> > order
> >> > >> > > > to reproduce build or test problems that can't
be reproduced
> >> > >> locally.
> >> > >> > > >
> >> > >> > > > wdyt?
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > > On Fri, Jan 5, 2018 at 10:41 AM, Marco de Abreu
<
> >> > >> > > > marco.g.abreu@googlemail.com> wrote:
> >> > >> > > >
> >> > >> > > > > Would it be an acceptable solution if we add
SSO or do you
> >> also
> >> > >> want
> >> > >> > > > access
> >> > >> > > > > to the actual AWS account and all machines?
> >> > >> > > > >
> >> > >> > > > > Yes, the build jobs are automatically getting
created for
> new
> >> > >> > branches.
> >> > >> > > > >
> >> > >> > > > > -Marco
> >> > >> > > > >
> >> > >> > > > > Am 05.01.2018 7:35 nachm. schrieb "Marco de
Abreu" <
> >> > >> > > > > marco.g.abreu@googlemail.com>:
> >> > >> > > > >
> >> > >> > > > > I totally agree, this is not the way it should
work in an
> >> Apache
> >> > >> > > Project.
> >> > >> > > > > It's running on an isengard account, meaning
it is only
> >> > accessible
> >> > >> > for
> >> > >> > > > > Amazon employees. The problem is that a compromised
account
> >> > could
> >> > >> > cause
> >> > >> > > > > damage up to 170,000$ per day. There are alarms
in place to
> >> > notice
> >> > >> > > those
> >> > >> > > > > cases, but we still have to be very careful.
These high
> >> limits
> >> > >> have
> >> > >> > > been
> >> > >> > > > > chosen due to auto scaling being added within
the next
> >> week's.
> >> > >> > > > >
> >> > >> > > > > I'd be happy to introduce a committer into
the CI process
> and
> >> > all
> >> > >> the
> >> > >> > > > > necessary steps as well as granting them permission.
The
> only
> >> > >> > > restriction
> >> > >> > > > > being that it has to be and Amazon employee
and access to
> >> > console,
> >> > >> > > master
> >> > >> > > > > and slave only being possible from the Corp
network.
> >> > >> > > > >
> >> > >> > > > > There is no open ticket. What would you like
to request?
> >> > >> > > > >
> >> > >> > > > > -Marco
> >> > >> > > > >
> >> > >> > > > >
> >> > >> > > > > Am 05.01.2018 7:22 nachm. schrieb "Chris Olivier"
<
> >> > >> > > cjolivier01@gmail.com
> >> > >> > > > >:
> >> > >> > > > >
> >> > >> > > > > Like John and other mentors were saying, it's
not proper
> for
> >> CI
> >> > to
> >> > >> > be a
> >> > >> > > > > closed/inaccessible environment.  Is it running
on an
> >> Isengard
> >> > >> > account
> >> > >> > > or
> >> > >> > > > > in PROD or CORP or just generic EC2?  I think
that we
> should
> >> > >> remedy
> >> > >> > > this.
> >> > >> > > > > It's very strange that no committers have
access at all.
> Is
> >> > >> there a
> >> > >> > > > ticket
> >> > >> > > > > open to IPSEC?
> >> > >> > > > >
> >> > >> > > > > On Fri, Jan 5, 2018 at 10:17 AM, Marco de
Abreu <
> >> > >> > > > > marco.g.abreu@googlemail.com> wrote:
> >> > >> > > > >
> >> > >> > > > > > Hello Chris,
> >> > >> > > > > >
> >> > >> > > > > > At the moment this is not possible due
Amazon AppSec
> >> > >> (Application
> >> > >> > > > > security)
> >> > >> > > > > > restrictions which does not permit user
data and
> >> credentials
> >> > on
> >> > >> > these
> >> > >> > > > > > machines.
> >> > >> > > > > >
> >> > >> > > > > > I have been thinking about adding single
sign on bound to
> >> > >> GitHub,
> >> > >> > but
> >> > >> > > > we
> >> > >> > > > > > would have to check back with AppSec.
> >> > >> > > > > >
> >> > >> > > > > > Is the reason for your request still
the ability to start
> >> and
> >> > >> stop
> >> > >> > > > > running
> >> > >> > > > > > builds?
> >> > >> > > > > >
> >> > >> > > > > > Best regards,
> >> > >> > > > > > Marco
> >> > >> > > > > >
> >> > >> > > > > > Am 05.01.2018 7:11 nachm. schrieb "Chris
Olivier" <
> >> > >> > > > cjolivier01@gmail.com
> >> > >> > > > > >:
> >> > >> > > > > >
> >> > >> > > > > > Marco,
> >> > >> > > > > >
> >> > >> > > > > > Are all committers able to get login
access to the
> Jenkins
> >> > >> Server?
> >> > >> > > If
> >> > >> > > > > not,
> >> > >> > > > > > why?
> >> > >> > > > > >
> >> > >> > > > > > -Chris
> >> > >> > > > > >
> >> > >> > > > >
> >> > >> > > >
> >> > >> > >
> >> > >> >
> >> > >>
> >> > >
> >> > >
> >> >
> >>
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message