mxnet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pedro Larroy <pedro.larroy.li...@gmail.com>
Subject Re: Stopping nightly releases to Pypi
Date Thu, 09 Jan 2020 00:21:06 GMT
Thanks for your detailed responses.

Having codebuild execute the recipe that is the apache repository is the
same effect and control that you would have in some service such as travis
CI. And the builds are fully reproducible. So it's under full control of
Apache the same way that any other hosted build solution is. Any
modification to the recipe would be executed on next commit, and the builds
are fully reproducible. There's no configuration in code build that would
be outside of the Apache MXNet repository in this case, since pipeline and
the config would be under the git repo.

And as you rightly pointed out, the Jenkins master is a weak point to the
restricted slaves. This was strongly criticized during the system review
and there is precedent of security flaws in the master. Insisting on mixing
CI and CD is not a good recommendation for what it has been explained above.

Pedro.

On Wed, Jan 8, 2020 at 2:41 PM Marco de Abreu <marco.g.abreu@gmail.com>
wrote:

> Correct, I'm not bothered by the s3 bucket but by way how it gets
> published. It's not in Jenkins, so it's outside of the projects control.
>
> The security design due to the restricted nodes makes sure that no third
> party can gain access to these machines. They use separate caches, separate
> volumes, different instance profiles etc - I personally would consider the
> restricted slaves safe. If you're telling me that restricted slaves have
> been compromised with a crypto Miner, I'd be happy to discuss that matter
> and assist.
>
> Another attack vector is the Jenkins master, correct. If somebody
> infiltrates the Jenkins master, they can use that to jump onto the
> restricted slaves. They might modify the created artifacts, but once the
> system gets cleaned up, we're good to go again (You might rather want to
> consider a virus scan on the machines and created artifacts).
>
> But now let's say Jenkins master gets comprised. In that case, the
> artifacts are not the issue but the credentials. Jenkins contains committer
> credentials, which would allow to inject malware into our repository. Don't
> forget that a committer can add commits to other PRs, manually fake the CI
> status and then squash the PR to basically hide most of the traces. Unless
> someone reviews every single commit on master, we're basically out of luck.
>
> So yeah, that attack vector through the Jenkins master is valid, but
> considering that there are bigger risks involved in the system and the
> slaves themselves are pretty well protected, I'd not consider CD a severe
> issue in relation to the overall risk score of our system.
>
> So in order to make sure that we're well protected, I'd recommend to spend
> a bit of time on adapting the Jenkins pipeline to upload to s3 and then use
> all the remaining time to actually harden the Jenkins master and make sure
> that everything is constantly kept up to date. Security-wise, I'd consider
> that a way better investment than developing a new CD.
>
> -Marco
>
> Pedro Larroy <pedro.larroy.lists@gmail.com> schrieb am Mi., 8. Jan. 2020,
> 22:49:
>
> > Marco, if you are fine publishing to an S3 bucket, what's your concern?
> > using a codebuild pipeline? The build logs could be push to the s3 bucket
> > if this is your concern.
> >
> > As I said before, having binary releases in the current CI doesn't stand
> a
> > chance to pass security review as it is today, it's not safe and is a bad
> > idea, alternatives are
> > 1 -Code Build (you don't support this because it's company owned, did I
> > understand this correctly?)
> > 2 - Apache owned Jenkins (can you help with this?)
> > 3 - Travis CI or similar, which in the end is similar to code build.
> > 4- Another Jenkins just for CD (who owns?)
> >
> > Pedro.
> >
> > On Wed, Jan 8, 2020 at 1:01 PM Marco de Abreu <marco.g.abreu@gmail.com>
> > wrote:
> >
> > > The risk of the current CD via Jenkins is known and was accepted as
> part
> > of
> > > adopting Jenkins. The solution for the initial issue - no longer
> > publishing
> > > to pypi - is to add a step to the existing CD pipeline which publishes
> > the
> > > package to the s3 bucket instead of pypi.
> > >
> > > -Marco
> > >
> > > Pedro Larroy <pedro.larroy.lists@gmail.com> schrieb am Mi., 8. Jan.
> > 2020,
> > > 21:55:
> > >
> > > > I understand your point. But you don't provide an alternative, and
> > > building
> > > > binary releases from the CI jenkins as it is today is a very bad idea
> > > since
> > > > it's an unsafe environment. I think it's fair to ask if you are
> vetoing
> > > > using codebuild for nightly releases you could provide an alternative
> > > > solution (for example Apache hosted Jenkins) or anything else. As you
> > are
> > > > well aware non-committers can't communicate with Apache Infra or make
> > > > requests, so the onus is on you or other Apache person to provide a
> > > > solution that aligns with Apache values.
> > > >
> > > > So far I see Sam trying to help with codebuild managed binary
> releases
> > > and
> > > > this is taken as a tinfoil hat corporate conspiracy. It's a pity that
> > you
> > > > claim to endorse Apache values but not support what's best for the
> > > project,
> > > > which is to have things clean and in working order. I don't think
> users
> > > > care where the binary releases are hosted.
> > > >
> > > > Pedro.
> > > >
> > > > On Sun, Jan 5, 2020 at 5:56 AM Marco de Abreu <
> marco.g.abreu@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Apache only cares about source releases as far as official releases
> > are
> > > > > concerned. But Apache also cares about it's brand and image. You
> are
> > > > right
> > > > > that anybody can compile an Apache project and distribute it, but
> > it's
> > > > > under the PMCs control what can be advertised as official. This
> > > includes
> > > > > the following examples:
> > > > >
> > > > > - The official MXNet pypi, dockerhub, maven, etc account
> > > > > - The MXNet website
> > > > > - anything advertising to be MXNet
> > > > >
> > > > > If you publish a binary release and call it
> > "AwesomeSpaghettiBolognese"
> > > > > while it's MXNet under the hood, that's totally in line with the
> > Apache
> > > > > license. But if you decide to publish an MXNet branded package,
> then
> > > > that's
> > > > > covered by the brand protection. I won't go into much more detail
> > about
> > > > > legal reasons since that's not helping this discussion.
> > > > >
> > > > > I personally am vetoing a company-owned distribution channel to be
> > > > > advertised on the MXNet website or any official documentation.
> Also,
> > > I'd
> > > > > like to make sure that users do not mistake it for being a release
> > that
> > > > is
> > > > > affiliated or endorsed by Apache MXNet.
> > > > >
> > > > > We are taking a step back here and it's a pity to see that some
> > people
> > > > are
> > > > > still not endorsing the Apache values. This will be my last email
> > > > regarding
> > > > > that topic and I will only follow up with actions after the 15th of
> > > > January
> > > > > has been reached.
> > > > >
> > > > > Best regards
> > > > > Marco
> > > > >
> > > > >
> > > > > Pedro Larroy <pedro.larroy.lists@gmail.com> schrieb am Sa., 4.
> Jan.
> > > > 2020,
> > > > > 02:38:
> > > > >
> > > > > > Hey Marco.
> > > > > >
> > > > > > As far as I have learned from other Apache mailing lists while
> > > lurking
> > > > is
> > > > > > that Apache only cares about making source releases, binaries
> are a
> > > > > > courtesy to users that some projects decide to do, but I'm not
> > sure I
> > > > > > understand your concerns regarding the PMC and what exactly are
> you
> > > > > vetoing
> > > > > > here, since everyone can compile, build and package our project
> as
> > > per
> > > > > the
> > > > > > open source license. I would suggest to have a constructive
> > approach
> > > > and
> > > > > > see how we can make this happen for the best of the project,
> > > specially
> > > > > > since somebody is volunteering to help with this and dedicate
> > > valuable
> > > > > > compute resources and people's time.
> > > > > >
> > > > > > Regarding manual changes I don't see any need to have access to a
> > > code
> > > > > > build control plane for *anybody*, for several reasons, first is
> > that
> > > > > > manual access to production account is a discouraged practice and
> > are
> > > > > best
> > > > > > managed through pipeline deployments, second is that Code build
> is
> > a
> > > > > hosted
> > > > > > service which is basically just using a build description file to
> > do
> > > > the
> > > > > > work, there's no need to do any manual fiddling or triggering. If
> > all
> > > > the
> > > > > > CD and description files are in the apache repository you can use
> > > your
> > > > > own
> > > > > > account or compute resources to do your own build flavor if you
> so
> > > > > desire.
> > > > > >
> > > > > > Is your proposal to host this in Apache infrastructure?  Maybe
> I'm
> > > > > missing
> > > > > > something on this conversation
> > > > > >
> > > > > > Pedro.
> > > > > >
> > > > > >
> > > > > > On Fri, Jan 3, 2020 at 3:21 PM Marco de Abreu <
> > > marco.g.abreu@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Sam, while I understand that this solution was developed out of
> > > > > > necessity,
> > > > > > > my question why a new system has been developed instead of
> fixing
> > > the
> > > > > > > existing one or adapting the solution. CodeBuild is a scheduler
> > in
> > > > the
> > > > > > same
> > > > > > > fashion as Jenkins is. It runs code. So you can adapt it to
> > Jenkins
> > > > > > without
> > > > > > > much hassle.
> > > > > > >
> > > > > > > I'm not volunteering for this - why should I? The role of a PMC
> > > > member
> > > > > is
> > > > > > > to steer the direction of the project. Just because a manager
> > > points
> > > > > > > towards a certain direction, if doesn't mean that they're going
> > to
> > > do
> > > > > it.
> > > > > > >
> > > > > > > Apparently there was enough time at some point to develop a new
> > > > > solution
> > > > > > > from scratch. It might have been a solution for your internal
> > team
> > > > and
> > > > > > > that's fine, but upgrading it "temporarily" to be the
> advertised
> > > way
> > > > on
> > > > > > the
> > > > > > > official website is something different.
> > > > > > >
> > > > > > > I won't argue about how the veto can be enforced. I think it's
> in
> > > the
> > > > > > best
> > > > > > > interest of the project if we try working on a solution instead
> > of
> > > > > > spending
> > > > > > > time on trying to figure out the power of the PMC.
> > > > > > >
> > > > > > > Pedro, that's certainly a step towards the right direction. But
> > > > > > committers
> > > > > > > would also need access to the control plane of the system - to
> > > > trigger,
> > > > > > > stop and audit builds. We could go down that road, but i think
> > the
> > > > > fewer
> > > > > > > systems, the better - also for the sake of maintainability.
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Marco
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Pedro Larroy <pedro.larroy.lists@gmail.com> schrieb am Fr., 3.
> > > Jan.
> > > > > > 2020,
> > > > > > > 20:55:
> > > > > > >
> > > > > > > > I'm not involved in such efforts, but one possibility is to
> > have
> > > > the
> > > > > > yaml
> > > > > > > > files that describe the pipelines for CD in the Apache
> > > > repositories,
> > > > > > > would
> > > > > > > > that be acceptable from the Apache POV? In the end they
> should
> > be
> > > > > very
> > > > > > > thin
> > > > > > > > and calling the scripts that are part of the CD packages.
> > > > > > > >
> > > > > > > > On Fri, Jan 3, 2020 at 6:56 AM Marco de Abreu <
> > > > > marco.g.abreu@gmail.com
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Agree, but the question how a non Amazonian is able to
> > maintain
> > > > and
> > > > > > > > access
> > > > > > > > > the system is still open. As it stands right now, the
> > community
> > > > has
> > > > > > > > taken a
> > > > > > > > > step back and loses some control if we continue down that
> > road.
> > > > > > > > >
> > > > > > > > > I personally am disapproving of that approach since
> > committers
> > > > are
> > > > > no
> > > > > > > > > longer in control of that process. So far it seems like my
> > > > > questions
> > > > > > > were
> > > > > > > > > skipped and further actions have been taken. As openness
> and
> > > the
> > > > > > > > community
> > > > > > > > > having control are part of our graduation criteria, I'm
> > putting
> > > > in
> > > > > my
> > > > > > > > veto
> > > > > > > > > with a grace period until 15th of January. Please bring the
> > > > system
> > > > > > > into a
> > > > > > > > > state that aligns with Apache values or revert the changes.
> > > > > > > > >
> > > > > > > > > -Marco
> > > > > > > > >
> > > > > > > > > Pedro Larroy <pedro.larroy.lists@gmail.com> schrieb am
> Fr.,
> > 3.
> > > > > Jan.
> > > > > > > > 2020,
> > > > > > > > > 03:33:
> > > > > > > > >
> > > > > > > > > > CD should be separate from CI for security reasons in any
> > > case.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Sat, Dec 7, 2019 at 10:04 AM Marco de Abreu <
> > > > > > > > marco.g.abreu@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Could you elaborate how a non-Amazonian is able to
> > access,
> > > > > > maintain
> > > > > > > > and
> > > > > > > > > > > review the CodeBuild pipeline? How come we've diverted
> > from
> > > > the
> > > > > > > > > community
> > > > > > > > > > > agreed-on standard where the public Jenkins serves for
> > the
> > > > > > purpose
> > > > > > > of
> > > > > > > > > > > testing and releasing MXNet? I'd be curious about the
> > > issues
> > > > > > you're
> > > > > > > > > > > encountering with Jenkins CI that led to a non-standard
> > > > > solution.
> > > > > > > > > > >
> > > > > > > > > > > -Marco
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Skalicky, Sam <sskalic@amazon.com.invalid> schrieb am
> > Sa.,
> > > > 7.
> > > > > > Dez.
> > > > > > > > > 2019,
> > > > > > > > > > > 18:39:
> > > > > > > > > > >
> > > > > > > > > > > > Hi MXNet Community,
> > > > > > > > > > > >
> > > > > > > > > > > > We have been working on getting nightly builds fixed
> > and
> > > > made
> > > > > > > > > available
> > > > > > > > > > > > again. We’ve made another system using AWS CodeBuild
> &
> > S3
> > > > to
> > > > > > work
> > > > > > > > > > around
> > > > > > > > > > > > the problems with Jenkins CI, PyPI, etc. It is
> > currently
> > > > > > building
> > > > > > > > all
> > > > > > > > > > the
> > > > > > > > > > > > flavors and publishing to an S3 bucket here:
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://us-west-2.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/?region=us-west-2
> > > > > > > > > > > >
> > > > > > > > > > > > There are folders for each set of nightly builds, try
> > out
> > > > the
> > > > > > > > wheels
> > > > > > > > > > > > starting today 2019-12-07. Builds start at 1:30am PT
> > > > (9:30am
> > > > > > GMT)
> > > > > > > > and
> > > > > > > > > > > > arrive in the bucket 30min-2hours later. Inside each
> > > folder
> > > > > are
> > > > > > > the
> > > > > > > > > > > wheels
> > > > > > > > > > > > for each flavor of MXNet. Currently we’re only
> building
> > > for
> > > > > > > linux,
> > > > > > > > > > builds
> > > > > > > > > > > > for windows/Mac will come later.
> > > > > > > > > > > >
> > > > > > > > > > > > If you want to download the wheels easily you can
> use a
> > > URL
> > > > > in
> > > > > > > the
> > > > > > > > > form
> > > > > > > > > > > of:
> > > > > > > > > > > >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> <YYYY-MM-DD>/dist/<mxnet_build>-1.6.0b<YYYYMMDD>-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > > >
> > > > > > > > > > > > Heres a set of links for today’s builds
> > > > > > > > > > > >
> > > > > > > > > > > > (Plain mxnet, no mkl no cuda)
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > > > (mxnet-mkl
> > > > > > > > > > > > <
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
> > > > > > > > > > > >
> > > > > > > > > > > > )
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > > > (mxnet-cuXXX
> > > > > > > > > > > > <
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
> > > > > > > > > > > >
> > > > > > > > > > > > )
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > > > (mxnet-cuXXXmkl
> > > > > > > > > > > > <
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
> > > > > > > > > > > >
> > > > > > > > > > > > )
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > > >
> > > > > > > > > > > > You can easily install these pip wheels in your
> system
> > > > either
> > > > > > by
> > > > > > > > > > > > downloading them to your machine first and then
> > > installing
> > > > by
> > > > > > > > doing:
> > > > > > > > > > > >
> > > > > > > > > > > > pip install /path/to/downloaded/wheel.whl
> > > > > > > > > > > >
> > > > > > > > > > > > Or you can install directly by just giving the link
> to
> > > pip
> > > > > like
> > > > > > > > this:
> > > > > > > > > > > >
> > > > > > > > > > > > pip install
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > > >
> > > > > > > > > > > > Credit goes to everyone involved (in no particular
> > order)
> > > > > > > > > > > > Rakesh Vasudevan
> > > > > > > > > > > > Zach Kimberg
> > > > > > > > > > > > Manu Seth
> > > > > > > > > > > > Sheng Zha
> > > > > > > > > > > > Jun Wu
> > > > > > > > > > > > Pedro Larroy
> > > > > > > > > > > > Chaitanya Bapat
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks!
> > > > > > > > > > > > Sam
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Dec 5, 2019, at 1:16 AM, Lausen, Leonard
> > > > > > > > > <lausen@amazon.com.INVALID
> > > > > > > > > > > > <mailto:lausen@amazon.com.INVALID>> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > We don't loose pip by hosting on S3. We just don't
> host
> > > > > nightly
> > > > > > > > > > releases
> > > > > > > > > > > > on Pypi
> > > > > > > > > > > > servers and mirror them to several hundred mirrors
> > > > > immediately
> > > > > > > > after
> > > > > > > > > > each
> > > > > > > > > > > > build
> > > > > > > > > > > > is published which is very expensive for the Pypi
> > > project..
> > > > > > > People
> > > > > > > > > can
> > > > > > > > > > > > still
> > > > > > > > > > > > install the nightly builds with pip by specifying the
> > -f
> > > > > > option.
> > > > > > > > > > > >
> > > > > > > > > > > > Uploading weekly releases to Pypi will reduce the
> cost
> > > for
> > > > > Pypi
> > > > > > > by
> > > > > > > > > ~75%
> > > > > > > > > > > > [1]. It
> > > > > > > > > > > > may be acceptable to Pypi, but does it make sense for
> > us?
> > > > I'm
> > > > > > not
> > > > > > > > > > > convinced
> > > > > > > > > > > > weekly release on Pypi is a good idea. Consider one
> > > release
> > > > > is
> > > > > > > > buggy,
> > > > > > > > > > > > users will
> > > > > > > > > > > > need to wait for 7 days for a fix. It doesn't provide
> > > good
> > > > > user
> > > > > > > > > > > experience.
> > > > > > > > > > > > If someone has a stronger conviction about the value
> of
> > > > > weekly
> > > > > > > > > releases
> > > > > > > > > > > on
> > > > > > > > > > > > Pypi,
> > > > > > > > > > > > that person shall please go ahead and propose it in a
> > > > > separate
> > > > > > > > > > discussion
> > > > > > > > > > > > thread.
> > > > > > > > > > > >
> > > > > > > > > > > > Currently we don't have generally working nightly
> > builds
> > > to
> > > > > > Pypi
> > > > > > > > and
> > > > > > > > > > as a
> > > > > > > > > > > > matter
> > > > > > > > > > > > of fact we know that we can't have them due to Pypi's
> > > > policy
> > > > > > and
> > > > > > > > our
> > > > > > > > > > > > apparent
> > > > > > > > > > > > need for large binaries. Given this fact and that no
> > > > > objection
> > > > > > > was
> > > > > > > > > > raised
> > > > > > > > > > > > by
> > > > > > > > > > > > 2019-12-05 at 05:42 UTC, I conclude we have lazy
> > > consensus
> > > > on
> > > > > > > > > stopping
> > > > > > > > > > > > upload
> > > > > > > > > > > > attempts of nightly builds to Pypi.
> > > > > > > > > > > >
> > > > > > > > > > > > With consensus established, we can change the CI job
> to
> > > > stop
> > > > > > > trying
> > > > > > > > > to
> > > > > > > > > > > > upload
> > > > > > > > > > > > the nightly builds and then request Pypi to increase
> > the
> > > > > limit.
> > > > > > > > Then
> > > > > > > > > we
> > > > > > > > > > > > have one
> > > > > > > > > > > > less blocker for the 1.6 release.
> > > > > > > > > > > >
> > > > > > > > > > > > Best regards
> > > > > > > > > > > > Leonard
> > > > > > > > > > > >
> > > > > > > > > > > > [1]: Lower cost due to less releases, but higher cost
> > due
> > > > to
> > > > > > > 500MB
> > > > > > > > ->
> > > > > > > > > > > 800MB
> > > > > > > > > > > > limit increase. Assuming that the limit increase
> > > translates
> > > > > > into
> > > > > > > > > > actually
> > > > > > > > > > > > larger
> > > > > > > > > > > > binaries.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Wed, 2019-12-04 at 22:20 +0100, Marco de Abreu
> > wrote:
> > > > > > > > > > > > Are weekly releases an option? It was brought up as
> > > concern
> > > > > > that
> > > > > > > we
> > > > > > > > > > might
> > > > > > > > > > > > lose pip as a pretty common distribution channel
> where
> > > > people
> > > > > > > > consume
> > > > > > > > > > > > nightly builds. I don't feel like that concern has
> been
> > > > > > properly
> > > > > > > > > > > addressed
> > > > > > > > > > > > so far.
> > > > > > > > > > > >
> > > > > > > > > > > > -Marco
> > > > > > > > > > > >
> > > > > > > > > > > > Lausen, Leonard <lausen@amazon.com.invalid<mailto:
> > > > > > > > > > > > lausen@amazon.com.invalid>> schrieb am Mi., 4. Dez.
> > > 2019,
> > > > > > > > > > > > 04:09:
> > > > > > > > > > > >
> > > > > > > > > > > > As a simple POC to test distribution, you can try
> > > > installing
> > > > > > > MXNet
> > > > > > > > > > based
> > > > > > > > > > > on
> > > > > > > > > > > > these 3 URLs:
> > > > > > > > > > > >
> > > > > > > > > > > > pip install --no-cache-dir
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://mxnet-dev.s3.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > > > pip install --no-cache-dir
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://mxnet-dev.s3-accelerate.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > > > pip install --no-cache-dir
> > > > > > > https://d19zq12jzu4w95.cloudfront.net/
> > > > > > > > > > > >
> > > mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > > > <
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > > >
> > > > > > > > > > > > <
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > where --no-cache-dir prevents caching the downloaded
> > > file,
> > > > > for
> > > > > > > the
> > > > > > > > > > > purpose
> > > > > > > > > > > > of
> > > > > > > > > > > > testing. (cu101 chosen based on large size)
> > > > > > > > > > > >
> > > > > > > > > > > > The first URL uses standard S3 bucket in US. The
> second
> > > > uses
> > > > > S3
> > > > > > > > > > > Accelerate
> > > > > > > > > > > > based
> > > > > > > > > > > > on CloudFront CDN. And the third uses CloudFront CDN.
> > I'm
> > > > > > adding
> > > > > > > > the
> > > > > > > > > > > third
> > > > > > > > > > > > URL,
> > > > > > > > > > > > as S3 Accelerate may or may not use all new
> CloudFront
> > > > > > endpoints
> > > > > > > > yet.
> > > > > > > > > > > >
> > > > > > > > > > > > Regarding voting: Uploading to Pypi is currently
> > > > impossible,
> > > > > > > which
> > > > > > > > > is a
> > > > > > > > > > > > reality
> > > > > > > > > > > > (so there is no option to continue as we do
> currently).
> > > > Pypi
> > > > > > > folks
> > > > > > > > > > > > indicated
> > > > > > > > > > > > they will unblock our uploads to Pypi once we stop
> > > > uploading
> > > > > > > > nightly
> > > > > > > > > > > > releases
> > > > > > > > > > > > and taking up 20% of their ressources [1].
> > > > > > > > > > > >
> > > > > > > > > > > > If there are any shortcomings or problems identified
> > with
> > > > > > > uploading
> > > > > > > > > to
> > > > > > > > > > > S3,
> > > > > > > > > > > > we
> > > > > > > > > > > > can work to address them. But for now, status quo is
> > > broken
> > > > > and
> > > > > > > > this
> > > > > > > > > > > seems
> > > > > > > > > > > > the
> > > > > > > > > > > > only solution addressing Pypi's problem.
> > > > > > > > > > > >
> > > > > > > > > > > > I don't mind if you state that you object to lazy
> > > consensus
> > > > > and
> > > > > > > > > start a
> > > > > > > > > > > > vote. If
> > > > > > > > > > > > your "maybe [...] start a proper vote" was supposed
> to
> > be
> > > > an
> > > > > > > > > objection
> > > > > > > > > > to
> > > > > > > > > > > > lazy
> > > > > > > > > > > > consensus, please state so clearly (I'm not sure if
> > > "maybe"
> > > > > > > > qualifies
> > > > > > > > > > as
> > > > > > > > > > > > objection). Though I think it only makes sense with
> at
> > > > least
> > > > > 2
> > > > > > > > > options
> > > > > > > > > > to
> > > > > > > > > > > > vote
> > > > > > > > > > > > on. Status quo is not a meaningful option, as it is
> > > already
> > > > > > > broken.
> > > > > > > > > > > >
> > > > > > > > > > > > Best regards
> > > > > > > > > > > > Leonard
> > > > > > > > > > > >
> > > > > > > > > > > > [1]:
> > > > > > > > > > >
> > > > > > > >
> > > > >
> > https://github.com/pypa/pypi-support/issues/50#issuecomment-560479706
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, 2019-12-03 at 19:28 +0100, Marco de Abreu
> > wrote:
> > > > > > > > > > > > Excellent! Could we maybe come up with a POC and a
> > quick
> > > > > > writeup
> > > > > > > > and
> > > > > > > > > > then
> > > > > > > > > > > > start a proper vote after everyone verified that it
> > > covers
> > > > > > their
> > > > > > > > > > > > use-cases?
> > > > > > > > > > > > -Marco
> > > > > > > > > > > >
> > > > > > > > > > > > Sheng Zha <zhasheng@apache.org> schrieb am Di., 3.
> > Dez.
> > > > > 2019,
> > > > > > > > 19:24:
> > > > > > > > > > > >
> > > > > > > > > > > > Yes, there is. We can also make it easier to access
> by
> > > > using
> > > > > a
> > > > > > > > > > > > geo-location based DNS server so that China users are
> > > > > directed
> > > > > > to
> > > > > > > > > that
> > > > > > > > > > > > local mirror. The rest of the world is already
> covered
> > by
> > > > the
> > > > > > > > global
> > > > > > > > > > > > cloudfront.
> > > > > > > > > > > >
> > > > > > > > > > > > -sz
> > > > > > > > > > > >
> > > > > > > > > > > > On 2019/12/03 18:22:22, Marco de Abreu <
> > > > > > marco.g.abreu@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > Isn't there an s3 endpoint in Beijing?
> > > > > > > > > > > >
> > > > > > > > > > > > It seems like this topic still warrants some
> discussion
> > > and
> > > > > > thus
> > > > > > > > I'd
> > > > > > > > > > > >
> > > > > > > > > > > > prefer
> > > > > > > > > > > > if we don't move forward with lazy consensus.
> > > > > > > > > > > >
> > > > > > > > > > > > -Marco
> > > > > > > > > > > >
> > > > > > > > > > > > Tao Lv <mutouorz@gmail.com> schrieb am Di., 3. Dez.
> > > 2019,
> > > > > > 14:31:
> > > > > > > > > > > >
> > > > > > > > > > > > * For pypi, we can use mirrors.
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Dec 3, 2019 at 9:28 PM Tao Lv <
> > > mutouorz@gmail.com>
> > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > As we have many users in China, I'm considering the
> > > > > > > > > > > > accessibility of
> > > > > > > > > > > > S3.
> > > > > > > > > > > > For pip, we can mirrors.
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard
> > > > > > > > > > > >
> > > > > > > > > > > > <lausen@amazon.com.invalid
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > I would like to remind everyone that lazy consensus
> is
> > > > > assumed
> > > > > > > > > > > > if no
> > > > > > > > > > > > objections
> > > > > > > > > > > > are raised before 2019-12-05 at 05:42 UTC. There has
> > been
> > > > > some
> > > > > > > > > > > >
> > > > > > > > > > > > discussion
> > > > > > > > > > > > about
> > > > > > > > > > > > the proposal, but to my understanding no objections
> > were
> > > > > > > > > > > > raised.
> > > > > > > > > > > > If the proposal is accepted, MXNet releases would be
> > > > > installed
> > > > > > > > > > > > via
> > > > > > > > > > > >   pip install mxnet
> > > > > > > > > > > >
> > > > > > > > > > > > And release candidates via
> > > > > > > > > > > >
> > > > > > > > > > > >  pip install --pre mxnet
> > > > > > > > > > > >
> > > > > > > > > > > > (or with the respective cuda version specifier
> appended
> > > > etc.)
> > > > > > > > > > > >
> > > > > > > > > > > > To obtain releases built automatically from the
> master
> > > > > branch,
> > > > > > > > > > > > users
> > > > > > > > > > > > would need
> > > > > > > > > > > > to specify something like "-f
> > > > > > > > > > > > http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html"
> > > option
> > > > > to
> > > > > > > > > > > > pip.
> > > > > > > > > > > > Best regards
> > > > > > > > > > > > Leonard
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, 2019-12-02 at 05:42 +0000, Lausen, Leonard
> > wrote:
> > > > > > > > > > > > Hi MXNet Community,
> > > > > > > > > > > >
> > > > > > > > > > > > since more than 2 months our binary Python nightly
> > > releases
> > > > > > > > > > > >
> > > > > > > > > > > > published
> > > > > > > > > > > > on Pypi
> > > > > > > > > > > > are broken. The problem is that our binaries exceed
> > > Pypi's
> > > > > > > > > > > > size
> > > > > > > > > > > > limit.
> > > > > > > > > > > > Decreasing the binary size by adding compression
> breaks
> > > > > > > > > > > >
> > > > > > > > > > > > third-party
> > > > > > > > > > > > libraries
> > > > > > > > > > > > loading libmxnet.so
> > > > > > > > > > > >
> > > > > > > > > > > >
> https://github.com/apache/incubator-mxnet/issues/16193
> > > > > > > > > > > > Sheng requested Pypi to increase their size limit:
> > > > > > > > > > > > https://github.com/pypa/pypi-support/issues/50
> > > > > > > > > > > >
> > > > > > > > > > > > Currently "the biggest cost for PyPI from [the many
> > MXNet
> > > > > > > > > > > > binaries
> > > > > > > > > > > > with
> > > > > > > > > > > > nightly
> > > > > > > > > > > > release to Pypi] is the bandwidth consumed when
> several
> > > > > > > > > > > > hundred
> > > > > > > > > > > > mirrors
> > > > > > > > > > > > attempt
> > > > > > > > > > > > to mirror each release immediately after it's
> > published".
> > > > So
> > > > > > > > > > > > Pypi
> > > > > > > > > > > > is
> > > > > > > > > > > > not
> > > > > > > > > > > > inclined to allow us to upload even larger binaries
> on
> > a
> > > > > > > > > > > > nightly
> > > > > > > > > > > > schedule.
> > > > > > > > > > > > Their compromise is to allow it on a weekly cadence.
> > > > > > > > > > > >
> > > > > > > > > > > > However, I would like the community to revisit the
> > > > necessity
> > > > > > > > > > > > of
> > > > > > > > > > > > releasing pre-
> > > > > > > > > > > > release binaries to Pypi on a nightly (or weekly)
> > > cadence.
> > > > > > > > > > > >
> > > > > > > > > > > > Instead, we
> > > > > > > > > > > > can
> > > > > > > > > > > > release nightly binaries ONLY to a public S3 bucket
> and
> > > > > > > > > > > > instruct
> > > > > > > > > > > > users
> > > > > > > > > > > > to
> > > > > > > > > > > > install from there. On our side, we only need to
> > prepare
> > > a
> > > > > > > > > > > > html
> > > > > > > > > > > > document that
> > > > > > > > > > > > contains links to all released nightly binaries.
> > > > > > > > > > > > Finally users will install the nightly releases via
> > > > > > > > > > > >
> > > > > > > > > > > >  pip install --pre mxnet-cu101 -f
> > > > > > > > > > > >
> > > > > > > > > > > > http://mxnet.s3.amazonaws.com/mxnet-cu101/
> > > > > > > > > > > > nightly.html
> > > > > > > > > > > >
> > > > > > > > > > > > Instead of
> > > > > > > > > > > >
> > > > > > > > > > > >  pip install --pre mxnet-cu101
> > > > > > > > > > > >
> > > > > > > > > > > > Of course proper releases and release candidates
> should
> > > > > > > > > > > > still be
> > > > > > > > > > > > made
> > > > > > > > > > > > available
> > > > > > > > > > > > via Pypi. Thus releases would be installed via
> > > > > > > > > > > >
> > > > > > > > > > > >  pip install mxnet-cu101
> > > > > > > > > > > >
> > > > > > > > > > > > And release candidates via
> > > > > > > > > > > >
> > > > > > > > > > > >  pip install --pre mxnet-cu101
> > > > > > > > > > > >
> > > > > > > > > > > > This will substantially reduce the costs of the Pypi
> > > > project
> > > > > > > > > > > > and
> > > > > > > > > > > > in
> > > > > > > > > > > > fact
> > > > > > > > > > > > matches
> > > > > > > > > > > > the installation experience provided by PyTorch. I
> > don't
> > > > > > > > > > > > think the
> > > > > > > > > > > > benefit of
> > > > > > > > > > > > not including "-f
> > > > > > > > > > > >
> > > > > > > > > > > >
> http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html
> > "
> > > > > > > > > > > > matches the costs we currently externalize to the
> Pypi
> > > > team.
> > > > > > > > > > > >
> > > > > > > > > > > > This suggestion seems uncontroversial to me. Thus I
> > would
> > > > > > > > > > > > like to
> > > > > > > > > > > > start
> > > > > > > > > > > > lazy
> > > > > > > > > > > > consensus. If there are no objections, I will assume
> > lazy
> > > > > > > > > > > >
> > > > > > > > > > > > consensus on
> > > > > > > > > > > > stopping
> > > > > > > > > > > > nightly releases to Pypi in 72hrs.
> > > > > > > > > > > >
> > > > > > > > > > > > Best regards
> > > > > > > > > > > > Leonard
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message