mxnet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lausen, Leonard" <lau...@amazon.com.INVALID>
Subject Re: Stopping nightly releases to Pypi
Date Mon, 02 Dec 2019 06:08:10 GMT
If we decide to do weekly pre-release builds to Pypi, what's the benefit? To
catch bugs and pinpoint when they were introduced, having weekly builds may be
too coarse. So people would likely prefer the nightly releases and install them
from S3 via: pip install --pre mxnet-cu101 -f 
http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html

For any future project that relies on nightly builds of MXNet 1.7 (or later), we
can include above line. (Similar to the Install instructions on pytorch.org if
you select "Nightly" and "Pip").

For existing projects, which taught users to use "pip install --pre mxnet-cu101" 
to get the MXNet 1.6 pre-release: There is no problem, because we will have a
stable release of 1.6 in the near future.
Based on my understanding, d2l.ai currently targets the MXNet 1.6 release.

On Mon, 2019-12-02 at 05:51 +0000, Chung, Alex wrote:
> Hi Leonard,
> 
> Is there any reason why we shouldn't take both options? Ie we do weekly build
> on PyPi and provide the s3 option. I would be inclined to make sure we provide
> as many avenues as possible to reduce friction for developers. The d2l.ai book
> by Alex Smola is attracting a community that so far has been relying on the
> nightly builds in advance of stable.
> 
> Sincerely,
> 
> Alex Chung
> 
> ________________________________________
> From: Lausen, Leonard <lausen@amazon.com.INVALID>
> Sent: Sunday, December 1, 2019 9:42 PM
> To: dev@mxnet.apache.org
> Subject: Stopping nightly releases to Pypi
> 
> Hi MXNet Community,
> 
> since more than 2 months our binary Python nightly releases published on Pypi
> are broken. The problem is that our binaries exceed Pypi's size limit.
> Decreasing the binary size by adding compression breaks third-party libraries
> loading libmxnet.so https://github.com/apache/incubator-mxnet/issues/16193
> 
> Sheng requested Pypi to increase their size limit:
> https://github.com/pypa/pypi-support/issues/50
> 
> Currently "the biggest cost for PyPI from [the many MXNet binaries with
> nightly
> release to Pypi] is the bandwidth consumed when several hundred mirrors
> attempt
> to mirror each release immediately after it's published". So Pypi is not
> inclined to allow us to upload even larger binaries on a nightly schedule.
> Their compromise is to allow it on a weekly cadence.
> 
> However, I would like the community to revisit the necessity of releasing pre-
> release binaries to Pypi on a nightly (or weekly) cadence. Instead, we can
> release nightly binaries ONLY to a public S3 bucket and instruct users to
> install from there. On our side, we only need to prepare a html document that
> contains links to all released nightly binaries.
> Finally users will install the nightly releases via
> 
>   pip install --pre mxnet-cu101 -f http://mxnet.s3.amazonaws.com/mxnet-cu101/
> nightly.html
> 
> Instead of
> 
>   pip install --pre mxnet-cu101
> 
> Of course proper releases and release candidates should still be made
> available
> via Pypi. Thus releases would be installed via
> 
>   pip install mxnet-cu101
> 
> And release candidates via
> 
>   pip install --pre mxnet-cu101
> 
> This will substantially reduce the costs of the Pypi project and in fact
> matches
> the installation experience provided by PyTorch. I don't think the benefit of
> not including "-f http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html"
> matches the costs we currently externalize to the Pypi team.
> 
> This suggestion seems uncontroversial to me. Thus I would like to start lazy
> consensus. If there are no objections, I will assume lazy consensus on
> stopping
> nightly releases to Pypi in 72hrs.
> 
> Best regards
> Leonard
Mime
View raw message