mxnet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joshua Z. Zhang" <cheungc...@gmail.com>
Subject Re: Stopping nightly releases to Pypi
Date Mon, 02 Dec 2019 19:17:42 GMT
Separating the nightly wheels from PYPI certainly can reduce our turnaround time for processing
new packages, overall I am in favor of this proposal.

However, one question is that would external private server causing problems when you are
trying to pin a nightly version of MXNet in Conda pip environment?

For example, GluonCV heavily rely on nightly builds of mxnet in CI (https://github.com/dmlc/gluon-cv/blob/master/docs/build.yml#L12
<https://github.com/dmlc/gluon-cv/blob/master/docs/build.yml#L12>) and I know conda
has bad reputation in response to pip installation options: https://github.com/conda/conda/issues/6805
<https://github.com/conda/conda/issues/6805>

Thanks,
Zhi

> On Dec 1, 2019, at 11:14 PM, Lausen, Leonard <lausen@amazon.com.INVALID> wrote:
> 
>> For the d2l.ai case, bugs within MXNet are fine, so long as users are sent to
>> a pinned version that corresponds to the notebooks that are created for it.
> 
> I'm not sure if this approach provides the best user-experience. Ideally we
> would like d2l users to continue using MXNet after going through the book. But
> such adoption could be limited if readers run into bugs due to usage of an old
> (pinned) nightly version when they begin to use more advanced features (not
> included in / tested by the book).
> 
> It seems preferable to have novice users to use a carefully vetted version of
> MXNet. Such version could be easily obtained by backporting the new operators to
> the latest stable release, and tagging a release candidate for a new stable
> minor release. We can set up a pipeline to automatically build such release
> candidates to Pypi?
> 
> But if that's not an option, including the link could be fine as long as users
> can copy-and-paste it. It seems we may expect copy-and-paste ability for the
> online version of the book. What do you think? Arguably any printed version
> should not rely on nightly releases.
> 
> Best regards
> Leonard
> 
> On Mon, 2019-12-02 at 06:13 +0000, Chung, Alex wrote:
>> For the d2l.ai case, bugs within MXNet are fine, so long as users are sent to
>> a pinned version that corresponds to the notebooks that are created for it. I
>> suppose we can once again provide the whole link, but getting directly from
>> pip is the familiar experience for most devs.
>> 
>> Yes, 1.6 is the target release, but I don't see a world where the team can
>> create new operators, and then get it pushed out to stable fast enough for the
>> book writers.
>> 
>> Sincerely,
>> 
>> Alex Chung
>> 
>> ________________________________________
>> From: Lausen, Leonard <lausen@amazon.com.INVALID>
>> Sent: Sunday, December 1, 2019 10:08 PM
>> To: dev@mxnet.incubator.apache.org
>> Cc: Kamakoti, Balaji
>> Subject: Re: Stopping nightly releases to Pypi
>> 
>> If we decide to do weekly pre-release builds to Pypi, what's the benefit? To
>> catch bugs and pinpoint when they were introduced, having weekly builds may be
>> too coarse. So people would likely prefer the nightly releases and install
>> them
>> from S3 via: pip install --pre mxnet-cu101 -f
>> http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html
>> 
>> For any future project that relies on nightly builds of MXNet 1.7 (or later),
>> we
>> can include above line. (Similar to the Install instructions on pytorch.org if
>> you select "Nightly" and "Pip").
>> 
>> For existing projects, which taught users to use "pip install --pre mxnet-
>> cu101"
>> to get the MXNet 1.6 pre-release: There is no problem, because we will have a
>> stable release of 1.6 in the near future.
>> Based on my understanding, d2l.ai currently targets the MXNet 1.6 release.
>> 
>> On Mon, 2019-12-02 at 05:51 +0000, Chung, Alex wrote:
>>> Hi Leonard,
>>> 
>>> Is there any reason why we shouldn't take both options? Ie we do weekly
>>> build
>>> on PyPi and provide the s3 option. I would be inclined to make sure we
>>> provide
>>> as many avenues as possible to reduce friction for developers. The d2l.ai
>>> book
>>> by Alex Smola is attracting a community that so far has been relying on the
>>> nightly builds in advance of stable.
>>> 
>>> Sincerely,
>>> 
>>> Alex Chung
>>> 
>>> ________________________________________
>>> From: Lausen, Leonard <lausen@amazon.com.INVALID>
>>> Sent: Sunday, December 1, 2019 9:42 PM
>>> To: dev@mxnet.apache.org
>>> Subject: Stopping nightly releases to Pypi
>>> 
>>> Hi MXNet Community,
>>> 
>>> since more than 2 months our binary Python nightly releases published on
>>> Pypi
>>> are broken. The problem is that our binaries exceed Pypi's size limit.
>>> Decreasing the binary size by adding compression breaks third-party
>>> libraries
>>> loading libmxnet.so https://github.com/apache/incubator-mxnet/issues/16193
>>> 
>>> Sheng requested Pypi to increase their size limit:
>>> https://github.com/pypa/pypi-support/issues/50
>>> 
>>> Currently "the biggest cost for PyPI from [the many MXNet binaries with
>>> nightly
>>> release to Pypi] is the bandwidth consumed when several hundred mirrors
>>> attempt
>>> to mirror each release immediately after it's published". So Pypi is not
>>> inclined to allow us to upload even larger binaries on a nightly schedule.
>>> Their compromise is to allow it on a weekly cadence.
>>> 
>>> However, I would like the community to revisit the necessity of releasing
>>> pre-
>>> release binaries to Pypi on a nightly (or weekly) cadence. Instead, we can
>>> release nightly binaries ONLY to a public S3 bucket and instruct users to
>>> install from there. On our side, we only need to prepare a html document
>>> that
>>> contains links to all released nightly binaries.
>>> Finally users will install the nightly releases via
>>> 
>>>  pip install --pre mxnet-cu101 -f 
>>> http://mxnet.s3.amazonaws.com/mxnet-cu101/
>>> nightly.html
>>> 
>>> Instead of
>>> 
>>>  pip install --pre mxnet-cu101
>>> 
>>> Of course proper releases and release candidates should still be made
>>> available
>>> via Pypi. Thus releases would be installed via
>>> 
>>>  pip install mxnet-cu101
>>> 
>>> And release candidates via
>>> 
>>>  pip install --pre mxnet-cu101
>>> 
>>> This will substantially reduce the costs of the Pypi project and in fact
>>> matches
>>> the installation experience provided by PyTorch. I don't think the benefit
>>> of
>>> not including "-f http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html"
>>> matches the costs we currently externalize to the Pypi team.
>>> 
>>> This suggestion seems uncontroversial to me. Thus I would like to start lazy
>>> consensus. If there are no objections, I will assume lazy consensus on
>>> stopping
>>> nightly releases to Pypi in 72hrs.
>>> 
>>> Best regards
>>> Leonard


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message