mxnet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joshua Z. Zhang" <>
Subject Re: Stopping nightly releases to Pypi
Date Mon, 02 Dec 2019 19:17:42 GMT
Separating the nightly wheels from PYPI certainly can reduce our turnaround time for processing
new packages, overall I am in favor of this proposal.

However, one question is that would external private server causing problems when you are
trying to pin a nightly version of MXNet in Conda pip environment?

For example, GluonCV heavily rely on nightly builds of mxnet in CI (
<>) and I know conda
has bad reputation in response to pip installation options:


> On Dec 1, 2019, at 11:14 PM, Lausen, Leonard <> wrote:
>> For the case, bugs within MXNet are fine, so long as users are sent to
>> a pinned version that corresponds to the notebooks that are created for it.
> I'm not sure if this approach provides the best user-experience. Ideally we
> would like d2l users to continue using MXNet after going through the book. But
> such adoption could be limited if readers run into bugs due to usage of an old
> (pinned) nightly version when they begin to use more advanced features (not
> included in / tested by the book).
> It seems preferable to have novice users to use a carefully vetted version of
> MXNet. Such version could be easily obtained by backporting the new operators to
> the latest stable release, and tagging a release candidate for a new stable
> minor release. We can set up a pipeline to automatically build such release
> candidates to Pypi?
> But if that's not an option, including the link could be fine as long as users
> can copy-and-paste it. It seems we may expect copy-and-paste ability for the
> online version of the book. What do you think? Arguably any printed version
> should not rely on nightly releases.
> Best regards
> Leonard
> On Mon, 2019-12-02 at 06:13 +0000, Chung, Alex wrote:
>> For the case, bugs within MXNet are fine, so long as users are sent to
>> a pinned version that corresponds to the notebooks that are created for it. I
>> suppose we can once again provide the whole link, but getting directly from
>> pip is the familiar experience for most devs.
>> Yes, 1.6 is the target release, but I don't see a world where the team can
>> create new operators, and then get it pushed out to stable fast enough for the
>> book writers.
>> Sincerely,
>> Alex Chung
>> ________________________________________
>> From: Lausen, Leonard <>
>> Sent: Sunday, December 1, 2019 10:08 PM
>> To:
>> Cc: Kamakoti, Balaji
>> Subject: Re: Stopping nightly releases to Pypi
>> If we decide to do weekly pre-release builds to Pypi, what's the benefit? To
>> catch bugs and pinpoint when they were introduced, having weekly builds may be
>> too coarse. So people would likely prefer the nightly releases and install
>> them
>> from S3 via: pip install --pre mxnet-cu101 -f
>> For any future project that relies on nightly builds of MXNet 1.7 (or later),
>> we
>> can include above line. (Similar to the Install instructions on if
>> you select "Nightly" and "Pip").
>> For existing projects, which taught users to use "pip install --pre mxnet-
>> cu101"
>> to get the MXNet 1.6 pre-release: There is no problem, because we will have a
>> stable release of 1.6 in the near future.
>> Based on my understanding, currently targets the MXNet 1.6 release.
>> On Mon, 2019-12-02 at 05:51 +0000, Chung, Alex wrote:
>>> Hi Leonard,
>>> Is there any reason why we shouldn't take both options? Ie we do weekly
>>> build
>>> on PyPi and provide the s3 option. I would be inclined to make sure we
>>> provide
>>> as many avenues as possible to reduce friction for developers. The
>>> book
>>> by Alex Smola is attracting a community that so far has been relying on the
>>> nightly builds in advance of stable.
>>> Sincerely,
>>> Alex Chung
>>> ________________________________________
>>> From: Lausen, Leonard <>
>>> Sent: Sunday, December 1, 2019 9:42 PM
>>> To:
>>> Subject: Stopping nightly releases to Pypi
>>> Hi MXNet Community,
>>> since more than 2 months our binary Python nightly releases published on
>>> Pypi
>>> are broken. The problem is that our binaries exceed Pypi's size limit.
>>> Decreasing the binary size by adding compression breaks third-party
>>> libraries
>>> loading
>>> Sheng requested Pypi to increase their size limit:
>>> Currently "the biggest cost for PyPI from [the many MXNet binaries with
>>> nightly
>>> release to Pypi] is the bandwidth consumed when several hundred mirrors
>>> attempt
>>> to mirror each release immediately after it's published". So Pypi is not
>>> inclined to allow us to upload even larger binaries on a nightly schedule.
>>> Their compromise is to allow it on a weekly cadence.
>>> However, I would like the community to revisit the necessity of releasing
>>> pre-
>>> release binaries to Pypi on a nightly (or weekly) cadence. Instead, we can
>>> release nightly binaries ONLY to a public S3 bucket and instruct users to
>>> install from there. On our side, we only need to prepare a html document
>>> that
>>> contains links to all released nightly binaries.
>>> Finally users will install the nightly releases via
>>>  pip install --pre mxnet-cu101 -f 
>>> nightly.html
>>> Instead of
>>>  pip install --pre mxnet-cu101
>>> Of course proper releases and release candidates should still be made
>>> available
>>> via Pypi. Thus releases would be installed via
>>>  pip install mxnet-cu101
>>> And release candidates via
>>>  pip install --pre mxnet-cu101
>>> This will substantially reduce the costs of the Pypi project and in fact
>>> matches
>>> the installation experience provided by PyTorch. I don't think the benefit
>>> of
>>> not including "-f"
>>> matches the costs we currently externalize to the Pypi team.
>>> This suggestion seems uncontroversial to me. Thus I would like to start lazy
>>> consensus. If there are no objections, I will assume lazy consensus on
>>> stopping
>>> nightly releases to Pypi in 72hrs.
>>> Best regards
>>> Leonard

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message