mxnet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marco de Abreu <marco.g.ab...@gmail.com>
Subject Re: Stopping nightly releases to Pypi
Date Tue, 03 Dec 2019 18:28:44 GMT
Excellent! Could we maybe come up with a POC and a quick writeup and then
start a proper vote after everyone verified that it covers their use-cases?

-Marco

Sheng Zha <zhasheng@apache.org> schrieb am Di., 3. Dez. 2019, 19:24:

> Yes, there is. We can also make it easier to access by using a
> geo-location based DNS server so that China users are directed to that
> local mirror. The rest of the world is already covered by the global
> cloudfront.
>
> -sz
>
> On 2019/12/03 18:22:22, Marco de Abreu <marco.g.abreu@gmail.com> wrote:
> > Isn't there an s3 endpoint in Beijing?
> >
> > It seems like this topic still warrants some discussion and thus I'd
> prefer
> > if we don't move forward with lazy consensus.
> >
> > -Marco
> >
> > Tao Lv <mutouorz@gmail.com> schrieb am Di., 3. Dez. 2019, 14:31:
> >
> > > * For pypi, we can use mirrors.
> > >
> > > On Tue, Dec 3, 2019 at 9:28 PM Tao Lv <mutouorz@gmail.com> wrote:
> > >
> > > > As we have many users in China, I'm considering the accessibility of
> S3.
> > > > For pip, we can mirrors.
> > > >
> > > > On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard
> <lausen@amazon.com.invalid
> > > >
> > > > wrote:
> > > >
> > > >> I would like to remind everyone that lazy consensus is assumed if
no
> > > >> objections
> > > >> are raised before 2019-12-05 at 05:42 UTC. There has been some
> > > discussion
> > > >> about
> > > >> the proposal, but to my understanding no objections were raised.
> > > >>
> > > >> If the proposal is accepted, MXNet releases would be installed via
> > > >>
> > > >>    pip install mxnet
> > > >>
> > > >> And release candidates via
> > > >>
> > > >>   pip install --pre mxnet
> > > >>
> > > >> (or with the respective cuda version specifier appended etc.)
> > > >>
> > > >> To obtain releases built automatically from the master branch, users
> > > >> would need
> > > >> to specify something like "-f
> > > >> http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html" option to pip.
> > > >>
> > > >> Best regards
> > > >> Leonard
> > > >>
> > > >> On Mon, 2019-12-02 at 05:42 +0000, Lausen, Leonard wrote:
> > > >> > Hi MXNet Community,
> > > >> >
> > > >> > since more than 2 months our binary Python nightly releases
> published
> > > >> on Pypi
> > > >> > are broken. The problem is that our binaries exceed Pypi's size
> limit.
> > > >> > Decreasing the binary size by adding compression breaks
> third-party
> > > >> libraries
> > > >> > loading libmxnet.so
> > > >> https://github.com/apache/incubator-mxnet/issues/16193
> > > >> >
> > > >> > Sheng requested Pypi to increase their size limit:
> > > >> > https://github.com/pypa/pypi-support/issues/50
> > > >> >
> > > >> > Currently "the biggest cost for PyPI from [the many MXNet binaries
> > > with
> > > >> > nightly
> > > >> > release to Pypi] is the bandwidth consumed when several hundred
> > > mirrors
> > > >> > attempt
> > > >> > to mirror each release immediately after it's published". So
Pypi
> is
> > > not
> > > >> > inclined to allow us to upload even larger binaries on a nightly
> > > >> schedule.
> > > >> > Their compromise is to allow it on a weekly cadence.
> > > >> >
> > > >> > However, I would like the community to revisit the necessity
of
> > > >> releasing pre-
> > > >> > release binaries to Pypi on a nightly (or weekly) cadence.
> Instead, we
> > > >> can
> > > >> > release nightly binaries ONLY to a public S3 bucket and instruct
> users
> > > >> to
> > > >> > install from there. On our side, we only need to prepare a html
> > > >> document that
> > > >> > contains links to all released nightly binaries.
> > > >> > Finally users will install the nightly releases via
> > > >> >
> > > >> >   pip install --pre mxnet-cu101 -f
> > > >> http://mxnet.s3.amazonaws.com/mxnet-cu101/
> > > >> > nightly.html
> > > >> >
> > > >> > Instead of
> > > >> >
> > > >> >   pip install --pre mxnet-cu101
> > > >> >
> > > >> > Of course proper releases and release candidates should still
be
> made
> > > >> > available
> > > >> > via Pypi. Thus releases would be installed via
> > > >> >
> > > >> >   pip install mxnet-cu101
> > > >> >
> > > >> > And release candidates via
> > > >> >
> > > >> >   pip install --pre mxnet-cu101
> > > >> >
> > > >> > This will substantially reduce the costs of the Pypi project
and
> in
> > > fact
> > > >> > matches
> > > >> > the installation experience provided by PyTorch. I don't think
the
> > > >> benefit of
> > > >> > not including "-f
> > > >> http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html"
> > > >> > matches the costs we currently externalize to the Pypi team.
> > > >> >
> > > >> > This suggestion seems uncontroversial to me. Thus I would like
to
> > > start
> > > >> lazy
> > > >> > consensus. If there are no objections, I will assume lazy
> consensus on
> > > >> > stopping
> > > >> > nightly releases to Pypi in 72hrs.
> > > >> >
> > > >> > Best regards
> > > >> > Leonard
> > > >>
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message