airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jarek Potiuk <Jarek.Pot...@polidea.com>
Subject Re: Latest Sphinx (2.1.2) causes master to fail
Date Mon, 24 Jun 2019 21:30:57 GMT
Indeed. Poetry is what we wanted to try already for quite a while and it
seems it has pretty much what we need including extras support and .lock
file - quite similar to npm approach + nice set of command line tools to
manage deps.

J.

On Mon, Jun 24, 2019 at 11:24 PM Ash Berlin-Taylor <ash@apache.org> wrote:

> Hi Max,
>
> Did you take a look at Poetry (https://poetry.eustace.io/ <
> https://poetry.eustace.io/>)? How does this PR compare?
>
> -ash
>
>
> > On 24 Jun 2019, at 22:02, Maxime Beauchemin <maximebeauchemin@gmail.com>
> wrote:
> >
> > This Superset PR may be relevant and useful for Airflow too (still WiP):
> > https://github.com/apache/incubator-superset/pull/7762
> >
> > Basically it allows your project to define a "requirements/" folder with
> a
> > "requirements.json" as a single source of truth for all deps, including
> the
> > details of what goes in the "extra_requires", and compiles/pins all sets
> of
> > dependencies in a set of useful files
> >
> > Now:
> > * this requirements.json is parsed and used directly as is in `setup.py`
> > (same as before)
> > * running "compile_requirements.py" effectively creates a set of
> > "pip-compiled" full tree and pinned requirements for each set of
> > requirements.
> > * these pinned files can be used in CI / docker / builds to avoid bad
> > surprises like it appears you did. The package (setup.py) is still
> > permissive and allows people to pick their versions, but CI can be pinned
> > and rock solid.
> >
> > The folder includes something like:
> > requirements/requirements-hive.in
> > requirements/requirements-hive.txt
> > requirements/requirements-postgres.in
> > requirements/requirements-postgres.txt
> > # ... (2 files for each extra_requires defined in the JSON file)
> >
> > There may be some subtleties around py2/py3 reqs diverging, but you could
> > probably bake that into that logic if you wanted to (for Superset it's a
> > non-issue since we only support py36).
> >
> > Max
> >
> > On Mon, Jun 24, 2019 at 1:16 AM Kaxil Naik <kaxilnaik@gmail.com> wrote:
> >
> >> It works :) - The CI passed for the commit on v1-10-test, so your PR
> should
> >> pass as well.
> >>
> >> On Mon, Jun 24, 2019 at 1:40 PM Jarek Potiuk <Jarek.Potiuk@polidea.com>
> >> wrote:
> >>
> >>> I pinned auto-api to below 1.1.0 and re-run the test - Let's hope it
> >> works
> >>> for now .
> >>>
> >>> On Mon, Jun 24, 2019 at 10:05 AM Jarek Potiuk <
> Jarek.Potiuk@polidea.com>
> >>> wrote:
> >>>
> >>>> OK. After some checks it turns out it's not only Sphinx really but
> also
> >>>> sphinx-autoapi (v 1.1.0 released yesterday).
> >>>>
> >>>> J.
> >>>>
> >>>>
> >>>> On Mon, Jun 24, 2019 at 9:29 AM Jarek Potiuk <
> Jarek.Potiuk@polidea.com
> >>>
> >>>> wrote:
> >>>>
> >>>>> Kamil prepared a fix https://github.com/apache/airflow/pull/5468
> >>> (thanks!) and
> >>>>> when I tested it on a clean pip-install with latest sphinx it seems
> to
> >>> work
> >>>>> fine (Running Travis CI build
> >>>>> <https://travis-ci.org/apache/airflow/builds/549600100>now).
Let's
> >> see
> >>>>> if it helps.
> >>>>>
> >>>>> BTW. The AIP-10 Docker image last step (pending Kubernetes fix)
> should
> >>>>> solve most of the dependency issues, however I think this particular
> >>>>> problem could be missed (it's an interesting one).
> >>>>>
> >>>>> I will soon start another thread (after dockerfile merging) about
> >>> finally
> >>>>> looking into making dependencies more manageable and less
> >> transient-deps
> >>>>> problems. I already have some ideas how we can approach the problem
> >> that
> >>>>> Airflow is both: library (no-dependency pinning recommended)  and
> >>>>> end-product (strong pinning recommended).
> >>>>>
> >>>>> J.
> >>>>>
> >>>>> On Mon, Jun 24, 2019 at 8:48 AM Ash Berlin-Taylor <ash@apache.org>
> >>> wrote:
> >>>>>
> >>>>>> Any cyclic imports we have aren't actually a problem and are
handled
> >>>>>> fine by python 3 (obviously, else it wouldn't work) so I think
this
> >> is
> >>> a
> >>>>>> big in sphinx with it being over zealous.
> >>>>>>
> >>>>>> This only affects tests and development, not our users in this
case.
> >>>>>>
> >>>>>> If it's difficult to fix I would go with pinning - I want to
get a
> >>>>>> 1.10.4 beta 1 out today.
> >>>>>>
> >>>>>> -ash
> >>>>>>
> >>>>>> On 24 June 2019 00:37:57 BST, Jarek Potiuk <
> Jarek.Potiuk@polidea.com
> >>>
> >>>>>> wrote:
> >>>>>>>
> >>>>>>> Hello Everyone,
> >>>>>>>
> >>>>>>> We have yet another case where relesing a package (Sphinx
2.1.2 in
> >>> this
> >>>>>>> case) - started to cause master to fail.
> >>>>>>>
> >>>>>>> Should we pin Sphinx to an earlier version  or maybe someone
can
> >> help
> >>> and
> >>>>>>> figure out and fix it quickly (seems that there are some
cyclic
> >>> imports
> >>>>>>> that probably need solving anyway if Sphinx is right about
them) ?
> >>> @Kamil
> >>>>>>> BreguĊ‚a <kamil.bregula@polidea.com> @Fokko Driesprong
> >>> <fokko@driesprong.frl>
> >>>>>>> ? WDYT ?
> >>>>>>>
> >>>>>>> I created an issue for that
> >>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-4841
> >>>>>>>
> >>>>>>> J.
> >>>>>>>
> >>>>>>>
> >>>>>
> >>>>> --
> >>>>>
> >>>>> Jarek Potiuk
> >>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
> >>>>>
> >>>>> M: +48 660 796 129 <+48660796129>
> >>>>> [image: Polidea] <https://www.polidea.com/>
> >>>>>
> >>>>>
> >>>>
> >>>> --
> >>>>
> >>>> Jarek Potiuk
> >>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
> >>>>
> >>>> M: +48 660 796 129 <+48660796129>
> >>>> [image: Polidea] <https://www.polidea.com/>
> >>>>
> >>>>
> >>>
> >>> --
> >>>
> >>> Jarek Potiuk
> >>> Polidea <https://www.polidea.com/> | Principal Software Engineer
> >>>
> >>> M: +48 660 796 129 <+48660796129>
> >>> [image: Polidea] <https://www.polidea.com/>
> >>>
> >>
> >>
> >> --
> >> *Kaxil Naik*
> >> *Big Data Consultant | DevOps Data Engineer*
> >> *Certified *Google Cloud Data Engineer | *Certified* Apache Spark &
> Neo4j
> >> Developer
> >> *LinkedIn*: https://www.linkedin.com/in/kaxil
> >>
>
>

-- 

Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer

M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message