airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Driesprong, Fokko" <fo...@driesprong.frl>
Subject Re: Apache Airflow 1.10.0b3
Date Fri, 13 Jul 2018 13:01:57 GMT
Hi all,

I just tried to install a local Airflow instance within Docker, using
Python 3.5, but I got an error on initdb. Anyone else experiencing this? It
has to do with the schema migrations.

Cheers,
Fokko

2018-07-09 22:56 GMT+02:00 Taylor Edmiston <tedmiston@gmail.com>:

> We discussed the splitting out of hooks & operators from core internally at
> Astronomer again today.  I should be able to allocate some time to
> splitting out hooks & operators for the purpose of speeding up the CI.  A
> faster CI on core would be hugely beneficial for all contributors.
>
> I created a Jira issue for this at https://issues.apache.org/
> jira/browse/AIRFLOW-2732.
>
> If anyone has pointers for this work, I'm all ears.  I've done a lot with
> Python unit tests in the past but this part of the Airflow codebase is new
> to me.
>
> Taylor
>
> *Taylor Edmiston*
> Blog <https://blog.tedmiston.com/> | CV
> <https://stackoverflow.com/cv/taylor> | LinkedIn
> <https://www.linkedin.com/in/tedmiston/> | AngelList
> <https://angel.co/taylor> | Stack Overflow
> <https://stackoverflow.com/users/149428/taylor-edmiston>
>
>
> On Sun, Jul 1, 2018 at 6:02 AM, Bolke de Bruin <bdbruin@gmail.com> wrote:
>
> > Separating the tests is where the effort lies. So having that as a
> > consequence of splitting the packages would be nice. It has come up a
> > couple of times but it was not picked up unfortunately.
> >
> > B.
> >
> > > On 28 Jun 2018, at 08:32, Maxime Beauchemin <
> maximebeauchemin@gmail.com>
> > wrote:
> > >
> > > It would be so nice to have a fast test suite. Having to wait for
> Travis
> > > for up to an hour makes many workflows (like working on a release)
> super
> > > painful.
> > >
> > > I spoke with folks at Astronomer recently about moving all operators
> and
> > > hooks to another Python package that airflow would import. This would
> > allow
> > > for independent test suites and to have a more regular release cadence
> on
> > > hooks and operators. What do you think?
> > >
> > > Max
> > >
> > > On Wed, Jun 27, 2018 at 11:18 PM Bolke de Bruin <bdbruin@gmail.com>
> > wrote:
> > >
> > >> Arghhh. The downside of doing this late at night and wanting to go to
> > >> bed... :-). Will make a new one
> > >>
> > >> Sent from my iPhone
> > >>
> > >>> On 28 Jun 2018, at 00:07, Chris Fei <cfei18@gmail.com> wrote:
> > >>>
> > >>> Great, thank you! I just took this for a quick spin and it looks like
> > >>> there's DB migration task missing. The task you committed just
> > recently,
> > >>> 9635ae0956e7_index_faskfail.py, has a down_revision of 856955da8476
> > >>> which can't be found when running airflow initdb (seehttps://
> > >> github.com/apache/incubator-airflow/tree/v1-10-test/airflow/
> > migrations/versions
> > >> ).
> > >>> Chris
> > >>>
> > >>>
> > >>>> On Wed, Jun 27, 2018, at 5:09 PM, Bolke de Bruin wrote:
> > >>>> Hi All,
> > >>>>
> > >>>> I have created a sdist package that is available at:
> > >>>>
> > >>>>
> > >> http://people.apache.org/~bolke/apache-airflow-1.10.0b3+incu
> > bating.tar.gz
> > >>>> <
> > >> http://people.apache.org/~bolke/apache-airflow-1.10.0b3+incu
> > bating.tar.gz>>
> > >>
> > >>>> In order to distinguish it from an actual (apache) release it is:
> > >>>>
> > >>>> 1. Marked as beta (python package managers do not install beta
> > >>>>  versions by default - PEP 440)> 2. It is not signed
> > >>>> 3. It is not at an official apache distribution location
> > >>>>
> > >>>> You can also put something like this in a requirements.txt file:
> > >>>>
> > >>>> git+
> > >>>>
> > >> https://github.com/apache/incubator-airflow@v1-10-test#egg=
> > apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,
> > postgres,redis,slack,s3
> > >>>> <
> > >> https://github.com/apache/incubator-airflow@v1-10-test#egg=
> > apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,
> > postgres,redis,slack,s3
> > >>>
> > >>>> ]> <https://github.com/rodrigc/incubator-airflow@master#egg=apache-
> > >>>> airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redi
> > s,slack,s-
> > >>>> 3]
> > >>>> <
> > >> https://github.com/rodrigc/incubator-airflow@master#egg=apac
> > he-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,
> > redis,slack,s3][1]
> > >>>
> > >>>>>>
> > >>>> and then "pip install -r requirements.txt”.
> > >>>>
> > >>>> I hope that after this beta we can go to RC and start voting on
> 1.10.>
> > >>>> Cheers
> > >>>> Bolke
> > >>>
> > >>>
> > >>> Links:
> > >>>
> > >>> 1.
> > >> https://github.com/rodrigc/incubator-airflow@master#egg=apac
> > he-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,
> > redis,slack,s3]%20%3Chttps://github.com/rodrigc/incubator-
> > airflow@master#egg=apache-airflow[celery,crypto,emr,
> > hive,hdfs,ldap,mysql,postgres,redis,slack,s3]
> > <https://github.com/rodrigc/incubator-airflow@master#egg=
> apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,
> postgres,redis,slack,s3]%20%3Chttps://github.com/rodrigc/
> incubator-airflow@master%23egg=apache-airflow[celery,
> crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3]>
> > >> <https://github.com/rodrigc/incubator-airflow@master#egg=apa
> > che-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,
> > redis,slack,s3]%20%3Chttps://github.com/rodrigc/incubator-
> > airflow@master%23egg=apache-airflow[celery,crypto,emr,
> > hive,hdfs,ldap,mysql,postgres,redis,slack,s3]>
> > >>
> >
> >
>

Mime
View raw message