airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maxime Beauchemin <maximebeauche...@gmail.com>
Subject Re: Apache Airflow 1.10.0b3
Date Thu, 28 Jun 2018 06:32:57 GMT
It would be so nice to have a fast test suite. Having to wait for Travis
for up to an hour makes many workflows (like working on a release) super
painful.

I spoke with folks at Astronomer recently about moving all operators and
hooks to another Python package that airflow would import. This would allow
for independent test suites and to have a more regular release cadence on
hooks and operators. What do you think?

Max

On Wed, Jun 27, 2018 at 11:18 PM Bolke de Bruin <bdbruin@gmail.com> wrote:

> Arghhh. The downside of doing this late at night and wanting to go to
> bed... :-). Will make a new one
>
> Sent from my iPhone
>
> > On 28 Jun 2018, at 00:07, Chris Fei <cfei18@gmail.com> wrote:
> >
> > Great, thank you! I just took this for a quick spin and it looks like
> > there's DB migration task missing. The task you committed just recently,
> > 9635ae0956e7_index_faskfail.py, has a down_revision of 856955da8476
> > which can't be found when running airflow initdb (seehttps://
> github.com/apache/incubator-airflow/tree/v1-10-test/airflow/migrations/versions
> ).
> > Chris
> >
> >
> >> On Wed, Jun 27, 2018, at 5:09 PM, Bolke de Bruin wrote:
> >> Hi All,
> >>
> >> I have created a sdist package that is available at:
> >>
> >>
> http://people.apache.org/~bolke/apache-airflow-1.10.0b3+incubating.tar.gz
> >> <
> http://people.apache.org/~bolke/apache-airflow-1.10.0b3+incubating.tar.gz>>
>
> >> In order to distinguish it from an actual (apache) release it is:
> >>
> >> 1. Marked as beta (python package managers do not install beta
> >>   versions by default - PEP 440)> 2. It is not signed
> >> 3. It is not at an official apache distribution location
> >>
> >> You can also put something like this in a requirements.txt file:
> >>
> >> git+
> >>
> https://github.com/apache/incubator-airflow@v1-10-test#egg=apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3
> >> <
> https://github.com/apache/incubator-airflow@v1-10-test#egg=apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3
> >
> >> ]> <https://github.com/rodrigc/incubator-airflow@master#egg=apache-
> >> airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s-
> >> 3]
> >> <
> https://github.com/rodrigc/incubator-airflow@master#egg=apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3][1]
> >
> >>>>
> >> and then "pip install -r requirements.txt”.
> >>
> >> I hope that after this beta we can go to RC and start voting on 1.10.>
> >> Cheers
> >> Bolke
> >
> >
> > Links:
> >
> >  1.
> https://github.com/rodrigc/incubator-airflow@master#egg=apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3]%20%3Chttps://github.com/rodrigc/incubator-airflow@master#egg=apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3]
> <https://github.com/rodrigc/incubator-airflow@master#egg=apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3]%20%3Chttps://github.com/rodrigc/incubator-airflow@master%23egg=apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3]>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message