airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bolke de Bruin <bdbr...@gmail.com>
Subject Re: Airflow 1.8.0 alpha 4
Date Wed, 11 Jan 2017 20:01:19 GMT
Ok.

For now to call it “beta” 4 items seems to be left:

Blocker:
* retry_delay not respected
* poison pill due to re-queue before process has finished (to be investigated)

Features:
* cgroups + impersonation 
* dag.catchup (Ben Tallman -> Only documentation is missing).

PRs that contain documentation would really be appreciated. In my opinion we are lacking there.
Think about docs covering:
* new scheduler behaviour and options
* task dependency engine
* api / kerberized api
* …

Cheers
Bolke

> On 11 Jan 2017, at 18:59, Arthur Wiedmer <arthur.wiedmer@gmail.com> wrote:
> 
> +1
> 
> We can always think about different ways of doing this later (fair share
> scheduling etc...)
> 
> Best,
> Arthur
> 
> On Wed, Jan 11, 2017 at 4:46 AM, Bolke de Bruin <bdbruin@gmail.com> wrote:
> 
>> Dear All,
>> 
>> I would like to drop "Schedule all pending DAG runs in a single scheduler
>> loop” from the 1.8.0 release (updated: https://github.com/apache/
>> incubator-airflow/pull/1980 <https://github.com/apache/
>> incubator-airflow/pull/1980>, original: https://github.com/apache/
>> incubator-airflow/pull/1906 <https://github.com/apache/
>> incubator-airflow/pull/1906>). The reason for this is that it, imho,
>> biases the scheduler towards a single DAG as it fills the queue with tasks
>> from one DAG and then goes to the next DAG. Starving DAGs that come after
>> the first for resources. As such it should be updated and that will take
>> time.
>> 
>> Please let me know if I am incorrect.
>> 
>> Thanks
>> Bolke
>> 
>>> On 10 Jan 2017, at 09:25, Bolke de Bruin <bdbruin@gmail.com> wrote:
>>> 
>>> Dear All,
>>> 
>>> I have made Airflow 1.8.0 alpha 4 available at
>> https://people.apache.org/~bolke/ <https://people.apache.org/~bolke/> .
>> Again no Apache release yet - this is for testing purposes. I consider this
>> Alpha to be a Beta if not for the pending features. If the pending features
>> are merged within a reasonable time frame (except for **, as no progress
>> currently) then I am planning to mark the tarball as Beta and only allow
>> bug fixes and (very) minor features. This week hopefully.
>>> 
>>> Blockers:
>>> 
>>> * None
>>> 
>>> Fixed issues
>>> * Regression in email
>>> * LDAP case sensitivity
>>> * one_failed task not being run: now seems to pass suddenly (so fixed?)
>> -> need to investigate why
>>> * Email attachments
>>> * Pinned jinja2 to < 2.9.0 (2.9.1 has a confirmed regression)
>>> * Improve time units for task performance charts
>>> * XCom throws an duplicate / locking error
>>> * Add execution_date to trigger_dag
>>> 
>>> Pending features:
>>> * DAG.catchup : minor changes needed, documentation still required,
>> integration tests seem to pass flawlessly
>>> * Cgroups + impersonation: clean up of patches on going, more tests and
>> more elaborate documentation required. Integration tests not executed yet
>>> * Schedule all pending DAG runs in a single scheduler loop: no progress
>> (**)
>>> 
>>> Cheers!
>>> Bolke
>> 
>> 


Mime
View raw message