airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bolke de Bruin <bdbr...@gmail.com>
Subject Re: Getting to RC5: Update
Date Wed, 01 Mar 2017 18:58:51 GMT
Please create a Jira and provide context when this happens. “REMOVED” marked means the
taskinstance does not have a task equivalent anymore in the dag (or so it should :)).

Bolke

> On 1 Mar 2017, at 19:55, Dan Davydov <dan.davydov@airbnb.com.INVALID> wrote:
> 
> We are seeing another major issue with backfills where task instances are
> being deleted and marked as "removed", I am still investigating. Let's keep
> discussion about these in https://issues.apache.org/jira/browse/AIRFLOW-921
> and the subtask comments to have it one place. I will look at the other
> points you cc'd me on too. Thanks for continuing to drive this forward!
> 
> On Wed, Mar 1, 2017 at 8:22 AM, Bolke de Bruin <bdbruin@gmail.com> wrote:
> 
>> Hi,
>> 
>> Just wanted to give an update about the progress getting to RC5. As
>> reported we have 6 blockers listed.
>> 
>> 1. Double run job should not terminate the existing running job. -> Patch
>> Available
>> 2. Parallelize dag runs in backfills -> Patch Available, Tests need to be
>> updated, see below
>> 3. Setting a task to running manually breaks a DAGs UI -> Patch merged
>> 4. Can't mark non-existent tasks as successful from graph view ->
>> Workaround available (t.b.c.), Patch Available unit tests to be added
>> 5. (Named)HivePartitionSensor broken if hook attr not set -> Patch merged
>> 6. Skipped tasks potentially cause a dagrun to be marked as
>> failure/success prematurely -> see below
>> 
>> On 2 I would like to have some more discussion of this would be acceptable
>> (https://github.com/apache/incubator-airflow/pull/2107). I have written
>> the patch for this, however we are not large backfill users. So I need
>> feedback specifically on ripping out the “executor” part: @dan, @max.
>> 
>> On 6 Alex has reported this earlier and written a PR for this (
>> https://github.com/apache/incubator-airflow/pull/1961). Maxime had some
>> thoughts about this, which are currently blocking the integration. However,
>> in testing it seems to solve the issue. Can we finalise the discussion
>> please @max @dan @alex?
>> 
>> Cheers
>> Bolke


Mime
View raw message