airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dan Davydov <dan.davy...@airbnb.com.INVALID>
Subject Re: Getting to RC5: Update
Date Wed, 01 Mar 2017 19:00:08 GMT
Agreed, I created a JIRA a couple of minutes ago (it's a subtask in the
JIRA I mentioned).

On Wed, Mar 1, 2017 at 10:58 AM, Bolke de Bruin <bdbruin@gmail.com> wrote:

> Please create a Jira and provide context when this happens. “REMOVED”
> marked means the taskinstance does not have a task equivalent anymore in
> the dag (or so it should :)).
>
> Bolke
>
> > On 1 Mar 2017, at 19:55, Dan Davydov <dan.davydov@airbnb.com.INVALID>
> wrote:
> >
> > We are seeing another major issue with backfills where task instances are
> > being deleted and marked as "removed", I am still investigating. Let's
> keep
> > discussion about these in https://issues.apache.org/
> jira/browse/AIRFLOW-921
> > and the subtask comments to have it one place. I will look at the other
> > points you cc'd me on too. Thanks for continuing to drive this forward!
> >
> > On Wed, Mar 1, 2017 at 8:22 AM, Bolke de Bruin <bdbruin@gmail.com>
> wrote:
> >
> >> Hi,
> >>
> >> Just wanted to give an update about the progress getting to RC5. As
> >> reported we have 6 blockers listed.
> >>
> >> 1. Double run job should not terminate the existing running job. ->
> Patch
> >> Available
> >> 2. Parallelize dag runs in backfills -> Patch Available, Tests need to
> be
> >> updated, see below
> >> 3. Setting a task to running manually breaks a DAGs UI -> Patch merged
> >> 4. Can't mark non-existent tasks as successful from graph view ->
> >> Workaround available (t.b.c.), Patch Available unit tests to be added
> >> 5. (Named)HivePartitionSensor broken if hook attr not set -> Patch
> merged
> >> 6. Skipped tasks potentially cause a dagrun to be marked as
> >> failure/success prematurely -> see below
> >>
> >> On 2 I would like to have some more discussion of this would be
> acceptable
> >> (https://github.com/apache/incubator-airflow/pull/2107). I have written
> >> the patch for this, however we are not large backfill users. So I need
> >> feedback specifically on ripping out the “executor” part: @dan, @max.
> >>
> >> On 6 Alex has reported this earlier and written a PR for this (
> >> https://github.com/apache/incubator-airflow/pull/1961). Maxime had some
> >> thoughts about this, which are currently blocking the integration.
> However,
> >> in testing it seems to solve the issue. Can we finalise the discussion
> >> please @max @dan @alex?
> >>
> >> Cheers
> >> Bolke
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message