airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dan Davydov <dan.davy...@airbnb.com.INVALID>
Subject Re: airflow backfill seems to ignore -I
Date Thu, 13 Jul 2017 00:44:35 GMT
Airflow dependencies were simplified a bit, -i no longer ignores failed
state tasks, check out the -A flag which ignores pretty much all
dependencies (including the failed state tasks), though depending on the
version you are using there is a bug that is being fixed here:
https://github.com/apache/incubator-airflow/pull/2327

On Wed, Jul 5, 2017 at 8:45 AM, Weiwei Zhang <vvianzhang@gmail.com> wrote:

> I am using airflow 1.8.1 as well. It is able to pick up the rest of the
> tasks when using backfill with the only exception which is when there is a
> task failed and I had to clear the status to allow the backfill to work.
> Any ideas why it is behaving like this? The previous version 1.6.2 didn't
> require clearing the failed task before doing backfill.
>
> Thx a lot,
> Viv
>
> > On Jul 5, 2017, at 7:38 AM, Tobias Feldhaus <
> Tobias.Feldhaus@localsearch.ch> wrote:
> >
> > I’ve just pulled the newest master and build it; the behaviour is the
> same. How can it be that “–i” is not honoured and dependencies are checked?
> >
> >
> > On 05.07.2017, 15:49, "Tobias Feldhaus" <Tobias.Feldhaus@localsearch.ch>
> wrote:
> >
> >    But nonetheless, is it not possible to backfill and ignore the
> upstream dependencies with “-i” ?
> >
> >    On 05.07.2017, 14:34, "Tobias Feldhaus" <Tobias.Feldhaus@localsearch.
> ch> wrote:
> >
> >        I meant –i , but I just needed to manually set the upstream
> things to success and it worked. Nevermind.
> >
> >        Best,
> >        Tobi
> >
> >        On 05.07.2017, 14:28, "Tobias Feldhaus" <
> Tobias.Feldhaus@localsearch.ch<mailto:Tobias.Feldhaus@localsearch.ch>>
> wrote:
> >
> >        Hi,
> >
> >        When running airflow (1.8.1) backfill with –I and –t like:
> >
> >        airflow backfill -t 'nonspider_sessions' -i -I -s 2017-05-30 -e
> 2017-05-31 google_pipelines
> >
> >        I would expect it to rerun that specific task and ignoring the
> dependencies. Instead I see this:
> >
> >        [2017-07-05 12:23:30,419] {base_task_runner.py:95} INFO -
> Subtask: [2017-07-05 12:23:30,419] {models.py:1145} INFO - Dependencies not
> met for <TaskInstance: google_pipelines.nonspider_sessions 2017-05-30
> 05:30:00 [queued]>, dependency 'Trigger Rule' FAILED: Task's trigger rule
> 'all_success' requires all upstream tasks to have succeeded, but found 3
> non-success(es). upstream_tasks_state={'successes': 0L, 'failed': 0L,
> 'upstream_failed': 0L, 'skipped': 0L, 'done': 0L},
> upstream_task_ids=['frontend_sensor', 'log_sensor', 'tracker_pipeline']
> >
> >        Am I doing it wrong?
> >
> >
> >
> >        Best,
> >        Tobi
> >
> >
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message