Mailing-List: contact dev-help@airflow.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@airflow.incubator.apache.org
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8;
 format=flowed
Content-Transfer-Encoding: 8bit
Date: Tue, 05 Jul 2016 15:56:35 -0400
From: Andrew Phillips <andrewp@apache.org>
To: dev@airflow.incubator.apache.org
Cc: Laura Lorenz <llorenz@industrydive.com>
Subject: Re: (Recommended) way to define a failure handler that *only*
 triggers on FAILED, not on UPSTREAM_FAILED?
In-Reply-To: <CAJQxwiFm1MQ0ZD9BKrER25B4p9jtAJWri8nn09UVxDq3TLiOCA@mail.gmail.com>
References: <0ed9b8fae3da5afb32a9ec31bc9087a9@qrmedia.com>
 <bbc506e6a03e69ace7832326ade4a68a@qrmedia.com>
 <CAJQxwiFm1MQ0ZD9BKrER25B4p9jtAJWri8nn09UVxDq3TLiOCA@mail.gmail.com>
Message-ID: <062e388472a63aa9242197d34f2af50b@qrmedia.com>
User-Agent: Roundcube Webmail/1.1.4
archived-at: Tue, 05 Jul 2016 19:56:48 -0000

> I'm confused that a failed status on short_circuit doesn't list its
> associated DagRun as failed. I was under the impression a DagRun was 
> given
> failed status if any of its tasks failed.

Thanks for the comments, Laura! Yes, we were slightly surprised by the 
effect of the short circuit operator as well. Looking at the 
documentation, though, it appears consistent enough, since no task 
actually fails:

"The ShortCircuitOperator is derived from the PythonOperator. It 
evaluates a condition and short-circuits the workflow if the condition 
is False. Any downstream tasks are marked with a state of “skipped”. If 
the condition is True, downstream tasks proceed as normal." (from [1])

 From the code [2]:

     def execute(self, context):
         condition = super(ShortCircuitOperator, self).execute(context)
         if condition:
             logging.info('Proceeding with downstream tasks...')
             return
         else:
             logging.info('Skipping downstream tasks...')
             ...
             logging.info("Done.")

Perhaps it would be worth considering a "fail_on_false" flag for the 
operator? That might be something we'll try ourselves as a simple custom 
extension of this operator.

Regards

ap

[1] 
https://pythonhosted.org/airflow/code.html#airflow.operators.ShortCircuitOperator
[2] 
https://pythonhosted.org/airflow/_modules/python_operator.html#ShortCircuitOperator