airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Phillips <andr...@apache.org>
Subject Re: (Recommended) way to define a failure handler that *only* triggers on FAILED, not on UPSTREAM_FAILED?
Date Tue, 05 Jul 2016 19:56:35 GMT
> I'm confused that a failed status on short_circuit doesn't list its
> associated DagRun as failed. I was under the impression a DagRun was 
> given
> failed status if any of its tasks failed.

Thanks for the comments, Laura! Yes, we were slightly surprised by the 
effect of the short circuit operator as well. Looking at the 
documentation, though, it appears consistent enough, since no task 
actually fails:

"The ShortCircuitOperator is derived from the PythonOperator. It 
evaluates a condition and short-circuits the workflow if the condition 
is False. Any downstream tasks are marked with a state of “skipped”. If 
the condition is True, downstream tasks proceed as normal." (from [1])

 From the code [2]:

     def execute(self, context):
         condition = super(ShortCircuitOperator, self).execute(context)
         if condition:
             logging.info('Proceeding with downstream tasks...')
             return
         else:
             logging.info('Skipping downstream tasks...')
             ...
             logging.info("Done.")

Perhaps it would be worth considering a "fail_on_false" flag for the 
operator? That might be something we'll try ourselves as a simple custom 
extension of this operator.

Regards

ap

[1] 
https://pythonhosted.org/airflow/code.html#airflow.operators.ShortCircuitOperator
[2] 
https://pythonhosted.org/airflow/_modules/python_operator.html#ShortCircuitOperator

Mime
View raw message