Mailing-List: contact dev-help@airflow.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@airflow.incubator.apache.org
MIME-Version: 1.0
In-Reply-To: <CAPdFGJsv=GOEqatpWoQU4+DNr35e=4jzstoBjHWNBerVUbD+PQ@mail.gmail.com>
References: <CAPdFGJsv=GOEqatpWoQU4+DNr35e=4jzstoBjHWNBerVUbD+PQ@mail.gmail.com>
From: Boris Tyukin <boris@boristyukin.com>
Date: Tue, 18 Oct 2016 16:12:59 -0400
Message-ID: <CANRT7T3=M-ceFUTk9-Vf5bwSLJ_mViWBnRiO9suidewvWVWBKQ@mail.gmail.com>
Subject: Re: Usage of "on_failure_callback" ?
To: dev@airflow.incubator.apache.org
Content-Type: multipart/alternative; boundary=001a113d028c76dda3053f295323
archived-at: Tue, 18 Oct 2016 20:13:37 -0000

--001a113d028c76dda3053f295323
Content-Type: text/plain; charset=UTF-8

hi Jason,

here is an example below - in  task0_python_callable i did 1/0 to raise an
error and Airflow then called task0_failure_cb. I could see FAIL in the
task log so my  task0_failure_cb was definitely invoked.

The task would still have failed status if look at UI. Are you expecting a
different behavior?


def task0_python_callable(ds, **kwargs):
    print ds
    print 1/0
    return 'Success!!! Execution time = {}'.format(kwargs['execution_date'])


def task0_failure_cb(ds, **kwargs):
    logging.info("FAIL")

task0 = PythonOperator(
    task_id='task0',
    python_callable=task0_python_callable,
    on_failure_callback=task0_failure_cb,
    provide_context=True,
    dag=dag)


[2016-10-18 16:07:02,631] {models.py:154} INFO - Filling up the DagBag from
/home/oracle/airflow/dags/fail_callback.py
[2016-10-18 16:07:03,239] {models.py:154} INFO - Filling up the DagBag from
/home/oracle/airflow/dags/fail_callback.py
[2016-10-18 16:07:03,312] {models.py:1196} INFO -
--------------------------------------------------------------------------------
Starting attempt 1 of 1
--------------------------------------------------------------------------------

[2016-10-18 16:07:03,317] {models.py:1219} INFO - Executing
<Task(PythonOperator): task0> on 2016-10-18 16:06:02
[2016-10-18 16:07:03,322] {models.py:1286} ERROR - integer division or
modulo by zero
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line
1245, in run
    result = task_copy.execute(context=context)
  File
"/usr/local/lib/python2.7/site-packages/airflow/operators/python_operator.py",
line 66, in execute
    return_value = self.python_callable(*self.op_args, **self.op_kwargs)
  File "/home/oracle/airflow/dags/fail_callback.py", line 33, in
task0_python_callable
    print 1/0
ZeroDivisionError: integer division or modulo by zero
[2016-10-18 16:07:03,324] {models.py:1306} INFO - Marking task as FAILED.
[2016-10-18 16:07:03,324] {fail_callback.py:38} INFO - FAIL
[2016-10-18 16:07:03,327] {models.py:1327} ERROR - integer division or
modulo by zero


On Tue, Oct 18, 2016 at 2:59 PM, Jason Chen <chingchien.chen@gmail.com>
wrote:

> Hi airflow team,
>
> Is there any sample code to use "on_failure_callback" ?
> I tried to use that as a callback to "post-process" when a task fails.
> However, I cannot make it work.
>
> The definition of my task is as below (just one task in my dag).
>
> It invokes "task0_python_callable" which executes a command-line script and
> it could drops exception. It in turns making task0 failed.
> In that case, I am thinking it should trigger the "on_failure_callback"
> function "task0_failure_cb", but it does not happen.
>
> Any suggestion?
> Thanks.
>
> Jason
>
> =========
> task0 = PythonOperator(
>     task_id='task0',
>     python_callable=task0_python_callable,
>     on_failure_callback=task0_failure_cb,
>     dag=dag)
>

--001a113d028c76dda3053f295323--