airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Maguire <andrewm4...@gmail.com>
Subject Re: BashOperator exit status
Date Tue, 14 Nov 2017 15:41:07 GMT
Thanks,

I've added this into my script and it now triggers error in Airflow.

# capture status of last command and exit if error
  status=$?
  if [ $status -ne 0 ]; then
    echo "Return code was not zero but $status"
    exit $status
  fi

Now it triggers below output.

I'll try figure out what's going on as would be a bit ugly to have to add
this everywhere (although for now will work fine if i add to just the last
step even).

[2017-11-14 15:33:57,545] {base_task_runner.py:95} INFO - Subtask:
[2017-11-14 15:33:57,545] {bash_operator.py:94} INFO -
##################################
[2017-11-14 15:33:57,546] {base_task_runner.py:95} INFO - Subtask:
[2017-11-14 15:33:57,545] {bash_operator.py:94} INFO - LOAD PREDICTIONS
[2017-11-14 15:33:57,546] {base_task_runner.py:95} INFO - Subtask:
[2017-11-14 15:33:57,545] {bash_operator.py:94} INFO -
##################################
[2017-11-14 15:34:00,147] {base_task_runner.py:95} INFO - Subtask:
[2017-11-14 15:34:00,146] {bash_operator.py:94} INFO - BigQuery error in
load operation: Error processing job
[2017-11-14 15:34:00,147] {base_task_runner.py:95} INFO - Subtask:
[2017-11-14 15:34:00,147] {bash_operator.py:94} INFO -
'pmc-analytical-data-mart:bqjob_r57a28bce8e342beb_0000015fbb2a633b_1': Not
[2017-11-14 15:34:00,147] {base_task_runner.py:95} INFO - Subtask:
[2017-11-14 15:34:00,147] {bash_operator.py:94} INFO - found: Uris
gs://pmc-ml/clickmodel/vy/pred//prediction.results*
[2017-11-14 15:34:00,209] {base_task_runner.py:95} INFO - Subtask:
[2017-11-14 15:34:00,208] {bash_operator.py:94} INFO - Return code was not
zero but 2
[2017-11-14 15:34:00,209] {base_task_runner.py:95} INFO - Subtask:
[2017-11-14 15:34:00,209] {bash_operator.py:97} INFO - Command exited with
return code 2
[2017-11-14 15:34:00,210] {base_task_runner.py:95} INFO - Subtask:
[2017-11-14 15:34:00,209] {models.py:1433} ERROR - Bash command failed
[2017-11-14 15:34:00,210] {base_task_runner.py:95} INFO - Subtask:
Traceback (most recent call last):
[2017-11-14 15:34:00,210] {base_task_runner.py:95} INFO - Subtask:   File
"/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 1390, in
run
[2017-11-14 15:34:00,210] {base_task_runner.py:95} INFO - Subtask:
 result = task_copy.execute(context=context)
[2017-11-14 15:34:00,210] {base_task_runner.py:95} INFO - Subtask:   File
"/usr/local/lib/python2.7/dist-packages/airflow/operators/bash_operator.py",
line 100, in execute
[2017-11-14 15:34:00,211] {base_task_runner.py:95} INFO - Subtask:
 raise AirflowException("Bash command failed")
[2017-11-14 15:34:00,211] {base_task_runner.py:95} INFO - Subtask:
AirflowException: Bash command failed
[2017-11-14 15:34:00,211] {base_task_runner.py:95} INFO - Subtask:
[2017-11-14 15:34:00,210] {models.py:1449} INFO - Marking task as
UP_FOR_RETRY
[2017-11-14 15:34:00,474] {base_task_runner.py:95} INFO - Subtask:
[2017-11-14 15:34:00,474] {email.py:109} INFO - Sent an alert email to ['
andrew.maguire@pmc.com']
[2017-11-14 15:34:02,181] {base_task_runner.py:95} INFO - Subtask:
[2017-11-14 15:34:02,181] {models.py:1478} ERROR - Bash command failed
[2017-11-14 15:34:02,182] {base_task_runner.py:95} INFO - Subtask:
Traceback (most recent call last):
[2017-11-14 15:34:02,183] {base_task_runner.py:95} INFO - Subtask:   File
"/usr/local/bin/airflow", line 28, in <module>
[2017-11-14 15:34:02,183] {base_task_runner.py:95} INFO - Subtask:
 args.func(args)
[2017-11-14 15:34:02,183] {base_task_runner.py:95} INFO - Subtask:   File
"/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py", line 422, in
run
[2017-11-14 15:34:02,184] {base_task_runner.py:95} INFO - Subtask:
 pool=args.pool,
[2017-11-14 15:34:02,184] {base_task_runner.py:95} INFO - Subtask:   File
"/usr/local/lib/python2.7/dist-packages/airflow/utils/db.py", line 53, in
wrapper
[2017-11-14 15:34:02,185] {base_task_runner.py:95} INFO - Subtask:
 result = func(*args, **kwargs)
[2017-11-14 15:34:02,185] {base_task_runner.py:95} INFO - Subtask:   File
"/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 1390, in
run
[2017-11-14 15:34:02,186] {base_task_runner.py:95} INFO - Subtask:
 result = task_copy.execute(context=context)
[2017-11-14 15:34:02,187] {base_task_runner.py:95} INFO - Subtask:   File
"/usr/local/lib/python2.7/dist-packages/airflow/operators/bash_operator.py",
line 100, in execute
[2017-11-14 15:34:02,187] {base_task_runner.py:95} INFO - Subtask:
 raise AirflowException("Bash command failed")
[2017-11-14 15:34:02,188] {base_task_runner.py:95} INFO - Subtask:
airflow.exceptions.AirflowException: Bash command failed
[2017-11-14 15:34:06,994] {jobs.py:2125} INFO - Task exited with return
code 1

On Tue, Nov 14, 2017 at 3:33 PM Bolke de Bruin <bdbruin@gmail.com> wrote:

> Hi Andrew,
>
> Your task is exiting with “code 0” which means success. I would verify
> that you are not swallowing the error/return code somewhere.
>
> Cheers
> Bolke
>
> Verstuurd vanaf mijn iPad
>
> > Op 14 nov. 2017 om 16:25 heeft Andrew Maguire <andrewm4894@gmail.com>
> het volgende geschreven:
> >
> > Oh, good to know.
> >
> > It was just an image of this log info:
> >
> > [2017-11-14 15:09:34,595] {base_task_runner.py:95} INFO - Subtask:
> > [2017-11-14 15:09:34,594] {bash_operator.py:94} INFO -
> > ##################################
> > [2017-11-14 15:09:34,595] {base_task_runner.py:95} INFO - Subtask:
> > [2017-11-14 15:09:34,594] {bash_operator.py:94} INFO - LOAD PREDICTION
> MAP
> > [2017-11-14 15:09:34,595] {base_task_runner.py:95} INFO - Subtask:
> > [2017-11-14 15:09:34,594] {bash_operator.py:94} INFO -
> > ##################################
> > *[2017-11-14 15:09:36,695] {base_task_runner.py:95} INFO - Subtask:
> > [2017-11-14 15:09:36,694] {bash_operator.py:94} INFO - BigQuery error in
> > load operation: Error processing job*
> > *[2017-11-14 15:09:36,695] {base_task_runner.py:95} INFO - Subtask:
> > [2017-11-14 15:09:36,695] {bash_operator.py:94} INFO -
> > 'pmc-analytical-data-mart:bqjob_r4352c5357a8f26c6_0000015fbb14107a_1':
> Not*
> > *[2017-11-14 15:09:36,695] {base_task_runner.py:95} INFO - Subtask:
> > [2017-11-14 15:09:36,695] {bash_operator.py:94} INFO - found: Uris
> > gs://pmc-ml/clickmodel/vy/pred//prediction_map**
> > *[2017-11-14 15:09:38,277] {base_task_runner.py:95} INFO - Subtask:
> > [2017-11-14 15:09:38,276] {bash_operator.py:94} INFO - BigQuery error in
> > show operation: Not found: Table*
> > *[2017-11-14 15:09:38,277] {base_task_runner.py:95} INFO - Subtask:
> > [2017-11-14 15:09:38,277] {bash_operator.py:94} INFO -
> > pmc-analytical-data-mart:variety.clickmodel_predictions_map_20171112*
> > [2017-11-14 15:09:38,332] {base_task_runner.py:95} INFO - Subtask:
> > [2017-11-14 15:09:38,331] {bash_operator.py:97} INFO - Command exited
> with
> > return code 0
> > [2017-11-14 15:09:40,783] {jobs.py:2125} INFO - Task exited with return
> > code 0
> >
> > So you can see the lines in bold are failed bq commands, but for some
> > reason (maybe how the bq cli operates) Airflow still thinks the task was
> > successful.
> >
> > I think if i was to put something like the below into the bash script
> then
> > that would be enough to trigger a failure to airflow - thoughts?
> >
> > # capture status of last command and exit if error
> > status=$?
> > if [ $status -ne 0 ]; then
> >  echo "Return code was not zero but $status"
> >  exit $status
> > fi
> >
> >> On Tue, Nov 14, 2017 at 3:21 PM Alek Storm <alek.storm@gmail.com>
> wrote:
> >>
> >> Hi Andy,
> >>
> >> The list doesn't allow inline images to be posted - can you paste your
> >> script content as text?
> >>
> >> Alek
> >>
> >>> On Nov 14, 2017 9:16 AM, "Andrew Maguire" <andrewm4894@gmail.com>
> wrote:
> >>>
> >>> Hi,
> >>>
> >>> I have some bash operators that are failing but airflow is not picking
> >> the
> >>> failure up.
> >>>
> >>> Here is an example:
> >>>
> >>> [image: image.png]
> >>>
> >>> This is a bash script that makes some "bq" and "gcloud" cli commands.
> >>>
> >>> I've used $? to get the status of such failed cli commands in the past
> >> and
> >>> then do something.
> >>>
> >>> I've just wondering - how could i use the $? from the failed bq command
> >> to
> >>> in turn pass an error to Airflow?
> >>>
> >>> Cheers,
> >>> Andy
> >>>
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message