airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Maguire <andrewm4...@gmail.com>
Subject Re: BashOperator exit status
Date Tue, 14 Nov 2017 15:25:06 GMT
Oh, good to know.

It was just an image of this log info:

[2017-11-14 15:09:34,595] {base_task_runner.py:95} INFO - Subtask:
[2017-11-14 15:09:34,594] {bash_operator.py:94} INFO -
##################################
[2017-11-14 15:09:34,595] {base_task_runner.py:95} INFO - Subtask:
[2017-11-14 15:09:34,594] {bash_operator.py:94} INFO - LOAD PREDICTION MAP
[2017-11-14 15:09:34,595] {base_task_runner.py:95} INFO - Subtask:
[2017-11-14 15:09:34,594] {bash_operator.py:94} INFO -
##################################
*[2017-11-14 15:09:36,695] {base_task_runner.py:95} INFO - Subtask:
[2017-11-14 15:09:36,694] {bash_operator.py:94} INFO - BigQuery error in
load operation: Error processing job*
*[2017-11-14 15:09:36,695] {base_task_runner.py:95} INFO - Subtask:
[2017-11-14 15:09:36,695] {bash_operator.py:94} INFO -
'pmc-analytical-data-mart:bqjob_r4352c5357a8f26c6_0000015fbb14107a_1': Not*
*[2017-11-14 15:09:36,695] {base_task_runner.py:95} INFO - Subtask:
[2017-11-14 15:09:36,695] {bash_operator.py:94} INFO - found: Uris
gs://pmc-ml/clickmodel/vy/pred//prediction_map**
*[2017-11-14 15:09:38,277] {base_task_runner.py:95} INFO - Subtask:
[2017-11-14 15:09:38,276] {bash_operator.py:94} INFO - BigQuery error in
show operation: Not found: Table*
*[2017-11-14 15:09:38,277] {base_task_runner.py:95} INFO - Subtask:
[2017-11-14 15:09:38,277] {bash_operator.py:94} INFO -
pmc-analytical-data-mart:variety.clickmodel_predictions_map_20171112*
[2017-11-14 15:09:38,332] {base_task_runner.py:95} INFO - Subtask:
[2017-11-14 15:09:38,331] {bash_operator.py:97} INFO - Command exited with
return code 0
[2017-11-14 15:09:40,783] {jobs.py:2125} INFO - Task exited with return
code 0

So you can see the lines in bold are failed bq commands, but for some
reason (maybe how the bq cli operates) Airflow still thinks the task was
successful.

I think if i was to put something like the below into the bash script then
that would be enough to trigger a failure to airflow - thoughts?

# capture status of last command and exit if error
status=$?
if [ $status -ne 0 ]; then
  echo "Return code was not zero but $status"
  exit $status
fi

On Tue, Nov 14, 2017 at 3:21 PM Alek Storm <alek.storm@gmail.com> wrote:

> Hi Andy,
>
> The list doesn't allow inline images to be posted - can you paste your
> script content as text?
>
> Alek
>
> On Nov 14, 2017 9:16 AM, "Andrew Maguire" <andrewm4894@gmail.com> wrote:
>
> > Hi,
> >
> > I have some bash operators that are failing but airflow is not picking
> the
> > failure up.
> >
> > Here is an example:
> >
> > [image: image.png]
> >
> > This is a bash script that makes some "bq" and "gcloud" cli commands.
> >
> > I've used $? to get the status of such failed cli commands in the past
> and
> > then do something.
> >
> > I've just wondering - how could i use the $? from the failed bq command
> to
> > in turn pass an error to Airflow?
> >
> > Cheers,
> > Andy
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message