airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mike Perry (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AIRFLOW-839) docker_operator.py attempts to log status key without first checking existence
Date Sun, 05 Feb 2017 06:22:41 GMT

     [ https://issues.apache.org/jira/browse/AIRFLOW-839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Mike Perry updated AIRFLOW-839:
-------------------------------
    Description: 
When pulling a docker image, docker_operator.py attempts to log the `status` key each time
it gets output. This is usually fine, but occasionaly no `status` key exists. We've seen this
happen when we run out of inode space in our cluster and the docker cli is unable to extract
the image. This is consistent with the docker HTTP api docs (https://docs.docker.com/engine/api/v1.24/#create-an-image).
If an error occurs, there won't be a `status` key. There will be an `error` key. 

This is a relatively minor bug, but it obscures the real issue and can sometimes make it difficult
to figure out what went wrong. 

If you agree this should be fixed, I can submit a pr that first checks for status before logging.


here's the code that would need changed: https://github.com/apache/incubator-airflow/blob/master/airflow/operators/docker_operator.py#L156

Stack Trace
[2017-01-10 22:47:21,980] {models.py:1286} ERROR - 'status'
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 1245, in run
    result = task_copy.execute(context=context)
  File "/usr/local/lib/python2.7/dist-packages/airflow/operators/docker_operator.py", line
150, in execute
    logging.info("{}".format(output['status']))
KeyError: 'status'
[2017-01-10 22:47:21,982] {models.py:1298} INFO - Marking task as UP_FOR_RETRY
[2017-01-10 22:47:22,006] {models.py:1327} ERROR - 'status'
[2017-01-10 22:47:22,704] {jobs.py:159} DEBUG - [heart] Boom.

  was:
When pulling a docker image, docker_operator.py attempts to log the `status` key each time
it gets output. This is usually fine, but occasionaly no `status` key exists. We've seen this
happen when we run out of inode space in our cluster and the docker cli is unable to extract
the image. This is consistent with the docker HTTP api docs (https://docs.docker.com/engine/api/v1.24/#create-an-image).
If an error occurs, there won't be a `status` key. There will be an `error` key. 

This is a relatively minor bug, but it obscures the real issue and can sometimes make it difficult
to figure out what went wrong. 

If you agree this should be fixed, I can submit a pr that first checks for status before logging.


here's the code that would need changed: https://github.com/apache/incubator-airflow/blob/master/airflow/operators/docker_operator.py#L156


> docker_operator.py attempts to log status key without first checking existence
> ------------------------------------------------------------------------------
>
>                 Key: AIRFLOW-839
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-839
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: docker
>    Affects Versions: Airflow 1.7.1
>         Environment: arch linux
> python 2.7 and python 3
>            Reporter: Mike Perry
>            Assignee: Mike Perry
>            Priority: Trivial
>
> When pulling a docker image, docker_operator.py attempts to log the `status` key each
time it gets output. This is usually fine, but occasionaly no `status` key exists. We've seen
this happen when we run out of inode space in our cluster and the docker cli is unable to
extract the image. This is consistent with the docker HTTP api docs (https://docs.docker.com/engine/api/v1.24/#create-an-image).
If an error occurs, there won't be a `status` key. There will be an `error` key. 
> This is a relatively minor bug, but it obscures the real issue and can sometimes make
it difficult to figure out what went wrong. 
> If you agree this should be fixed, I can submit a pr that first checks for status before
logging. 
> here's the code that would need changed: https://github.com/apache/incubator-airflow/blob/master/airflow/operators/docker_operator.py#L156
> Stack Trace
> [2017-01-10 22:47:21,980] {models.py:1286} ERROR - 'status'
> Traceback (most recent call last):
>   File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 1245, in run
>     result = task_copy.execute(context=context)
>   File "/usr/local/lib/python2.7/dist-packages/airflow/operators/docker_operator.py",
line 150, in execute
>     logging.info("{}".format(output['status']))
> KeyError: 'status'
> [2017-01-10 22:47:21,982] {models.py:1298} INFO - Marking task as UP_FOR_RETRY
> [2017-01-10 22:47:22,006] {models.py:1327} ERROR - 'status'
> [2017-01-10 22:47:22,704] {jobs.py:159} DEBUG - [heart] Boom.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message