airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vitor Baptista (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-565) DockerOperator doesn't work on Python3.4
Date Tue, 11 Oct 2016 18:03:20 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15566109#comment-15566109
] 

Vitor Baptista commented on AIRFLOW-565:
----------------------------------------

Went ahead and sent a pull request for it on https://github.com/apache/incubator-airflow/pull/1832

> DockerOperator doesn't work on Python3.4
> ----------------------------------------
>
>                 Key: AIRFLOW-565
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-565
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: docker, operators
>    Affects Versions: Airflow 1.7.1.3
>         Environment: Python 3.4.3
>            Reporter: Vitor Baptista
>
> On {{DockerOperator.execute()}} we have:
> {code}
> if self.force_pull or len(self.cli.images(name=image)) == 0:
>         logging.info('Pulling docker image ' + image)
>         for l in self.cli.pull(image, stream=True):
>             output = json.loads(l)
>             logging.info("{}".format(output['status']))
> {code}
> https://github.com/apache/incubator-airflow/blob/master/airflow/operators/docker_operator.py#L152-L156
> The {{self.cli.pull()}} method returns {{bytes}} in Python3.4, and {{json.loads()}} expects
a string, so we end up with:
> {code}
> Traceback (most recent call last):
>   File "/home/vitor/Projetos/okfn/opentrials/airflow/env/lib/python3.4/site-packages/airflow/models.py",
line 1245, in run
>     result = task_copy.execute(context=context)
>   File "/home/vitor/Projetos/okfn/opentrials/airflow/env/lib/python3.4/site-packages/airflow/operators/docker_operator.py",
line 142, in execute
>     logging.info("{}".format(output['status']))
>   File "/usr/lib/python3.4/json/__init__.py", line 312, in loads
>     s.__class__.__name__))
> TypeError: the JSON object must be str, not 'bytes'
> {code}
> To avoid this, we could simply change it to {{output = json.loads(l.encode('utf-8'))}}.
This hardcodes the string as UTF-8, which should be fine, considering the JSON spec requires
the use of UTF-8, UTF-16 or UTF-32. As we're dealing with a Docker server, we can assume they'll
be well behaved.
> I'm happy to submit a pull request for this if you agree with the solution.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message